Sep 11 17:54:02.045022 ip-10-0-128-29 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 17:54:02.464670 ip-10-0-128-29 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:54:02.464670 ip-10-0-128-29 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 17:54:02.464670 ip-10-0-128-29 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:54:02.464670 ip-10-0-128-29 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 17:54:02.464670 ip-10-0-128-29 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 17:54:02.466021 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.465919    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 17:54:02.472615 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472593    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:54:02.472615 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472613    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:54:02.472615 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472617    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:54:02.472615 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472620    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:54:02.472796 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472623    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:54:02.472796 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472626    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:54:02.472796 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472629    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:54:02.472796 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472631    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:54:02.472796 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472634    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:54:02.472796 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472639    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:54:02.472796 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472643    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:54:02.472796 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472645    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:54:02.472796 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472648    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:54:02.472796 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472651    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:54:02.472796 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472654    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:54:02.472796 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472656    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:54:02.472796 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472659    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:54:02.472796 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472662    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:54:02.472796 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472664    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:54:02.472796 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472667    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:54:02.472796 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472669    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:54:02.472796 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472672    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:54:02.472796 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472674    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:54:02.472796 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472677    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:54:02.473298 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472680    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:54:02.473298 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472682    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:54:02.473298 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472685    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:54:02.473298 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472687    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:54:02.473298 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472690    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:54:02.473298 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472693    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:54:02.473298 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472696    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:54:02.473298 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472699    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:54:02.473298 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472702    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:54:02.473298 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472704    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:54:02.473298 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472706    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:54:02.473298 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472709    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:54:02.473298 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472712    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:54:02.473298 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472715    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:54:02.473298 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472717    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:54:02.473298 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472720    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:54:02.473298 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472722    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:54:02.473298 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472725    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:54:02.473298 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472728    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:54:02.473298 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472730    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:54:02.473801 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472733    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:54:02.473801 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472736    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:54:02.473801 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472738    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:54:02.473801 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472741    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:54:02.473801 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472744    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:54:02.473801 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472758    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:54:02.473801 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472761    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:54:02.473801 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472764    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:54:02.473801 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472766    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:54:02.473801 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472769    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:54:02.473801 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472772    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:54:02.473801 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472774    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:54:02.473801 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472777    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:54:02.473801 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472780    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:54:02.473801 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472783    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:54:02.473801 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472785    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:54:02.473801 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472788    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:54:02.473801 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472790    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:54:02.473801 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472793    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:54:02.473801 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472795    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:54:02.474290 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472798    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:54:02.474290 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472800    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:54:02.474290 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472803    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:54:02.474290 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472806    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:54:02.474290 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472809    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:54:02.474290 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472812    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:54:02.474290 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472815    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:54:02.474290 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472817    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:54:02.474290 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472820    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:54:02.474290 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472822    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:54:02.474290 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472824    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:54:02.474290 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472829    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:54:02.474290 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472832    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:54:02.474290 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472835    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:54:02.474290 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472838    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:54:02.474290 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472841    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:54:02.474290 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472843    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:54:02.474290 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472846    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:54:02.474290 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472848    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:54:02.474787 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472851    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:54:02.474787 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472854    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:54:02.474787 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472856    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:54:02.474787 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472859    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:54:02.474787 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472861    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:54:02.474787 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472863    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:54:02.474787 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472866    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:54:02.474787 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472868    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:54:02.474787 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472871    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:54:02.474787 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472873    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:54:02.474787 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472875    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:54:02.474787 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472877    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:54:02.474787 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472880    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:54:02.474787 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472883    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:54:02.474787 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472888    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:54:02.474787 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.472890    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:54:02.474787 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473621    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:54:02.474787 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473629    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:54:02.474787 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473633    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:54:02.475260 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473636    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:54:02.475260 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473639    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:54:02.475260 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473642    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:54:02.475260 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473644    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:54:02.475260 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473647    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:54:02.475260 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473650    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:54:02.475260 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473652    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:54:02.475260 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473655    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:54:02.475260 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473658    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:54:02.475260 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473660    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:54:02.475260 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473663    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:54:02.475260 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473665    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:54:02.475260 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473668    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:54:02.475260 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473670    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:54:02.475260 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473673    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:54:02.475260 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473676    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:54:02.475260 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473679    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:54:02.475260 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473681    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:54:02.475260 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473684    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:54:02.475260 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473686    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:54:02.475867 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473689    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:54:02.475867 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473693    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:54:02.475867 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473695    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:54:02.475867 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473697    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:54:02.475867 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473700    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:54:02.475867 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473702    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:54:02.475867 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473705    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:54:02.475867 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473708    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:54:02.475867 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473710    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:54:02.475867 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473712    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:54:02.475867 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473715    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:54:02.475867 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473720    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:54:02.475867 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473722    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:54:02.475867 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473725    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:54:02.475867 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473728    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:54:02.475867 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473730    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:54:02.475867 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473733    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:54:02.475867 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473736    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:54:02.475867 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473738    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:54:02.475867 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473740    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:54:02.476362 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473743    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:54:02.476362 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473766    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:54:02.476362 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473771    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:54:02.476362 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473775    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:54:02.476362 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473779    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:54:02.476362 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473782    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:54:02.476362 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473785    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:54:02.476362 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473789    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:54:02.476362 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473792    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:54:02.476362 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473794    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:54:02.476362 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473797    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:54:02.476362 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473799    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:54:02.476362 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473802    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:54:02.476362 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473805    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:54:02.476362 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473808    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:54:02.476362 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473810    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:54:02.476362 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473813    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:54:02.476362 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473816    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:54:02.476362 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473818    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:54:02.476362 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473821    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:54:02.476859 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473824    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:54:02.476859 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473826    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:54:02.476859 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473830    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:54:02.476859 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473833    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:54:02.476859 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473835    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:54:02.476859 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473838    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:54:02.476859 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473841    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:54:02.476859 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473843    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:54:02.476859 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473846    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:54:02.476859 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473848    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:54:02.476859 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473851    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:54:02.476859 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473854    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:54:02.476859 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473856    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:54:02.476859 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473861    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:54:02.476859 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473865    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:54:02.476859 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473869    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:54:02.476859 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473872    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:54:02.476859 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473875    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:54:02.476859 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473879    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:54:02.477347 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473883    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:54:02.477347 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473886    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:54:02.477347 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473889    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:54:02.477347 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473892    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:54:02.477347 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473894    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:54:02.477347 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473896    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:54:02.477347 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473899    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:54:02.477347 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473901    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:54:02.477347 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473904    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:54:02.477347 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473906    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:54:02.477347 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473909    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:54:02.477347 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473911    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:54:02.477347 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473914    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:54:02.477347 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473916    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:54:02.477347 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473919    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:54:02.477347 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473924    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:54:02.477347 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.473926    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:54:02.477347 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474288    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:54:02.477347 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474292    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:54:02.477347 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474296    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:54:02.477851 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474298    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:54:02.477851 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474301    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:54:02.477851 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474304    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:54:02.477851 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474306    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:54:02.477851 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474309    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:54:02.477851 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474312    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:54:02.477851 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474314    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:54:02.477851 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474317    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:54:02.477851 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474319    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:54:02.477851 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474323    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:54:02.477851 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474327    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:54:02.477851 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474329    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:54:02.477851 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474332    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:54:02.477851 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474335    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:54:02.477851 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474337    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:54:02.477851 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474340    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:54:02.477851 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474342    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:54:02.477851 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474346    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:54:02.477851 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474349    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:54:02.478320 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474352    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:54:02.478320 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474355    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:54:02.478320 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474358    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:54:02.478320 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474361    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:54:02.478320 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474364    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:54:02.478320 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474367    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:54:02.478320 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474370    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:54:02.478320 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474372    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:54:02.478320 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474376    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:54:02.478320 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474379    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:54:02.478320 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474382    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:54:02.478320 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474385    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:54:02.478320 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474387    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:54:02.478320 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474390    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:54:02.478320 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474393    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:54:02.478320 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474395    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:54:02.478320 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474398    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:54:02.478320 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474401    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:54:02.478320 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474403    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:54:02.478840 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474405    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:54:02.478840 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474408    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:54:02.478840 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474410    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:54:02.478840 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474413    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:54:02.478840 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474415    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:54:02.478840 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474418    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:54:02.478840 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474420    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:54:02.478840 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474423    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:54:02.478840 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474426    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:54:02.478840 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474428    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:54:02.478840 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474431    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:54:02.478840 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474433    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:54:02.478840 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474436    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:54:02.478840 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474438    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:54:02.478840 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474441    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:54:02.478840 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474443    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:54:02.478840 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474445    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:54:02.478840 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474448    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:54:02.478840 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474451    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:54:02.478840 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474453    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:54:02.479312 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474455    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:54:02.479312 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474459    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:54:02.479312 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474462    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:54:02.479312 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474465    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:54:02.479312 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474467    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:54:02.479312 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474470    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:54:02.479312 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474472    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:54:02.479312 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474475    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:54:02.479312 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474478    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:54:02.479312 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474480    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:54:02.479312 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474482    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:54:02.479312 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474485    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:54:02.479312 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474488    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:54:02.479312 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474490    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:54:02.479312 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474494    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:54:02.479312 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474497    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:54:02.479312 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474500    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:54:02.479312 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474502    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:54:02.479312 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474505    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:54:02.479312 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474508    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474510    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474513    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474516    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474518    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474521    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474523    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474525    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474528    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474530    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474533    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474535    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474537    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474540    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474544    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474546    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474549    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474551    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.474554    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474630    2560 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474664    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 17:54:02.479819 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474671    2560 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474675    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474680    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474683    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474687    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474691    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474694    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474697    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474700    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474704    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474707    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474709    2560 flags.go:64] FLAG: --cgroup-root=""
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474712    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474715    2560 flags.go:64] FLAG: --client-ca-file=""
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474718    2560 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474722    2560 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474727    2560 flags.go:64] FLAG: --cluster-domain=""
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474729    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474732    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474735    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474739    2560 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474761    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474764    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 17:54:02.480360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474767    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474770    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474775    2560 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474778    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474781    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474784    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474787    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474791    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474801    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474805    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474808    2560 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474811    2560 flags.go:64] FLAG: --enable-server="true"
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474814    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474819    2560 flags.go:64] FLAG: --event-burst="100"
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474821    2560 flags.go:64] FLAG: --event-qps="50"
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474824    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474827    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474830    2560 flags.go:64] FLAG: --eviction-hard=""
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474834    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474838    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474841    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474844    2560 flags.go:64] FLAG: --eviction-soft=""
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474847    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474849    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474853    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 17:54:02.480941 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474856    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474858    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474861    2560 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474864    2560 flags.go:64] FLAG: --feature-gates=""
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474870    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474873    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474876    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474879    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474882    2560 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474885    2560 flags.go:64] FLAG: --help="false"
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474888    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-128-29.ec2.internal"
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474891    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474893    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474896    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474899    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474902    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474905    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474908    2560 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474910    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474913    2560 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474916    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474919    2560 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474921    2560 flags.go:64] FLAG: --kube-reserved=""
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474924    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 17:54:02.481555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474927    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474930    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474934    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474936    2560 flags.go:64] FLAG: --lock-file=""
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474939    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474942    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474945    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474950    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474953    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474956    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474959    2560 flags.go:64] FLAG: --logging-format="text"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474961    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474964    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474967    2560 flags.go:64] FLAG: --manifest-url=""
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474970    2560 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474974    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474977    2560 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474981    2560 flags.go:64] FLAG: --max-pods="110"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474984    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474987    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474989    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474992    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474995    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.474998    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475001    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 17:54:02.482154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475009    2560 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475012    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475014    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475017    2560 flags.go:64] FLAG: --pod-cidr=""
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475020    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475025    2560 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475028    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475030    2560 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475033    2560 flags.go:64] FLAG: --port="10250"
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475038    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475040    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02ec8c81a62ef5b3f"
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475043    2560 flags.go:64] FLAG: --qos-reserved=""
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475046    2560 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475049    2560 flags.go:64] FLAG: --register-node="true"
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475052    2560 flags.go:64] FLAG: --register-with-taints=""
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475056    2560 flags.go:64] FLAG: --registry-burst="10"
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475059    2560 flags.go:64] FLAG: --registry-qps="5"
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475062    2560 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475064    2560 flags.go:64] FLAG: --reserved-memory=""
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475068    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475071    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475073    2560 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475076    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475079    2560 flags.go:64] FLAG: --runonce="false"
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475081    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 17:54:02.482766 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475084    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475087    2560 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475090    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475093    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475095    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475098    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475101    2560 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475103    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475106    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475109    2560 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475111    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475114    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475117    2560 flags.go:64] FLAG: --system-cgroups=""
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475120    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475125    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475128    2560 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475131    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475135    2560 flags.go:64] FLAG: --tls-min-version=""
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475138    2560 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475141    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475144    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475146    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475149    2560 flags.go:64] FLAG: --v="2"
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475153    2560 flags.go:64] FLAG: --version="false"
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475157    2560 flags.go:64] FLAG: --vmodule=""
Sep 11 17:54:02.483378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475160    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 17:54:02.484046 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475164    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 17:54:02.484046 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475274    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:54:02.484046 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475278    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:54:02.484046 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475282    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:54:02.484046 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475284    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:54:02.484046 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475287    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:54:02.484046 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475290    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:54:02.484046 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475292    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:54:02.484046 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475295    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:54:02.484046 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475297    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:54:02.484046 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475300    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:54:02.484046 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475302    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:54:02.484046 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475305    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:54:02.484046 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475307    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:54:02.484046 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475309    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:54:02.484046 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475312    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:54:02.484046 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475314    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:54:02.484046 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475317    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:54:02.484046 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475319    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:54:02.484046 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475322    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:54:02.484535 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475324    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:54:02.484535 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475326    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:54:02.484535 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475331    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:54:02.484535 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475333    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:54:02.484535 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475336    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:54:02.484535 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475338    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:54:02.484535 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475340    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:54:02.484535 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475343    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:54:02.484535 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475346    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:54:02.484535 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475348    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:54:02.484535 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475350    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:54:02.484535 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475353    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:54:02.484535 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475356    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:54:02.484535 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475359    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:54:02.484535 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475361    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:54:02.484535 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475364    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:54:02.484535 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475366    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:54:02.484535 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475371    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:54:02.484535 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475374    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:54:02.484535 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475376    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:54:02.485027 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475379    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:54:02.485027 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475381    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:54:02.485027 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475384    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:54:02.485027 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475386    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:54:02.485027 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475389    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:54:02.485027 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475392    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:54:02.485027 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475394    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:54:02.485027 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475396    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:54:02.485027 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475399    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:54:02.485027 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475403    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:54:02.485027 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475406    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:54:02.485027 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475409    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:54:02.485027 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475411    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:54:02.485027 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475414    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:54:02.485027 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475418    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:54:02.485027 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475420    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:54:02.485027 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475423    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:54:02.485027 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475425    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:54:02.485027 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475427    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:54:02.485027 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475430    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:54:02.485527 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475433    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:54:02.485527 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475435    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:54:02.485527 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475438    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:54:02.485527 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475440    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:54:02.485527 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475443    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:54:02.485527 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475446    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:54:02.485527 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475448    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:54:02.485527 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475451    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:54:02.485527 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475453    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:54:02.485527 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475457    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:54:02.485527 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475460    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:54:02.485527 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475462    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:54:02.485527 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475466    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:54:02.485527 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475470    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:54:02.485527 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475472    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:54:02.485527 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475475    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:54:02.485527 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475478    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:54:02.485527 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475481    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:54:02.485527 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475484    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:54:02.486006 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475486    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:54:02.486006 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475489    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:54:02.486006 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475492    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:54:02.486006 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475495    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:54:02.486006 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475497    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:54:02.486006 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475500    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:54:02.486006 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475502    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:54:02.486006 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475506    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:54:02.486006 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475509    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:54:02.486006 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475511    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:54:02.486006 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475513    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:54:02.486006 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475516    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:54:02.486006 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475519    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:54:02.486006 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475521    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:54:02.486006 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475523    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:54:02.486006 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475526    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:54:02.486006 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475528    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:54:02.486006 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475530    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:54:02.486006 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475535    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:54:02.486463 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475538    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:54:02.486463 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.475541    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:54:02.486463 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.475546    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:54:02.486463 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.482933    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 17:54:02.486463 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.482949    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 17:54:02.486463 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.483774    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 17:54:02.486463 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.483782    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 17:54:02.486463 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483838    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:54:02.486463 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483842    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:54:02.486463 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483846    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:54:02.486463 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483849    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:54:02.486463 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483851    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:54:02.486463 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483854    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:54:02.486463 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483857    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:54:02.486916 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483859    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:54:02.486916 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483862    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:54:02.486916 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483865    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:54:02.486916 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483867    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:54:02.486916 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483870    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:54:02.486916 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483872    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:54:02.486916 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483875    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:54:02.486916 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483877    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:54:02.486916 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483880    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:54:02.486916 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483882    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:54:02.486916 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483885    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:54:02.486916 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483887    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:54:02.486916 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483890    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:54:02.486916 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483892    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:54:02.486916 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483895    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:54:02.486916 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483898    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:54:02.486916 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483901    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:54:02.486916 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483905    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:54:02.486916 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483908    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:54:02.486916 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483911    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:54:02.487429 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483914    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:54:02.487429 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483918    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:54:02.487429 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483923    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:54:02.487429 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483926    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:54:02.487429 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483928    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:54:02.487429 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483931    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:54:02.487429 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483934    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:54:02.487429 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483936    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:54:02.487429 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483939    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:54:02.487429 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483941    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:54:02.487429 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483944    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:54:02.487429 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483947    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:54:02.487429 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483949    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:54:02.487429 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483951    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:54:02.487429 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483954    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:54:02.487429 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483956    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:54:02.487429 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483959    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:54:02.487429 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483961    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:54:02.487429 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483964    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:54:02.487904 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483966    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:54:02.487904 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483969    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:54:02.487904 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483972    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:54:02.487904 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483974    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:54:02.487904 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483977    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:54:02.487904 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483979    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:54:02.487904 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483982    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:54:02.487904 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483984    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:54:02.487904 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483987    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:54:02.487904 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483989    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:54:02.487904 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483992    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:54:02.487904 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483995    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:54:02.487904 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.483997    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:54:02.487904 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484000    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:54:02.487904 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484003    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:54:02.487904 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484006    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:54:02.487904 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484009    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:54:02.487904 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484012    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:54:02.487904 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484014    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:54:02.487904 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484017    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:54:02.488382 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484020    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:54:02.488382 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484022    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:54:02.488382 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484024    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:54:02.488382 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484027    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:54:02.488382 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484029    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:54:02.488382 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484032    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:54:02.488382 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484035    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:54:02.488382 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484037    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:54:02.488382 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484040    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:54:02.488382 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484042    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:54:02.488382 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484045    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:54:02.488382 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484047    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:54:02.488382 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484051    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:54:02.488382 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484055    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:54:02.488382 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484058    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:54:02.488382 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484061    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:54:02.488382 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484063    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:54:02.488382 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484066    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:54:02.488382 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484069    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:54:02.488382 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484071    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:54:02.488884 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484074    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:54:02.488884 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484077    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:54:02.488884 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484079    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:54:02.488884 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484081    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:54:02.488884 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484084    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:54:02.488884 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484086    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:54:02.488884 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484089    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:54:02.488884 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484092    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:54:02.488884 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484095    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:54:02.488884 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484098    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:54:02.488884 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484100    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:54:02.488884 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484103    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:54:02.488884 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484106    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:54:02.488884 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.484110    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:54:02.488884 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484230    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 17:54:02.489278 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484234    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 17:54:02.489278 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484237    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 17:54:02.489278 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484240    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 17:54:02.489278 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484243    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 17:54:02.489278 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484246    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 17:54:02.489278 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484249    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 17:54:02.489278 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484251    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 17:54:02.489278 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484254    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 17:54:02.489278 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484256    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 17:54:02.489278 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484258    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 17:54:02.489278 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484261    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 17:54:02.489278 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484264    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 17:54:02.489278 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484266    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 17:54:02.489278 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484268    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 17:54:02.489278 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484271    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 17:54:02.489278 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484273    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 17:54:02.489278 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484276    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 17:54:02.489278 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484278    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 17:54:02.489278 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484281    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 17:54:02.489278 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484285    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 17:54:02.489783 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484288    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 17:54:02.489783 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484290    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 17:54:02.489783 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484293    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 17:54:02.489783 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484295    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 17:54:02.489783 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484298    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 17:54:02.489783 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484300    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 17:54:02.489783 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484303    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 17:54:02.489783 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484306    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 17:54:02.489783 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484308    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 17:54:02.489783 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484311    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 17:54:02.489783 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484315    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 17:54:02.489783 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484318    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 17:54:02.489783 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484321    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 17:54:02.489783 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484324    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 17:54:02.489783 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484326    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 17:54:02.489783 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484329    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 17:54:02.489783 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484332    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 17:54:02.489783 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484335    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 17:54:02.489783 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484338    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 17:54:02.489783 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484340    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 17:54:02.490270 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484343    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 17:54:02.490270 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484345    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 17:54:02.490270 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484347    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 17:54:02.490270 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484350    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 17:54:02.490270 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484352    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 17:54:02.490270 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484356    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 17:54:02.490270 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484359    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 17:54:02.490270 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484362    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 17:54:02.490270 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484365    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 17:54:02.490270 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484368    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 17:54:02.490270 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484371    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 17:54:02.490270 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484373    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 17:54:02.490270 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484376    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 17:54:02.490270 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484378    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 17:54:02.490270 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484381    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 17:54:02.490270 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484383    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 17:54:02.490270 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484386    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 17:54:02.490270 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484388    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 17:54:02.490723 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484391    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 17:54:02.490723 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484393    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 17:54:02.490723 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484396    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 17:54:02.490723 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484399    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 17:54:02.490723 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484401    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 17:54:02.490723 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484404    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 17:54:02.490723 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484406    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 17:54:02.490723 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484409    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 17:54:02.490723 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484411    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 17:54:02.490723 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484414    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 17:54:02.490723 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484416    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 17:54:02.490723 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484419    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 17:54:02.490723 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484421    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 17:54:02.490723 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484424    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 17:54:02.490723 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484426    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 17:54:02.490723 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484428    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 17:54:02.490723 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484431    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 17:54:02.490723 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484433    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 17:54:02.490723 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484435    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 17:54:02.490723 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484438    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 17:54:02.491242 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484440    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 17:54:02.491242 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484443    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 17:54:02.491242 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484445    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 17:54:02.491242 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484448    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 17:54:02.491242 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484450    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 17:54:02.491242 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484453    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 17:54:02.491242 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484455    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 17:54:02.491242 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484457    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 17:54:02.491242 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484460    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 17:54:02.491242 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484462    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 17:54:02.491242 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484465    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 17:54:02.491242 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484468    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 17:54:02.491242 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484470    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 17:54:02.491242 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484473    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 17:54:02.491242 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484475    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 17:54:02.491242 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484478    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 17:54:02.491242 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484481    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 17:54:02.491242 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484483    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 17:54:02.491242 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484485    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 17:54:02.491242 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:02.484488    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 17:54:02.491712 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.484492    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 17:54:02.491712 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.484592    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 17:54:02.491712 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.486768    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 17:54:02.491712 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.487821    2560 server.go:1019] "Starting client certificate rotation"
Sep 11 17:54:02.491712 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.487917    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 17:54:02.491712 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.487946    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 17:54:02.491712 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.490703    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 17:54:02.493550 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.493535    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 17:54:02.505233 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.505136    2560 log.go:25] "Validated CRI v1 runtime API"
Sep 11 17:54:02.510144 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.510129    2560 log.go:25] "Validated CRI v1 image API"
Sep 11 17:54:02.510977 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.510957    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 17:54:02.513465 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.513446    2560 fs.go:135] Filesystem UUIDs: map[769ec3bf-df62-4f04-8c7f-2b24e745ef7b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fcbf3740-6d31-4ed2-8eb9-a546767caf0e:/dev/nvme0n1p4]
Sep 11 17:54:02.513537 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.513464    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 17:54:02.518518 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.518406    2560 manager.go:216] Machine: {Timestamp:2026-09-11 17:54:02.517026424 +0000 UTC m=+0.360261868 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098059 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2cefdad547f8fdb94daf1f12d7d2d0 SystemUUID:ec2cefda-d547-f8fd-b94d-af1f12d7d2d0 BootID:247a832b-4d20-428d-bb11-436aae6e8988 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d5:0a:87:aa:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d5:0a:87:aa:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:9b:63:c4:12:7e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 17:54:02.518518 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.518510    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 17:54:02.518640 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.518609    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 17:54:02.521968 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.521943    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 17:54:02.522178 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.522161    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 17:54:02.522231 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.521972    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-29.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 17:54:02.523630 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.523617    2560 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 17:54:02.523684 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.523638    2560 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 17:54:02.523684 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.523648    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 17:54:02.524321 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.524308    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 17:54:02.525570 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.525559    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 17:54:02.529832 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.529819    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 17:54:02.529962 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.529951    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 17:54:02.530021 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.529992    2560 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 17:54:02.530021 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.530005    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 17:54:02.530102 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.530021    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 17:54:02.530102 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.530035    2560 kubelet.go:398] "Adding apiserver pod source"
Sep 11 17:54:02.530102 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.530048    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 17:54:02.530396 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.530382    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 17:54:02.530461 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.530405    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 17:54:02.533646 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.533619    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 17:54:02.536976 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.536959    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 17:54:02.537037 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.536994    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 17:54:02.538843 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.538830    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 17:54:02.538916 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.538853    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 17:54:02.538916 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.538862    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 17:54:02.538916 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.538868    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 17:54:02.538916 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.538877    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 17:54:02.538916 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.538883    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 17:54:02.538916 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.538897    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 17:54:02.538916 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.538902    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 17:54:02.538916 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.538913    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 17:54:02.538916 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.538921    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 17:54:02.539221 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.538929    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 17:54:02.539221 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.538934    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 17:54:02.539799 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.539785    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 17:54:02.539870 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.539802    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 17:54:02.539940 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:02.539923    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 17:54:02.539940 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:02.539927    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-29.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 17:54:02.543531 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.543517    2560 server.go:1268] "Started kubelet"
Sep 11 17:54:02.543604 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.543573    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 17:54:02.543695 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.543644    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 17:54:02.543778 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.543718    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 17:54:02.544364 ip-10-0-128-29 systemd[1]: Started Kubernetes Kubelet.
Sep 11 17:54:02.547944 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.547918    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-29.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 17:54:02.548227 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.548207    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 17:54:02.549091 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.549071    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xp2jl"
Sep 11 17:54:02.549201 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:02.549176    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 17:54:02.549773 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.549742    2560 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 17:54:02.554359 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:02.553192    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-29.ec2.internal.18d455c1bbf6c201  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-29.ec2.internal,UID:ip-10-0-128-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-29.ec2.internal,},FirstTimestamp:2026-09-11 17:54:02.543489537 +0000 UTC m=+0.386724968,LastTimestamp:2026-09-11 17:54:02.543489537 +0000 UTC m=+0.386724968,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-29.ec2.internal,}"
Sep 11 17:54:02.555852 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.555816    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xp2jl"
Sep 11 17:54:02.562420 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.562403    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 17:54:02.563209 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.563191    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 17:54:02.563314 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:02.563295    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 11 17:54:02.564007 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.563984    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 17:54:02.564007 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.563988    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 17:54:02.564007 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.564007    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 17:54:02.564157 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.564078    2560 factory.go:55] Registering systemd factory
Sep 11 17:54:02.564157 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.564091    2560 factory.go:223] Registration of the systemd container factory successfully
Sep 11 17:54:02.564157 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.564079    2560 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 17:54:02.564157 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.564123    2560 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 17:54:02.564318 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.564284    2560 factory.go:153] Registering CRI-O factory
Sep 11 17:54:02.564318 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.564293    2560 factory.go:223] Registration of the crio container factory successfully
Sep 11 17:54:02.564377 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.564332    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 17:54:02.564377 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.564346    2560 factory.go:103] Registering Raw factory
Sep 11 17:54:02.564377 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.564367    2560 manager.go:1196] Started watching for new ooms in manager
Sep 11 17:54:02.564646 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.564635    2560 manager.go:349] Starting recovery of all containers
Sep 11 17:54:02.565643 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:02.565620    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 17:54:02.565762 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:02.565711    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-29.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 17:54:02.573807 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.573793    2560 manager.go:354] Recovery completed
Sep 11 17:54:02.577585 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.577568    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:54:02.580483 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.580468    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:54:02.580555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.580498    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:54:02.580555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.580507    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:54:02.581103 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.581088    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 17:54:02.581103 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.581099    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 17:54:02.581203 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.581116    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 17:54:02.581203 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.581100    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:54:02.582865 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.582850    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:54:02.582953 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.582870    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:54:02.582953 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.582879    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:54:02.583130 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.583119    2560 policy_none.go:49] "None policy: Start"
Sep 11 17:54:02.583173 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.583134    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 17:54:02.583173 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.583144    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 17:54:02.585093 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.585082    2560 policy_none.go:47] "Start"
Sep 11 17:54:02.627379 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.627363    2560 manager.go:337] "Starting Device Plugin manager"
Sep 11 17:54:02.641408 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:02.627406    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 17:54:02.641408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.627418    2560 server.go:87] "Starting device plugin registration server"
Sep 11 17:54:02.641408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.627680    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 17:54:02.641408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.627691    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 17:54:02.641408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.627794    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 17:54:02.641408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.627866    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 17:54:02.641408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.627872    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 17:54:02.641408 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:02.628395    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 17:54:02.641408 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:02.628451    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 11 17:54:02.679374 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.679340    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 17:54:02.680683 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.680664    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 17:54:02.680799 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.680709    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 17:54:02.680799 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.680735    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 17:54:02.680887 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:02.680796    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 17:54:02.683104 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.683083    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:54:02.727887 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.727779    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:54:02.728804 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.728777    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:54:02.728915 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.728818    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:54:02.728915 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.728830    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:54:02.728915 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.728853    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-29.ec2.internal"
Sep 11 17:54:02.738244 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.738228    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-29.ec2.internal"
Sep 11 17:54:02.738339 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:02.738251    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-29.ec2.internal\": node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 11 17:54:02.755812 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:02.755793    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 11 17:54:02.781664 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.781645    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal"]
Sep 11 17:54:02.781719 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.781700    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:54:02.782559 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.782543    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:54:02.782633 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.782572    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:54:02.782633 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.782583    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:54:02.784894 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.784881    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:54:02.785007 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.784993    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Sep 11 17:54:02.785055 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.785021    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:54:02.785630 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.785610    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:54:02.785703 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.785641    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:54:02.785703 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.785651    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:54:02.785703 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.785610    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:54:02.785839 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.785717    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:54:02.785839 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.785728    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:54:02.787896 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.787880    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal"
Sep 11 17:54:02.787986 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.787904    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 17:54:02.788623 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.788608    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 17:54:02.788697 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.788631    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 17:54:02.788697 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.788640    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeHasSufficientPID"
Sep 11 17:54:02.817240 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:02.817224    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-29.ec2.internal\" not found" node="ip-10-0-128-29.ec2.internal"
Sep 11 17:54:02.821465 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:02.821450    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-29.ec2.internal\" not found" node="ip-10-0-128-29.ec2.internal"
Sep 11 17:54:02.856878 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:02.856856    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 11 17:54:02.957771 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:02.957727    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 11 17:54:02.965407 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.965385    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdf7e40390d26689763c696c18fb8f21-config\") pod \"kube-apiserver-proxy-ip-10-0-128-29.ec2.internal\" (UID: \"fdf7e40390d26689763c696c18fb8f21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal"
Sep 11 17:54:02.965505 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.965415    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/42daa9f52c2b3824ddc9a288af86477f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal\" (UID: \"42daa9f52c2b3824ddc9a288af86477f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Sep 11 17:54:02.965505 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:02.965439    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42daa9f52c2b3824ddc9a288af86477f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal\" (UID: \"42daa9f52c2b3824ddc9a288af86477f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Sep 11 17:54:03.058099 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:03.058032    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 11 17:54:03.066361 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:03.066335    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42daa9f52c2b3824ddc9a288af86477f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal\" (UID: \"42daa9f52c2b3824ddc9a288af86477f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Sep 11 17:54:03.066478 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:03.066379    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdf7e40390d26689763c696c18fb8f21-config\") pod \"kube-apiserver-proxy-ip-10-0-128-29.ec2.internal\" (UID: \"fdf7e40390d26689763c696c18fb8f21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal"
Sep 11 17:54:03.066478 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:03.066400    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/42daa9f52c2b3824ddc9a288af86477f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal\" (UID: \"42daa9f52c2b3824ddc9a288af86477f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Sep 11 17:54:03.066478 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:03.066463    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fdf7e40390d26689763c696c18fb8f21-config\") pod \"kube-apiserver-proxy-ip-10-0-128-29.ec2.internal\" (UID: \"fdf7e40390d26689763c696c18fb8f21\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal"
Sep 11 17:54:03.066579 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:03.066515    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/42daa9f52c2b3824ddc9a288af86477f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal\" (UID: \"42daa9f52c2b3824ddc9a288af86477f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Sep 11 17:54:03.066579 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:03.066538    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/42daa9f52c2b3824ddc9a288af86477f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal\" (UID: \"42daa9f52c2b3824ddc9a288af86477f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Sep 11 17:54:03.119475 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:03.119443    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Sep 11 17:54:03.124254 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:03.124097    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal"
Sep 11 17:54:03.158604 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:03.158576    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 11 17:54:03.259169 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:03.259133    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 11 17:54:03.359695 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:03.359618    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 11 17:54:03.460295 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:03.460266    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 11 17:54:03.487741 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:03.487718    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 17:54:03.488152 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:03.487875    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 17:54:03.560377 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:03.560353    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 11 17:54:03.565999 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:03.565969    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 17:49:03 +0000 UTC" deadline="2028-03-08 12:05:26.415523717 +0000 UTC"
Sep 11 17:54:03.565999 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:03.565999    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13050h11m22.849527524s"
Sep 11 17:54:03.660926 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:03.660869    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 11 17:54:03.703959 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:03.703939    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:54:03.761534 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:03.761507    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 11 17:54:03.817651 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:03.817620    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42daa9f52c2b3824ddc9a288af86477f.slice/crio-6978b85216070ea3b4817864a9683f4ec64d8cd021adbcc96995faf105ddcb0a WatchSource:0}: Error finding container 6978b85216070ea3b4817864a9683f4ec64d8cd021adbcc96995faf105ddcb0a: Status 404 returned error can't find the container with id 6978b85216070ea3b4817864a9683f4ec64d8cd021adbcc96995faf105ddcb0a
Sep 11 17:54:03.817914 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:03.817896    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf7e40390d26689763c696c18fb8f21.slice/crio-3d1b9bfff7df5dc12e889bedea890a5580821f88b06ca0056743d771aa9cc193 WatchSource:0}: Error finding container 3d1b9bfff7df5dc12e889bedea890a5580821f88b06ca0056743d771aa9cc193: Status 404 returned error can't find the container with id 3d1b9bfff7df5dc12e889bedea890a5580821f88b06ca0056743d771aa9cc193
Sep 11 17:54:03.821142 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:03.821124    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:54:03.861848 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:03.861824    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 11 17:54:03.915988 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:03.915925    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:54:03.962486 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:03.962441    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-29.ec2.internal\" not found"
Sep 11 17:54:04.011083 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.011057    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:54:04.063387 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.063367    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal"
Sep 11 17:54:04.076061 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.076044    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 17:54:04.076934 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.076923    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal"
Sep 11 17:54:04.090657 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.090640    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 17:54:04.532429 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.531952    2560 apiserver.go:52] "Watching apiserver"
Sep 11 17:54:04.542927 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.542033    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 17:54:04.542927 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.542554    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-vw8qd","kube-system/konnectivity-agent-v4zkn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz","openshift-cluster-node-tuning-operator/tuned-7vgrl","openshift-dns/node-resolver-pjbh8","openshift-image-registry/node-ca-fsfql","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal","openshift-multus/multus-additional-cni-plugins-z9p7p","openshift-network-diagnostics/network-check-target-xfh2z","kube-system/global-pull-secret-syncer-r7cvn","kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal","openshift-multus/multus-xxtsv","openshift-multus/network-metrics-daemon-8xg8d","openshift-network-operator/iptables-alerter-w5twj"]
Sep 11 17:54:04.545522 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.545499    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:04.545644 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:04.545597    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7cvn" podUID="c5702497-d40f-4a13-b5b8-0be72c784011"
Sep 11 17:54:04.548995 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.548968    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:04.549101 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:04.549061    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xg8d" podUID="8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf"
Sep 11 17:54:04.552556 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.552505    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.556488 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.556140    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.557679 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.557645    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pjbh8"
Sep 11 17:54:04.559703 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.559682    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fsfql"
Sep 11 17:54:04.562317 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.562289    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:04.562426 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:04.562363    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfh2z" podUID="2c3f9d2a-8267-48f4-9c2e-d06ff6f15485"
Sep 11 17:54:04.563653 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.563561    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 17:54:04.565893 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.565568    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w5twj"
Sep 11 17:54:04.571813 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.571791    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 17:54:04.571954 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.571917    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 17:54:04.572159 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.572140    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qf8gv\""
Sep 11 17:54:04.572225 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.572200    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v4zkn"
Sep 11 17:54:04.572377 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.572355    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:54:04.572671 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.572653    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.574165 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.573307    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-48tgb\""
Sep 11 17:54:04.574165 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.573317    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 17:54:04.574165 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.573324    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:54:04.574165 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.573544    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 17:54:04.574165 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.573599    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 17:54:04.574165 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.573649    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 17:54:04.574165 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.573755    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-46tn5\""
Sep 11 17:54:04.574165 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.573900    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rh8gd\""
Sep 11 17:54:04.574607 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.574319    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 17:54:04.575985 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.575960    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8970f95d-bfe6-4d2c-97ea-be9489333243-tmp-dir\") pod \"node-resolver-pjbh8\" (UID: \"8970f95d-bfe6-4d2c-97ea-be9489333243\") " pod="openshift-dns/node-resolver-pjbh8"
Sep 11 17:54:04.576079 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576001    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-var-lib-kubelet\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.576079 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576020    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-host\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.576079 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576038    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-tmp\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.576250 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576094    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zp9l\" (UniqueName: \"kubernetes.io/projected/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-kube-api-access-5zp9l\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.576250 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576120    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-sys\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.576250 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576145    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91dbf02a-fc59-4467-9c9f-aafb5b1594c4-host\") pod \"node-ca-fsfql\" (UID: \"91dbf02a-fc59-4467-9c9f-aafb5b1594c4\") " pod="openshift-image-registry/node-ca-fsfql"
Sep 11 17:54:04.576250 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576171    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrdrw\" (UniqueName: \"kubernetes.io/projected/8970f95d-bfe6-4d2c-97ea-be9489333243-kube-api-access-hrdrw\") pod \"node-resolver-pjbh8\" (UID: \"8970f95d-bfe6-4d2c-97ea-be9489333243\") " pod="openshift-dns/node-resolver-pjbh8"
Sep 11 17:54:04.576250 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576197    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krcpd\" (UniqueName: \"kubernetes.io/projected/6124639f-d16d-443d-9d14-ee9fe666f0d7-kube-api-access-krcpd\") pod \"iptables-alerter-w5twj\" (UID: \"6124639f-d16d-443d-9d14-ee9fe666f0d7\") " pod="openshift-network-operator/iptables-alerter-w5twj"
Sep 11 17:54:04.576250 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576221    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5702497-d40f-4a13-b5b8-0be72c784011-kubelet-config\") pod \"global-pull-secret-syncer-r7cvn\" (UID: \"c5702497-d40f-4a13-b5b8-0be72c784011\") " pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:04.576250 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576245    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.576521 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576270    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-registration-dir\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.576521 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576294    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs\") pod \"network-metrics-daemon-8xg8d\" (UID: \"8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf\") " pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:04.576521 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576317    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-run\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.576521 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576338    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.576521 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576343    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-lib-modules\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.576521 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576366    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-socket-dir\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.576521 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576389    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-kubernetes\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.576521 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576418    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-sys-fs\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.576521 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576445    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6124639f-d16d-443d-9d14-ee9fe666f0d7-iptables-alerter-script\") pod \"iptables-alerter-w5twj\" (UID: \"6124639f-d16d-443d-9d14-ee9fe666f0d7\") " pod="openshift-network-operator/iptables-alerter-w5twj"
Sep 11 17:54:04.576521 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576469    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6124639f-d16d-443d-9d14-ee9fe666f0d7-host-slash\") pod \"iptables-alerter-w5twj\" (UID: \"6124639f-d16d-443d-9d14-ee9fe666f0d7\") " pod="openshift-network-operator/iptables-alerter-w5twj"
Sep 11 17:54:04.576521 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576492    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-sysctl-d\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.576521 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576515    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-sysctl-conf\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.576982 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576541    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5702497-d40f-4a13-b5b8-0be72c784011-dbus\") pod \"global-pull-secret-syncer-r7cvn\" (UID: \"c5702497-d40f-4a13-b5b8-0be72c784011\") " pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:04.576982 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576567    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.576982 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576599    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc59g\" (UniqueName: \"kubernetes.io/projected/91dbf02a-fc59-4467-9c9f-aafb5b1594c4-kube-api-access-xc59g\") pod \"node-ca-fsfql\" (UID: \"91dbf02a-fc59-4467-9c9f-aafb5b1594c4\") " pod="openshift-image-registry/node-ca-fsfql"
Sep 11 17:54:04.576982 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576630    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-modprobe-d\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.576982 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576662    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-sysconfig\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.576982 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576712    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-systemd\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.576982 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576797    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw5d9\" (UniqueName: \"kubernetes.io/projected/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-kube-api-access-rw5d9\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.576982 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576860    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb5bc\" (UniqueName: \"kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc\") pod \"network-check-target-xfh2z\" (UID: \"2c3f9d2a-8267-48f4-9c2e-d06ff6f15485\") " pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:04.576982 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576885    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret\") pod \"global-pull-secret-syncer-r7cvn\" (UID: \"c5702497-d40f-4a13-b5b8-0be72c784011\") " pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:04.576982 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576911    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8970f95d-bfe6-4d2c-97ea-be9489333243-hosts-file\") pod \"node-resolver-pjbh8\" (UID: \"8970f95d-bfe6-4d2c-97ea-be9489333243\") " pod="openshift-dns/node-resolver-pjbh8"
Sep 11 17:54:04.576982 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576935    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdqrb\" (UniqueName: \"kubernetes.io/projected/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-kube-api-access-jdqrb\") pod \"network-metrics-daemon-8xg8d\" (UID: \"8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf\") " pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:04.576982 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.576978    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-tuned\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.577416 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.577001    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91dbf02a-fc59-4467-9c9f-aafb5b1594c4-serviceca\") pod \"node-ca-fsfql\" (UID: \"91dbf02a-fc59-4467-9c9f-aafb5b1594c4\") " pod="openshift-image-registry/node-ca-fsfql"
Sep 11 17:54:04.577416 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.577040    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-device-dir\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.577416 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.577303    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 17:54:04.578160 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.578133    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 17:54:04.578245 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.578202    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ww77d\""
Sep 11 17:54:04.578245 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.578150    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 17:54:04.578430 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.578408    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 17:54:04.578509 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.578459    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 17:54:04.578509 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.578408    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 17:54:04.578509 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.578500    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 17:54:04.578695 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.578463    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 17:54:04.579864 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.579842    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.581491 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.581468    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 17:54:04.584100 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.584080    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sfpw5\""
Sep 11 17:54:04.588584 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.588562    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b8ctk\""
Sep 11 17:54:04.588789 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.588772    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 17:54:04.588973 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.588952    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 17:54:04.589217 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.589203    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 17:54:04.589293 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.589232    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 17:54:04.589362 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.589331    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 17:54:04.589709 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.589527    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 17:54:04.589709 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.589539    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 17:54:04.589709 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.589548    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9xv89\""
Sep 11 17:54:04.589709 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.589565    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 17:54:04.589709 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.589643    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 17:54:04.589709 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.589674    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-42pz4\""
Sep 11 17:54:04.616626 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.616604    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 17:54:04.642954 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.642926    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7v72s"
Sep 11 17:54:04.655829 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.655805    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7v72s"
Sep 11 17:54:04.665215 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.665184    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 17:54:04.677314 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677287    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91dbf02a-fc59-4467-9c9f-aafb5b1594c4-serviceca\") pod \"node-ca-fsfql\" (UID: \"91dbf02a-fc59-4467-9c9f-aafb5b1594c4\") " pod="openshift-image-registry/node-ca-fsfql"
Sep 11 17:54:04.677436 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677330    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-cnibin\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.677436 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677364    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-host-var-lib-cni-multus\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.677436 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677393    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.677436 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677418    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0274e7fb-2731-44a2-8f19-2670eb16afca-system-cni-dir\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.677658 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677444    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-host\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.677658 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677470    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zp9l\" (UniqueName: \"kubernetes.io/projected/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-kube-api-access-5zp9l\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.677658 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677496    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d274725b-319d-4396-8035-c63e2f6a13a7-multus-daemon-config\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.677658 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677521    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-ovnkube-config\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.677658 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677547    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-sys\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.677658 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677571    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0274e7fb-2731-44a2-8f19-2670eb16afca-cnibin\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.677658 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677595    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0274e7fb-2731-44a2-8f19-2670eb16afca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.677658 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677621    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrdrw\" (UniqueName: \"kubernetes.io/projected/8970f95d-bfe6-4d2c-97ea-be9489333243-kube-api-access-hrdrw\") pod \"node-resolver-pjbh8\" (UID: \"8970f95d-bfe6-4d2c-97ea-be9489333243\") " pod="openshift-dns/node-resolver-pjbh8"
Sep 11 17:54:04.677658 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677647    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krcpd\" (UniqueName: \"kubernetes.io/projected/6124639f-d16d-443d-9d14-ee9fe666f0d7-kube-api-access-krcpd\") pod \"iptables-alerter-w5twj\" (UID: \"6124639f-d16d-443d-9d14-ee9fe666f0d7\") " pod="openshift-network-operator/iptables-alerter-w5twj"
Sep 11 17:54:04.678124 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677674    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.678124 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677699    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-registration-dir\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.678124 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d274725b-319d-4396-8035-c63e2f6a13a7-cni-binary-copy\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.678124 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677751    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-etc-openvswitch\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.678124 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677776    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs\") pod \"network-metrics-daemon-8xg8d\" (UID: \"8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf\") " pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:04.678124 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677820    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0274e7fb-2731-44a2-8f19-2670eb16afca-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.678124 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677867    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/91dbf02a-fc59-4467-9c9f-aafb5b1594c4-serviceca\") pod \"node-ca-fsfql\" (UID: \"91dbf02a-fc59-4467-9c9f-aafb5b1594c4\") " pod="openshift-image-registry/node-ca-fsfql"
Sep 11 17:54:04.678124 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677888    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-kubernetes\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.678124 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677900    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-sys\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.678124 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677916    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-system-cni-dir\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.678124 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677954    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-multus-socket-dir-parent\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.678124 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677960    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-registration-dir\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.678124 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677981    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-host-var-lib-cni-bin\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.678124 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.677993    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-host\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.678124 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678052    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-run-netns\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.678124 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678070    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.678124 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678085    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-env-overrides\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.678935 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678238    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-kubernetes\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.678935 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:04.678352    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:04.678935 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678395    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6124639f-d16d-443d-9d14-ee9fe666f0d7-iptables-alerter-script\") pod \"iptables-alerter-w5twj\" (UID: \"6124639f-d16d-443d-9d14-ee9fe666f0d7\") " pod="openshift-network-operator/iptables-alerter-w5twj"
Sep 11 17:54:04.678935 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:04.678443    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs podName:8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf nodeName:}" failed. No retries permitted until 2026-09-11 17:54:05.17839647 +0000 UTC m=+3.021631894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs") pod "network-metrics-daemon-8xg8d" (UID: "8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:04.678935 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678541    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6124639f-d16d-443d-9d14-ee9fe666f0d7-host-slash\") pod \"iptables-alerter-w5twj\" (UID: \"6124639f-d16d-443d-9d14-ee9fe666f0d7\") " pod="openshift-network-operator/iptables-alerter-w5twj"
Sep 11 17:54:04.678935 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678579    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-sysctl-d\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.678935 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678619    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dbe2052d-5694-4782-815b-0975eced80b8-konnectivity-ca\") pod \"konnectivity-agent-v4zkn\" (UID: \"dbe2052d-5694-4782-815b-0975eced80b8\") " pod="kube-system/konnectivity-agent-v4zkn"
Sep 11 17:54:04.678935 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678650    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0274e7fb-2731-44a2-8f19-2670eb16afca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.678935 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678694    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-modprobe-d\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.678935 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678719    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-sysconfig\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.678935 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678734    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-sysctl-d\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.678935 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678755    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-systemd\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.678935 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret\") pod \"global-pull-secret-syncer-r7cvn\" (UID: \"c5702497-d40f-4a13-b5b8-0be72c784011\") " pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:04.678935 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678822    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6124639f-d16d-443d-9d14-ee9fe666f0d7-host-slash\") pod \"iptables-alerter-w5twj\" (UID: \"6124639f-d16d-443d-9d14-ee9fe666f0d7\") " pod="openshift-network-operator/iptables-alerter-w5twj"
Sep 11 17:54:04.678935 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678824    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-kubelet\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.678935 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678864    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-systemd\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.678935 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678864    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.679555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678895    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8970f95d-bfe6-4d2c-97ea-be9489333243-hosts-file\") pod \"node-resolver-pjbh8\" (UID: \"8970f95d-bfe6-4d2c-97ea-be9489333243\") " pod="openshift-dns/node-resolver-pjbh8"
Sep 11 17:54:04.679555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678916    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdqrb\" (UniqueName: \"kubernetes.io/projected/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-kube-api-access-jdqrb\") pod \"network-metrics-daemon-8xg8d\" (UID: \"8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf\") " pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:04.679555 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:04.678927    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:04.679555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678957    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-modprobe-d\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.679555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.678970    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8970f95d-bfe6-4d2c-97ea-be9489333243-hosts-file\") pod \"node-resolver-pjbh8\" (UID: \"8970f95d-bfe6-4d2c-97ea-be9489333243\") " pod="openshift-dns/node-resolver-pjbh8"
Sep 11 17:54:04.679555 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:04.678992    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret podName:c5702497-d40f-4a13-b5b8-0be72c784011 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:05.178976808 +0000 UTC m=+3.022212250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret") pod "global-pull-secret-syncer-r7cvn" (UID: "c5702497-d40f-4a13-b5b8-0be72c784011") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:04.679555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679011    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-sysconfig\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.679555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679068    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6124639f-d16d-443d-9d14-ee9fe666f0d7-iptables-alerter-script\") pod \"iptables-alerter-w5twj\" (UID: \"6124639f-d16d-443d-9d14-ee9fe666f0d7\") " pod="openshift-network-operator/iptables-alerter-w5twj"
Sep 11 17:54:04.679555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679105    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-tuned\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.679555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679123    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dbe2052d-5694-4782-815b-0975eced80b8-agent-certs\") pod \"konnectivity-agent-v4zkn\" (UID: \"dbe2052d-5694-4782-815b-0975eced80b8\") " pod="kube-system/konnectivity-agent-v4zkn"
Sep 11 17:54:04.679555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679138    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-host-run-netns\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.679555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679161    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-hostroot\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.679555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679186    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkq4l\" (UniqueName: \"kubernetes.io/projected/d274725b-319d-4396-8035-c63e2f6a13a7-kube-api-access-qkq4l\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.679555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679209    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-cni-netd\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.679555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679238    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8970f95d-bfe6-4d2c-97ea-be9489333243-tmp-dir\") pod \"node-resolver-pjbh8\" (UID: \"8970f95d-bfe6-4d2c-97ea-be9489333243\") " pod="openshift-dns/node-resolver-pjbh8"
Sep 11 17:54:04.679555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679261    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-var-lib-kubelet\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.679555 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679284    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-tmp\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.680344 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679307    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-host-var-lib-kubelet\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.680344 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679332    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-slash\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.680344 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679356    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-var-lib-openvswitch\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.680344 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679395    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-run-ovn\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.680344 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679417    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg8vx\" (UniqueName: \"kubernetes.io/projected/0274e7fb-2731-44a2-8f19-2670eb16afca-kube-api-access-zg8vx\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.680344 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679439    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91dbf02a-fc59-4467-9c9f-aafb5b1594c4-host\") pod \"node-ca-fsfql\" (UID: \"91dbf02a-fc59-4467-9c9f-aafb5b1594c4\") " pod="openshift-image-registry/node-ca-fsfql"
Sep 11 17:54:04.680344 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679461    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-ovnkube-script-lib\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.680344 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679492    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5702497-d40f-4a13-b5b8-0be72c784011-kubelet-config\") pod \"global-pull-secret-syncer-r7cvn\" (UID: \"c5702497-d40f-4a13-b5b8-0be72c784011\") " pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:04.680344 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679512    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-systemd-units\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.680344 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679532    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-run\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.680344 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679552    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-lib-modules\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.680344 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679573    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-socket-dir\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.680344 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679593    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-host-run-k8s-cni-cncf-io\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.680344 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679620    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-host-run-multus-certs\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.680344 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679646    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-etc-kubernetes\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.680344 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679668    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-node-log\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.680344 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679688    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfvjb\" (UniqueName: \"kubernetes.io/projected/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-kube-api-access-nfvjb\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.681256 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679711    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-sys-fs\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.681256 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679734    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-multus-cni-dir\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.681256 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679772    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 17:54:04.681256 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679781    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/91dbf02a-fc59-4467-9c9f-aafb5b1594c4-host\") pod \"node-ca-fsfql\" (UID: \"91dbf02a-fc59-4467-9c9f-aafb5b1594c4\") " pod="openshift-image-registry/node-ca-fsfql"
Sep 11 17:54:04.681256 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679940    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-run\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.681256 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.680084    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-lib-modules\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.681256 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.680164    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8970f95d-bfe6-4d2c-97ea-be9489333243-tmp-dir\") pod \"node-resolver-pjbh8\" (UID: \"8970f95d-bfe6-4d2c-97ea-be9489333243\") " pod="openshift-dns/node-resolver-pjbh8"
Sep 11 17:54:04.681256 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.680212    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-socket-dir\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.681256 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.680250    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-var-lib-kubelet\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.681256 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.680330    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-sys-fs\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.681256 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.680340    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5702497-d40f-4a13-b5b8-0be72c784011-kubelet-config\") pod \"global-pull-secret-syncer-r7cvn\" (UID: \"c5702497-d40f-4a13-b5b8-0be72c784011\") " pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:04.681256 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.679776    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-run-systemd\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.681256 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.680607    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-ovn-node-metrics-cert\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.681256 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.680637    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-sysctl-conf\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.681256 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.680658    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5702497-d40f-4a13-b5b8-0be72c784011-dbus\") pod \"global-pull-secret-syncer-r7cvn\" (UID: \"c5702497-d40f-4a13-b5b8-0be72c784011\") " pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:04.681256 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.680691    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.681256 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.680710    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-multus-conf-dir\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.681256 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.680730    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-run-openvswitch\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.682113 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.680796    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-log-socket\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.682113 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.680819    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-device-dir\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.682113 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.680853    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc59g\" (UniqueName: \"kubernetes.io/projected/91dbf02a-fc59-4467-9c9f-aafb5b1594c4-kube-api-access-xc59g\") pod \"node-ca-fsfql\" (UID: \"91dbf02a-fc59-4467-9c9f-aafb5b1594c4\") " pod="openshift-image-registry/node-ca-fsfql"
Sep 11 17:54:04.682113 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.680875    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw5d9\" (UniqueName: \"kubernetes.io/projected/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-kube-api-access-rw5d9\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.682113 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.680897    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb5bc\" (UniqueName: \"kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc\") pod \"network-check-target-xfh2z\" (UID: \"2c3f9d2a-8267-48f4-9c2e-d06ff6f15485\") " pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:04.682113 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.680918    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-cni-bin\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.682113 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.680940    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0274e7fb-2731-44a2-8f19-2670eb16afca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.682113 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.681081    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-sysctl-conf\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.682113 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.681137    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5702497-d40f-4a13-b5b8-0be72c784011-dbus\") pod \"global-pull-secret-syncer-r7cvn\" (UID: \"c5702497-d40f-4a13-b5b8-0be72c784011\") " pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:04.682113 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.681207    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-etc-selinux\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.682113 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.681269    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-device-dir\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.687254 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.687224    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-tmp\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.687389 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.687334    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal" event={"ID":"fdf7e40390d26689763c696c18fb8f21","Type":"ContainerStarted","Data":"3d1b9bfff7df5dc12e889bedea890a5580821f88b06ca0056743d771aa9cc193"}
Sep 11 17:54:04.689469 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.689282    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-etc-tuned\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.689469 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.689420    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal" event={"ID":"42daa9f52c2b3824ddc9a288af86477f","Type":"ContainerStarted","Data":"6978b85216070ea3b4817864a9683f4ec64d8cd021adbcc96995faf105ddcb0a"}
Sep 11 17:54:04.692483 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.692459    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrdrw\" (UniqueName: \"kubernetes.io/projected/8970f95d-bfe6-4d2c-97ea-be9489333243-kube-api-access-hrdrw\") pod \"node-resolver-pjbh8\" (UID: \"8970f95d-bfe6-4d2c-97ea-be9489333243\") " pod="openshift-dns/node-resolver-pjbh8"
Sep 11 17:54:04.698567 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.698543    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdqrb\" (UniqueName: \"kubernetes.io/projected/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-kube-api-access-jdqrb\") pod \"network-metrics-daemon-8xg8d\" (UID: \"8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf\") " pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:04.700596 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:04.700576    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:54:04.700698 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:04.700601    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:54:04.700698 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:04.700624    2560 projected.go:196] Error preparing data for projected volume kube-api-access-zb5bc for pod openshift-network-diagnostics/network-check-target-xfh2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:04.700698 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:04.700692    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc podName:2c3f9d2a-8267-48f4-9c2e-d06ff6f15485 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:05.200668771 +0000 UTC m=+3.043904206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zb5bc" (UniqueName: "kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc") pod "network-check-target-xfh2z" (UID: "2c3f9d2a-8267-48f4-9c2e-d06ff6f15485") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:04.703144 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.703113    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc59g\" (UniqueName: \"kubernetes.io/projected/91dbf02a-fc59-4467-9c9f-aafb5b1594c4-kube-api-access-xc59g\") pod \"node-ca-fsfql\" (UID: \"91dbf02a-fc59-4467-9c9f-aafb5b1594c4\") " pod="openshift-image-registry/node-ca-fsfql"
Sep 11 17:54:04.703649 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.703625    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krcpd\" (UniqueName: \"kubernetes.io/projected/6124639f-d16d-443d-9d14-ee9fe666f0d7-kube-api-access-krcpd\") pod \"iptables-alerter-w5twj\" (UID: \"6124639f-d16d-443d-9d14-ee9fe666f0d7\") " pod="openshift-network-operator/iptables-alerter-w5twj"
Sep 11 17:54:04.704620 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.704597    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw5d9\" (UniqueName: \"kubernetes.io/projected/b4bd3b89-6b91-420b-aa10-1c5ac23cfabc-kube-api-access-rw5d9\") pod \"tuned-7vgrl\" (UID: \"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc\") " pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.705722 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.705701    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zp9l\" (UniqueName: \"kubernetes.io/projected/a88dccff-c714-4d6d-88e3-0f5e31b51b8f-kube-api-access-5zp9l\") pod \"aws-ebs-csi-driver-node-l6dtz\" (UID: \"a88dccff-c714-4d6d-88e3-0f5e31b51b8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.781947 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.781913    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dbe2052d-5694-4782-815b-0975eced80b8-agent-certs\") pod \"konnectivity-agent-v4zkn\" (UID: \"dbe2052d-5694-4782-815b-0975eced80b8\") " pod="kube-system/konnectivity-agent-v4zkn"
Sep 11 17:54:04.781947 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.781955    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-host-run-netns\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.782140 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.781973    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-hostroot\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.782140 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.781991    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkq4l\" (UniqueName: \"kubernetes.io/projected/d274725b-319d-4396-8035-c63e2f6a13a7-kube-api-access-qkq4l\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.782140 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782016    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-cni-netd\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.782140 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782044    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-host-var-lib-kubelet\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.782140 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782059    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-slash\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.782140 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782074    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-hostroot\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.782140 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782127    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-host-var-lib-kubelet\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.782140 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782133    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-cni-netd\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.782419 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782074    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-var-lib-openvswitch\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.782419 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782165    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-slash\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.782419 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782177    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-var-lib-openvswitch\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.782419 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782198    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-host-run-netns\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.782419 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782243    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-run-ovn\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.782419 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782267    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg8vx\" (UniqueName: \"kubernetes.io/projected/0274e7fb-2731-44a2-8f19-2670eb16afca-kube-api-access-zg8vx\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.782419 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782292    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-run-ovn\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.782419 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782317    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-ovnkube-script-lib\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.782419 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782335    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-systemd-units\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.782419 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782358    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-host-run-k8s-cni-cncf-io\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.782419 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782391    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-host-run-multus-certs\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.782910 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782429    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-host-run-multus-certs\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.782910 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782467    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-systemd-units\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.782910 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782490    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-etc-kubernetes\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.782910 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782501    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-host-run-k8s-cni-cncf-io\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.782910 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782513    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-node-log\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.782910 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782545    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-node-log\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.782910 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782551    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfvjb\" (UniqueName: \"kubernetes.io/projected/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-kube-api-access-nfvjb\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.782910 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782580    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-etc-kubernetes\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.782910 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782626    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-multus-cni-dir\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.782910 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782582    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-multus-cni-dir\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.782910 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782650    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-run-systemd\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.782910 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782665    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-ovn-node-metrics-cert\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.782910 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-multus-conf-dir\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.782910 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782699    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-run-openvswitch\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.782910 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782716    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-log-socket\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.782910 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782744    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-cni-bin\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.782910 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782760    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0274e7fb-2731-44a2-8f19-2670eb16afca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.782910 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782776    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-cnibin\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.783506 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782792    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-host-var-lib-cni-multus\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.783506 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782810    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.783506 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0274e7fb-2731-44a2-8f19-2670eb16afca-system-cni-dir\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.783506 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782850    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-log-socket\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.783506 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782867    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d274725b-319d-4396-8035-c63e2f6a13a7-multus-daemon-config\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.783506 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782904    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-run-systemd\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.783506 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782936    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-ovnkube-config\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.783506 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782967    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0274e7fb-2731-44a2-8f19-2670eb16afca-cnibin\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.783506 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782979    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-cnibin\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.783506 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.782995    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0274e7fb-2731-44a2-8f19-2670eb16afca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.783506 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783011    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-ovnkube-script-lib\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.783506 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783030    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d274725b-319d-4396-8035-c63e2f6a13a7-cni-binary-copy\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.783506 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783036    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-cni-bin\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.783506 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783059    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-etc-openvswitch\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.783506 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783097    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0274e7fb-2731-44a2-8f19-2670eb16afca-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.783506 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783125    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-system-cni-dir\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.783506 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783129    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0274e7fb-2731-44a2-8f19-2670eb16afca-system-cni-dir\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.784538 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783151    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-multus-socket-dir-parent\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.784538 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783186    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-host-var-lib-cni-bin\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.784538 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783211    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-run-netns\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.784538 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783240    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-env-overrides\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.784538 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783253    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-multus-conf-dir\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.784538 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783270    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dbe2052d-5694-4782-815b-0975eced80b8-konnectivity-ca\") pod \"konnectivity-agent-v4zkn\" (UID: \"dbe2052d-5694-4782-815b-0975eced80b8\") " pod="kube-system/konnectivity-agent-v4zkn"
Sep 11 17:54:04.784538 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783295    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0274e7fb-2731-44a2-8f19-2670eb16afca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.784538 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783337    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-kubelet\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.784538 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783347    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d274725b-319d-4396-8035-c63e2f6a13a7-multus-daemon-config\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.784538 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783366    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.784538 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783416    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-multus-socket-dir-parent\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.784538 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783438    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.784538 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783464    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0274e7fb-2731-44a2-8f19-2670eb16afca-cnibin\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.784538 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783482    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-host-var-lib-cni-bin\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.784538 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783526    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-run-netns\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.784538 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783181    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-host-var-lib-cni-multus\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.784538 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783611    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0274e7fb-2731-44a2-8f19-2670eb16afca-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.785459 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783615    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-ovnkube-config\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.785459 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783628    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-run-ovn-kubernetes\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.785459 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783640    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0274e7fb-2731-44a2-8f19-2670eb16afca-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.785459 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783657    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-etc-openvswitch\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.785459 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783667    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-run-openvswitch\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.785459 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783719    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d274725b-319d-4396-8035-c63e2f6a13a7-system-cni-dir\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.785459 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783765    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-host-kubelet\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.785459 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.783989    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-env-overrides\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.785459 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.784108    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d274725b-319d-4396-8035-c63e2f6a13a7-cni-binary-copy\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.785459 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.784223    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0274e7fb-2731-44a2-8f19-2670eb16afca-cni-binary-copy\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.785459 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.784272    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dbe2052d-5694-4782-815b-0975eced80b8-konnectivity-ca\") pod \"konnectivity-agent-v4zkn\" (UID: \"dbe2052d-5694-4782-815b-0975eced80b8\") " pod="kube-system/konnectivity-agent-v4zkn"
Sep 11 17:54:04.785459 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.784331    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0274e7fb-2731-44a2-8f19-2670eb16afca-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.785459 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.784933    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dbe2052d-5694-4782-815b-0975eced80b8-agent-certs\") pod \"konnectivity-agent-v4zkn\" (UID: \"dbe2052d-5694-4782-815b-0975eced80b8\") " pod="kube-system/konnectivity-agent-v4zkn"
Sep 11 17:54:04.786117 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.786051    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-ovn-node-metrics-cert\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.799219 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.799187    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfvjb\" (UniqueName: \"kubernetes.io/projected/8e883fa7-9ffc-42a3-bbe5-4822684e8a4b-kube-api-access-nfvjb\") pod \"ovnkube-node-vw8qd\" (UID: \"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b\") " pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.799559 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.799521    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkq4l\" (UniqueName: \"kubernetes.io/projected/d274725b-319d-4396-8035-c63e2f6a13a7-kube-api-access-qkq4l\") pod \"multus-xxtsv\" (UID: \"d274725b-319d-4396-8035-c63e2f6a13a7\") " pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:04.800738 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.800715    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg8vx\" (UniqueName: \"kubernetes.io/projected/0274e7fb-2731-44a2-8f19-2670eb16afca-kube-api-access-zg8vx\") pod \"multus-additional-cni-plugins-z9p7p\" (UID: \"0274e7fb-2731-44a2-8f19-2670eb16afca\") " pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.867113 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.867078    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz"
Sep 11 17:54:04.877101 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.877080    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7vgrl"
Sep 11 17:54:04.887953 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.887930    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pjbh8"
Sep 11 17:54:04.890006 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.889986    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 17:54:04.895940 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.895919    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fsfql"
Sep 11 17:54:04.903911 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.903872    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w5twj"
Sep 11 17:54:04.911988 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.911963    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v4zkn"
Sep 11 17:54:04.919950 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.919917    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:04.928992 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.928846    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z9p7p"
Sep 11 17:54:04.934702 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:04.934683    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xxtsv"
Sep 11 17:54:05.186887 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:05.186798    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs\") pod \"network-metrics-daemon-8xg8d\" (UID: \"8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf\") " pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:05.186887 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:05.186844    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret\") pod \"global-pull-secret-syncer-r7cvn\" (UID: \"c5702497-d40f-4a13-b5b8-0be72c784011\") " pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:05.187054 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:05.186942    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:05.187054 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:05.186970    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:05.187054 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:05.187007    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs podName:8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf nodeName:}" failed. No retries permitted until 2026-09-11 17:54:06.186989369 +0000 UTC m=+4.030224792 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs") pod "network-metrics-daemon-8xg8d" (UID: "8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:05.187054 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:05.187023    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret podName:c5702497-d40f-4a13-b5b8-0be72c784011 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:06.187016311 +0000 UTC m=+4.030251732 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret") pod "global-pull-secret-syncer-r7cvn" (UID: "c5702497-d40f-4a13-b5b8-0be72c784011") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:05.287178 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:05.287147    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb5bc\" (UniqueName: \"kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc\") pod \"network-check-target-xfh2z\" (UID: \"2c3f9d2a-8267-48f4-9c2e-d06ff6f15485\") " pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:05.287367 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:05.287349    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:54:05.287367 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:05.287366    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:54:05.287471 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:05.287381    2560 projected.go:196] Error preparing data for projected volume kube-api-access-zb5bc for pod openshift-network-diagnostics/network-check-target-xfh2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:05.287471 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:05.287442    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc podName:2c3f9d2a-8267-48f4-9c2e-d06ff6f15485 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:06.287422479 +0000 UTC m=+4.130657902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zb5bc" (UniqueName: "kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc") pod "network-check-target-xfh2z" (UID: "2c3f9d2a-8267-48f4-9c2e-d06ff6f15485") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:05.660076 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:05.659998    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 17:49:05 +0000 UTC" deadline="2028-04-07 22:56:48.893963835 +0000 UTC"
Sep 11 17:54:05.660076 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:05.660077    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13781h2m43.23389644s"
Sep 11 17:54:05.924244 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:05.924210    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd274725b_319d_4396_8035_c63e2f6a13a7.slice/crio-35b3255f46fe957e2e3b7510110025e9cc2c9234c535a84cccc58650990a3e02 WatchSource:0}: Error finding container 35b3255f46fe957e2e3b7510110025e9cc2c9234c535a84cccc58650990a3e02: Status 404 returned error can't find the container with id 35b3255f46fe957e2e3b7510110025e9cc2c9234c535a84cccc58650990a3e02
Sep 11 17:54:05.926301 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:05.926194    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8970f95d_bfe6_4d2c_97ea_be9489333243.slice/crio-0d8f6aaf30b21c3b8c0292ad53d6f675f74773db0e61535e46c4c42da8c70f2a WatchSource:0}: Error finding container 0d8f6aaf30b21c3b8c0292ad53d6f675f74773db0e61535e46c4c42da8c70f2a: Status 404 returned error can't find the container with id 0d8f6aaf30b21c3b8c0292ad53d6f675f74773db0e61535e46c4c42da8c70f2a
Sep 11 17:54:05.928649 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:05.928495    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0274e7fb_2731_44a2_8f19_2670eb16afca.slice/crio-eaf8a3d53e613ebff3427efdce1f98fe4f83cbb66e93f4b72cd733950272c9c8 WatchSource:0}: Error finding container eaf8a3d53e613ebff3427efdce1f98fe4f83cbb66e93f4b72cd733950272c9c8: Status 404 returned error can't find the container with id eaf8a3d53e613ebff3427efdce1f98fe4f83cbb66e93f4b72cd733950272c9c8
Sep 11 17:54:05.933220 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:05.933199    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91dbf02a_fc59_4467_9c9f_aafb5b1594c4.slice/crio-b30f551a514ce60823a28e5d41965bf40ff251974dfc9bdeb5d4624e2f85ccc4 WatchSource:0}: Error finding container b30f551a514ce60823a28e5d41965bf40ff251974dfc9bdeb5d4624e2f85ccc4: Status 404 returned error can't find the container with id b30f551a514ce60823a28e5d41965bf40ff251974dfc9bdeb5d4624e2f85ccc4
Sep 11 17:54:05.934897 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:05.933760    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e883fa7_9ffc_42a3_bbe5_4822684e8a4b.slice/crio-9f895d3a88b0848527f7a9245174f9ac77492094c5a2dbc13dd4abcc4624bfb4 WatchSource:0}: Error finding container 9f895d3a88b0848527f7a9245174f9ac77492094c5a2dbc13dd4abcc4624bfb4: Status 404 returned error can't find the container with id 9f895d3a88b0848527f7a9245174f9ac77492094c5a2dbc13dd4abcc4624bfb4
Sep 11 17:54:05.934897 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:05.934029    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4bd3b89_6b91_420b_aa10_1c5ac23cfabc.slice/crio-53e160a5eb6e11f82655232e40d4b236507a45cba37e2d8545fdfd78731e7a69 WatchSource:0}: Error finding container 53e160a5eb6e11f82655232e40d4b236507a45cba37e2d8545fdfd78731e7a69: Status 404 returned error can't find the container with id 53e160a5eb6e11f82655232e40d4b236507a45cba37e2d8545fdfd78731e7a69
Sep 11 17:54:05.936255 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:05.936232    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6124639f_d16d_443d_9d14_ee9fe666f0d7.slice/crio-b4189d1f2b796c5361ea8f82d57e7347d8c2f2ae126f400a4d0585aa056e0e72 WatchSource:0}: Error finding container b4189d1f2b796c5361ea8f82d57e7347d8c2f2ae126f400a4d0585aa056e0e72: Status 404 returned error can't find the container with id b4189d1f2b796c5361ea8f82d57e7347d8c2f2ae126f400a4d0585aa056e0e72
Sep 11 17:54:05.938759 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:05.937490    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88dccff_c714_4d6d_88e3_0f5e31b51b8f.slice/crio-678e80c29ed1de1f5e9b866b4647b39b0e1fe1a156c40cdac15f0be0f978a75e WatchSource:0}: Error finding container 678e80c29ed1de1f5e9b866b4647b39b0e1fe1a156c40cdac15f0be0f978a75e: Status 404 returned error can't find the container with id 678e80c29ed1de1f5e9b866b4647b39b0e1fe1a156c40cdac15f0be0f978a75e
Sep 11 17:54:05.938759 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:05.938419    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe2052d_5694_4782_815b_0975eced80b8.slice/crio-fcfdbd491ebb12dd842f9e093251661d9cba04ea61d0a114065bbf193818008f WatchSource:0}: Error finding container fcfdbd491ebb12dd842f9e093251661d9cba04ea61d0a114065bbf193818008f: Status 404 returned error can't find the container with id fcfdbd491ebb12dd842f9e093251661d9cba04ea61d0a114065bbf193818008f
Sep 11 17:54:06.194318 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.194237    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs\") pod \"network-metrics-daemon-8xg8d\" (UID: \"8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf\") " pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:06.194318 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.194282    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret\") pod \"global-pull-secret-syncer-r7cvn\" (UID: \"c5702497-d40f-4a13-b5b8-0be72c784011\") " pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:06.194490 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:06.194384    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:06.194490 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:06.194422    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:06.194490 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:06.194453    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs podName:8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf nodeName:}" failed. No retries permitted until 2026-09-11 17:54:08.194438298 +0000 UTC m=+6.037673718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs") pod "network-metrics-daemon-8xg8d" (UID: "8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:06.194490 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:06.194472    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret podName:c5702497-d40f-4a13-b5b8-0be72c784011 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:08.19445975 +0000 UTC m=+6.037695171 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret") pod "global-pull-secret-syncer-r7cvn" (UID: "c5702497-d40f-4a13-b5b8-0be72c784011") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:06.295424 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.295390    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb5bc\" (UniqueName: \"kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc\") pod \"network-check-target-xfh2z\" (UID: \"2c3f9d2a-8267-48f4-9c2e-d06ff6f15485\") " pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:06.295570 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:06.295550    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:54:06.295635 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:06.295572    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:54:06.295635 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:06.295585    2560 projected.go:196] Error preparing data for projected volume kube-api-access-zb5bc for pod openshift-network-diagnostics/network-check-target-xfh2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:06.295740 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:06.295655    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc podName:2c3f9d2a-8267-48f4-9c2e-d06ff6f15485 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:08.295633122 +0000 UTC m=+6.138868543 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zb5bc" (UniqueName: "kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc") pod "network-check-target-xfh2z" (UID: "2c3f9d2a-8267-48f4-9c2e-d06ff6f15485") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:06.661258 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.661210    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 17:49:05 +0000 UTC" deadline="2028-03-23 06:06:04.707023947 +0000 UTC"
Sep 11 17:54:06.661258 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.661255    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13404h11m58.045773114s"
Sep 11 17:54:06.684751 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.684723    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:06.684933 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:06.684879    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfh2z" podUID="2c3f9d2a-8267-48f4-9c2e-d06ff6f15485"
Sep 11 17:54:06.685346 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.685324    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:06.685453 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:06.685424    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7cvn" podUID="c5702497-d40f-4a13-b5b8-0be72c784011"
Sep 11 17:54:06.685522 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.685512    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:06.686040 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:06.686013    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xg8d" podUID="8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf"
Sep 11 17:54:06.717192 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.717165    2560 generic.go:356] "Generic (PLEG): container finished" podID="42daa9f52c2b3824ddc9a288af86477f" containerID="27a8bf3a5aefaf577841e7aa530b0aa09e797731802cdb0cb0c58152591ee2e5" exitCode=0
Sep 11 17:54:06.717476 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.717246    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal" event={"ID":"42daa9f52c2b3824ddc9a288af86477f","Type":"ContainerDied","Data":"27a8bf3a5aefaf577841e7aa530b0aa09e797731802cdb0cb0c58152591ee2e5"}
Sep 11 17:54:06.727696 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.727660    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal" event={"ID":"fdf7e40390d26689763c696c18fb8f21","Type":"ContainerStarted","Data":"279fb1557073dea58acce4d11645b67ff50df3188e4c00d2c912b22f25fb6d5a"}
Sep 11 17:54:06.743951 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.741609    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v4zkn" event={"ID":"dbe2052d-5694-4782-815b-0975eced80b8","Type":"ContainerStarted","Data":"fcfdbd491ebb12dd842f9e093251661d9cba04ea61d0a114065bbf193818008f"}
Sep 11 17:54:06.752197 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.752158    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7vgrl" event={"ID":"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc","Type":"ContainerStarted","Data":"53e160a5eb6e11f82655232e40d4b236507a45cba37e2d8545fdfd78731e7a69"}
Sep 11 17:54:06.758225 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.758198    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9p7p" event={"ID":"0274e7fb-2731-44a2-8f19-2670eb16afca","Type":"ContainerStarted","Data":"eaf8a3d53e613ebff3427efdce1f98fe4f83cbb66e93f4b72cd733950272c9c8"}
Sep 11 17:54:06.772074 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.772048    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz" event={"ID":"a88dccff-c714-4d6d-88e3-0f5e31b51b8f","Type":"ContainerStarted","Data":"678e80c29ed1de1f5e9b866b4647b39b0e1fe1a156c40cdac15f0be0f978a75e"}
Sep 11 17:54:06.777118 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.777096    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w5twj" event={"ID":"6124639f-d16d-443d-9d14-ee9fe666f0d7","Type":"ContainerStarted","Data":"b4189d1f2b796c5361ea8f82d57e7347d8c2f2ae126f400a4d0585aa056e0e72"}
Sep 11 17:54:06.780697 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.780673    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd" event={"ID":"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b","Type":"ContainerStarted","Data":"9f895d3a88b0848527f7a9245174f9ac77492094c5a2dbc13dd4abcc4624bfb4"}
Sep 11 17:54:06.783710 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.783652    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-29.ec2.internal" podStartSLOduration=2.783633699 podStartE2EDuration="2.783633699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:06.783226091 +0000 UTC m=+4.626461538" watchObservedRunningTime="2026-09-11 17:54:06.783633699 +0000 UTC m=+4.626869144"
Sep 11 17:54:06.790293 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.790261    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fsfql" event={"ID":"91dbf02a-fc59-4467-9c9f-aafb5b1594c4","Type":"ContainerStarted","Data":"b30f551a514ce60823a28e5d41965bf40ff251974dfc9bdeb5d4624e2f85ccc4"}
Sep 11 17:54:06.812098 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.812064    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pjbh8" event={"ID":"8970f95d-bfe6-4d2c-97ea-be9489333243","Type":"ContainerStarted","Data":"0d8f6aaf30b21c3b8c0292ad53d6f675f74773db0e61535e46c4c42da8c70f2a"}
Sep 11 17:54:06.816995 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:06.816963    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xxtsv" event={"ID":"d274725b-319d-4396-8035-c63e2f6a13a7","Type":"ContainerStarted","Data":"35b3255f46fe957e2e3b7510110025e9cc2c9234c535a84cccc58650990a3e02"}
Sep 11 17:54:07.839162 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:07.838906    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal" event={"ID":"42daa9f52c2b3824ddc9a288af86477f","Type":"ContainerStarted","Data":"0f7725fdc8b0d22e0f6c7582180197ec84cbd88e9ed4476437da382294f8c392"}
Sep 11 17:54:08.216021 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:08.215986    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs\") pod \"network-metrics-daemon-8xg8d\" (UID: \"8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf\") " pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:08.216295 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:08.216048    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret\") pod \"global-pull-secret-syncer-r7cvn\" (UID: \"c5702497-d40f-4a13-b5b8-0be72c784011\") " pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:08.216295 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:08.216231    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:08.216295 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:08.216297    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret podName:c5702497-d40f-4a13-b5b8-0be72c784011 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:12.216278874 +0000 UTC m=+10.059514311 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret") pod "global-pull-secret-syncer-r7cvn" (UID: "c5702497-d40f-4a13-b5b8-0be72c784011") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:08.216725 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:08.216703    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:08.216725 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:08.216759    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs podName:8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf nodeName:}" failed. No retries permitted until 2026-09-11 17:54:12.216743662 +0000 UTC m=+10.059979084 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs") pod "network-metrics-daemon-8xg8d" (UID: "8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:08.316594 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:08.316549    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb5bc\" (UniqueName: \"kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc\") pod \"network-check-target-xfh2z\" (UID: \"2c3f9d2a-8267-48f4-9c2e-d06ff6f15485\") " pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:08.316795 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:08.316731    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:54:08.316795 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:08.316755    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:54:08.316795 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:08.316769    2560 projected.go:196] Error preparing data for projected volume kube-api-access-zb5bc for pod openshift-network-diagnostics/network-check-target-xfh2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:08.317035 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:08.316859    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc podName:2c3f9d2a-8267-48f4-9c2e-d06ff6f15485 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:12.316835033 +0000 UTC m=+10.160070458 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zb5bc" (UniqueName: "kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc") pod "network-check-target-xfh2z" (UID: "2c3f9d2a-8267-48f4-9c2e-d06ff6f15485") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:08.685090 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:08.684014    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:08.685090 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:08.684173    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xg8d" podUID="8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf"
Sep 11 17:54:08.685090 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:08.684599    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:08.685090 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:08.684705    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfh2z" podUID="2c3f9d2a-8267-48f4-9c2e-d06ff6f15485"
Sep 11 17:54:08.685090 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:08.684774    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:08.685090 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:08.684915    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7cvn" podUID="c5702497-d40f-4a13-b5b8-0be72c784011"
Sep 11 17:54:10.681385 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:10.681350    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:10.681850 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:10.681491    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7cvn" podUID="c5702497-d40f-4a13-b5b8-0be72c784011"
Sep 11 17:54:10.681934 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:10.681911    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:10.682068 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:10.682030    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xg8d" podUID="8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf"
Sep 11 17:54:10.682191 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:10.682176    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:10.682288 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:10.682274    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfh2z" podUID="2c3f9d2a-8267-48f4-9c2e-d06ff6f15485"
Sep 11 17:54:12.248566 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:12.248527    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret\") pod \"global-pull-secret-syncer-r7cvn\" (UID: \"c5702497-d40f-4a13-b5b8-0be72c784011\") " pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:12.249018 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:12.248629    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs\") pod \"network-metrics-daemon-8xg8d\" (UID: \"8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf\") " pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:12.249018 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:12.248793    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:12.249018 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:12.248853    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs podName:8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf nodeName:}" failed. No retries permitted until 2026-09-11 17:54:20.248838053 +0000 UTC m=+18.092073472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs") pod "network-metrics-daemon-8xg8d" (UID: "8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:12.249229 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:12.249208    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:12.249361 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:12.249275    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret podName:c5702497-d40f-4a13-b5b8-0be72c784011 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:20.249246219 +0000 UTC m=+18.092481640 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret") pod "global-pull-secret-syncer-r7cvn" (UID: "c5702497-d40f-4a13-b5b8-0be72c784011") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:12.349628 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:12.349581    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb5bc\" (UniqueName: \"kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc\") pod \"network-check-target-xfh2z\" (UID: \"2c3f9d2a-8267-48f4-9c2e-d06ff6f15485\") " pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:12.349820 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:12.349767    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:54:12.349820 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:12.349794    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:54:12.349820 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:12.349806    2560 projected.go:196] Error preparing data for projected volume kube-api-access-zb5bc for pod openshift-network-diagnostics/network-check-target-xfh2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:12.349923 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:12.349868    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc podName:2c3f9d2a-8267-48f4-9c2e-d06ff6f15485 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:20.349849753 +0000 UTC m=+18.193085175 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zb5bc" (UniqueName: "kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc") pod "network-check-target-xfh2z" (UID: "2c3f9d2a-8267-48f4-9c2e-d06ff6f15485") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:12.682280 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:12.681827    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:12.682280 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:12.681917    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xg8d" podUID="8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf"
Sep 11 17:54:12.682280 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:12.681941    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:12.682280 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:12.682030    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfh2z" podUID="2c3f9d2a-8267-48f4-9c2e-d06ff6f15485"
Sep 11 17:54:12.682280 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:12.682072    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:12.682280 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:12.682121    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7cvn" podUID="c5702497-d40f-4a13-b5b8-0be72c784011"
Sep 11 17:54:14.682940 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:14.682910    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:14.683389 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:14.683028    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7cvn" podUID="c5702497-d40f-4a13-b5b8-0be72c784011"
Sep 11 17:54:14.683475 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:14.683408    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:14.683542 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:14.683519    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xg8d" podUID="8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf"
Sep 11 17:54:14.683598 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:14.683573    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:14.683669 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:14.683644    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfh2z" podUID="2c3f9d2a-8267-48f4-9c2e-d06ff6f15485"
Sep 11 17:54:16.681640 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:16.681603    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:16.682110 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:16.681603    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:16.682110 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:16.681729    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfh2z" podUID="2c3f9d2a-8267-48f4-9c2e-d06ff6f15485"
Sep 11 17:54:16.682110 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:16.681618    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:16.682110 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:16.681818    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xg8d" podUID="8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf"
Sep 11 17:54:16.682110 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:16.681868    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7cvn" podUID="c5702497-d40f-4a13-b5b8-0be72c784011"
Sep 11 17:54:18.681942 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:18.681909    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:18.682360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:18.681909    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:18.682360 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:18.682018    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfh2z" podUID="2c3f9d2a-8267-48f4-9c2e-d06ff6f15485"
Sep 11 17:54:18.682360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:18.681909    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:18.682360 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:18.682129    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xg8d" podUID="8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf"
Sep 11 17:54:18.682360 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:18.682166    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7cvn" podUID="c5702497-d40f-4a13-b5b8-0be72c784011"
Sep 11 17:54:20.307591 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:20.307555    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs\") pod \"network-metrics-daemon-8xg8d\" (UID: \"8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf\") " pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:20.307591 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:20.307594    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret\") pod \"global-pull-secret-syncer-r7cvn\" (UID: \"c5702497-d40f-4a13-b5b8-0be72c784011\") " pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:20.308104 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:20.307704    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:20.308104 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:20.307704    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:20.308104 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:20.307769    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret podName:c5702497-d40f-4a13-b5b8-0be72c784011 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:36.307741032 +0000 UTC m=+34.150976450 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret") pod "global-pull-secret-syncer-r7cvn" (UID: "c5702497-d40f-4a13-b5b8-0be72c784011") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:20.308104 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:20.307782    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs podName:8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf nodeName:}" failed. No retries permitted until 2026-09-11 17:54:36.307775969 +0000 UTC m=+34.151011388 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs") pod "network-metrics-daemon-8xg8d" (UID: "8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:20.408365 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:20.408333    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb5bc\" (UniqueName: \"kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc\") pod \"network-check-target-xfh2z\" (UID: \"2c3f9d2a-8267-48f4-9c2e-d06ff6f15485\") " pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:20.408506 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:20.408455    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:54:20.408506 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:20.408470    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:54:20.408506 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:20.408478    2560 projected.go:196] Error preparing data for projected volume kube-api-access-zb5bc for pod openshift-network-diagnostics/network-check-target-xfh2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:20.408604 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:20.408519    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc podName:2c3f9d2a-8267-48f4-9c2e-d06ff6f15485 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:36.408507343 +0000 UTC m=+34.251742762 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zb5bc" (UniqueName: "kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc") pod "network-check-target-xfh2z" (UID: "2c3f9d2a-8267-48f4-9c2e-d06ff6f15485") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:20.681655 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:20.681564    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:20.681655 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:20.681589    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:20.681655 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:20.681563    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:20.681915 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:20.681699    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7cvn" podUID="c5702497-d40f-4a13-b5b8-0be72c784011"
Sep 11 17:54:20.681915 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:20.681798    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xg8d" podUID="8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf"
Sep 11 17:54:20.681915 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:20.681872    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfh2z" podUID="2c3f9d2a-8267-48f4-9c2e-d06ff6f15485"
Sep 11 17:54:22.681884 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:22.681855    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:22.682274 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:22.681968    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7cvn" podUID="c5702497-d40f-4a13-b5b8-0be72c784011"
Sep 11 17:54:22.682274 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:22.682059    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:22.682274 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:22.682071    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:22.682274 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:22.682183    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xg8d" podUID="8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf"
Sep 11 17:54:22.682274 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:22.682265    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfh2z" podUID="2c3f9d2a-8267-48f4-9c2e-d06ff6f15485"
Sep 11 17:54:24.681313 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.681160    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:24.681945 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.681160    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:24.681945 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:24.681388    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xg8d" podUID="8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf"
Sep 11 17:54:24.681945 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:24.681443    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfh2z" podUID="2c3f9d2a-8267-48f4-9c2e-d06ff6f15485"
Sep 11 17:54:24.681945 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.681220    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:24.681945 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:24.681560    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7cvn" podUID="c5702497-d40f-4a13-b5b8-0be72c784011"
Sep 11 17:54:24.877737 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.877457    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v4zkn" event={"ID":"dbe2052d-5694-4782-815b-0975eced80b8","Type":"ContainerStarted","Data":"41be8d3d987baf613541eecd177691b456e4b758f40594b8125d89ea93b5ac50"}
Sep 11 17:54:24.879221 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.879196    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7vgrl" event={"ID":"b4bd3b89-6b91-420b-aa10-1c5ac23cfabc","Type":"ContainerStarted","Data":"c5de016c27e05d53d9ac2501bc549c48ec72a69886dbbb91524bc48405138e33"}
Sep 11 17:54:24.881085 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.881060    2560 generic.go:356] "Generic (PLEG): container finished" podID="0274e7fb-2731-44a2-8f19-2670eb16afca" containerID="bee62c85db47d09434c3ee24c0d16a83d9586fcf2b0fb96eb9abe6aec606e1c5" exitCode=0
Sep 11 17:54:24.881193 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.881140    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9p7p" event={"ID":"0274e7fb-2731-44a2-8f19-2670eb16afca","Type":"ContainerDied","Data":"bee62c85db47d09434c3ee24c0d16a83d9586fcf2b0fb96eb9abe6aec606e1c5"}
Sep 11 17:54:24.883207 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.883181    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz" event={"ID":"a88dccff-c714-4d6d-88e3-0f5e31b51b8f","Type":"ContainerStarted","Data":"f83c515dc0a72b78b52e1ac946cf9fb9f4b1a5d310423b920ae7782faa7f5a99"}
Sep 11 17:54:24.888895 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.888869    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd" event={"ID":"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b","Type":"ContainerStarted","Data":"fe3dc482adfcbf3476fb5a4e80cf64575737d1d069da62808f5212b7ef8a3109"}
Sep 11 17:54:24.888993 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.888939    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd" event={"ID":"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b","Type":"ContainerStarted","Data":"b9a3e3096ffd95a1967f2a960b162e78329c9ea26baf3ae39a318d6f1174a06d"}
Sep 11 17:54:24.889055 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.888991    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd" event={"ID":"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b","Type":"ContainerStarted","Data":"754024d2da91ec7b411254895333acf79308d1556ffc0f5870691cbe32744b2a"}
Sep 11 17:54:24.889055 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.889039    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd" event={"ID":"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b","Type":"ContainerStarted","Data":"5e65b6f9b3aa04ef995a996d6d3effadaecc93a4abce32458efa3d205b56a462"}
Sep 11 17:54:24.889153 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.889086    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd" event={"ID":"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b","Type":"ContainerStarted","Data":"eaf9871fcd0164d66463406bf60d6a4baa2ebb05ab77b645ffed163f0368c0f2"}
Sep 11 17:54:24.889153 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.889131    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd" event={"ID":"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b","Type":"ContainerStarted","Data":"77f5b0d0e1d992f14935a2851babc284a4fad126732f1f6134fcc02c1cce7f6e"}
Sep 11 17:54:24.890698 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.890677    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fsfql" event={"ID":"91dbf02a-fc59-4467-9c9f-aafb5b1594c4","Type":"ContainerStarted","Data":"b79e562a0f30495b8c8158f008e987b811d3bb04c2f5c1eb85fa5ee0576f7a07"}
Sep 11 17:54:24.892241 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.892215    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pjbh8" event={"ID":"8970f95d-bfe6-4d2c-97ea-be9489333243","Type":"ContainerStarted","Data":"fdf08ed643258844c880845dbced46d0d7d9b21230666e7f8f21418aafc3b97c"}
Sep 11 17:54:24.893784 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.893739    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xxtsv" event={"ID":"d274725b-319d-4396-8035-c63e2f6a13a7","Type":"ContainerStarted","Data":"2d5aaee3c5c4afaf1f9916dc0af39b53dc13f60f41f89fc05768d4db2abc330b"}
Sep 11 17:54:24.896744 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.896699    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-29.ec2.internal" podStartSLOduration=20.896684442 podStartE2EDuration="20.896684442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:07.85703142 +0000 UTC m=+5.700266870" watchObservedRunningTime="2026-09-11 17:54:24.896684442 +0000 UTC m=+22.739919884"
Sep 11 17:54:24.897259 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.897234    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v4zkn" podStartSLOduration=10.955709818999999 podStartE2EDuration="21.897226413s" totalImagesPullingTime="10.941516594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:24.896148246 +0000 UTC m=+22.739383687" watchObservedRunningTime="2026-09-11 17:54:24.897226413 +0000 UTC m=+22.740461852"
Sep 11 17:54:24.920524 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.920343    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7vgrl" podStartSLOduration=3.759142173 podStartE2EDuration="21.920330331s" totalImagesPullingTime="18.161188158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:24.920295917 +0000 UTC m=+22.763531360" watchObservedRunningTime="2026-09-11 17:54:24.920330331 +0000 UTC m=+22.763565772"
Sep 11 17:54:24.956534 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.956496    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pjbh8" podStartSLOduration=3.79094401 podStartE2EDuration="21.956484148s" totalImagesPullingTime="18.165540138s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:24.938596269 +0000 UTC m=+22.781831708" watchObservedRunningTime="2026-09-11 17:54:24.956484148 +0000 UTC m=+22.799719589"
Sep 11 17:54:24.956629 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:24.956556    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fsfql" podStartSLOduration=11.010298988 podStartE2EDuration="21.956551716s" totalImagesPullingTime="10.946252728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:24.956073137 +0000 UTC m=+22.799308577" watchObservedRunningTime="2026-09-11 17:54:24.956551716 +0000 UTC m=+22.799787193"
Sep 11 17:54:25.614557 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:25.614368    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 17:54:25.640637 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:25.640545    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T17:54:25.614552012Z","UUID":"9b71a921-ebef-47c3-8861-045942fb2d59","Handler":null,"Name":"","Endpoint":""}
Sep 11 17:54:25.642203 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:25.642174    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 17:54:25.642300 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:25.642211    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 17:54:25.898188 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:25.898107    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz" event={"ID":"a88dccff-c714-4d6d-88e3-0f5e31b51b8f","Type":"ContainerStarted","Data":"a54a4977ff8c418cb5ffeec4e498fbec21060530509b5791a42d6e15a76d9875"}
Sep 11 17:54:25.899653 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:25.899623    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w5twj" event={"ID":"6124639f-d16d-443d-9d14-ee9fe666f0d7","Type":"ContainerStarted","Data":"67e9bcad0a2a27d314ef475df1ee7b1f82199778bb294f7dd2b86b029ceb725c"}
Sep 11 17:54:25.938022 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:25.937956    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xxtsv" podStartSLOduration=4.706206606 podStartE2EDuration="22.937938371s" totalImagesPullingTime="18.231731765s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:25.021168201 +0000 UTC m=+22.864403640" watchObservedRunningTime="2026-09-11 17:54:25.937938371 +0000 UTC m=+23.781173812"
Sep 11 17:54:25.938175 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:25.938148    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w5twj" podStartSLOduration=4.7823404830000005 podStartE2EDuration="22.938140249s" totalImagesPullingTime="18.155799766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:25.938088154 +0000 UTC m=+23.781323586" watchObservedRunningTime="2026-09-11 17:54:25.938140249 +0000 UTC m=+23.781375690"
Sep 11 17:54:26.681478 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:26.681447    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:26.681690 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:26.681561    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7cvn" podUID="c5702497-d40f-4a13-b5b8-0be72c784011"
Sep 11 17:54:26.681690 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:26.681586    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:26.681690 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:26.681449    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:26.681843 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:26.681694    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfh2z" podUID="2c3f9d2a-8267-48f4-9c2e-d06ff6f15485"
Sep 11 17:54:26.681843 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:26.681804    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xg8d" podUID="8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf"
Sep 11 17:54:27.907100 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:27.907069    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz" event={"ID":"a88dccff-c714-4d6d-88e3-0f5e31b51b8f","Type":"ContainerStarted","Data":"50d006da1ccd4a99d86194cb714178975df17f7d84f2f583177905e958fb7948"}
Sep 11 17:54:27.911012 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:27.910983    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd" event={"ID":"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b","Type":"ContainerStarted","Data":"bc0609d4f6e837666ddb040e9e2291926b86110b801bcc2791bb47ca56fc0144"}
Sep 11 17:54:27.927108 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:27.927061    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l6dtz" podStartSLOduration=4.451478863 podStartE2EDuration="24.927044347s" totalImagesPullingTime="20.475565484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:27.926871365 +0000 UTC m=+25.770106804" watchObservedRunningTime="2026-09-11 17:54:27.927044347 +0000 UTC m=+25.770279788"
Sep 11 17:54:28.681156 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:28.681120    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:28.681350 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:28.681120    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:28.681350 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:28.681254    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfh2z" podUID="2c3f9d2a-8267-48f4-9c2e-d06ff6f15485"
Sep 11 17:54:28.681350 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:28.681341    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xg8d" podUID="8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf"
Sep 11 17:54:28.681509 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:28.681132    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:28.681509 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:28.681431    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7cvn" podUID="c5702497-d40f-4a13-b5b8-0be72c784011"
Sep 11 17:54:29.616475 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:29.616317    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v4zkn"
Sep 11 17:54:29.616955 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:29.616833    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v4zkn"
Sep 11 17:54:29.918235 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:29.918164    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd" event={"ID":"8e883fa7-9ffc-42a3-bbe5-4822684e8a4b","Type":"ContainerStarted","Data":"87879e60da21147692c1cc67c82e01adf600c09f8d55d59f80bd65fc3a078e1e"}
Sep 11 17:54:29.918416 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:29.918400    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:29.918504 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:29.918468    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:29.918553 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:29.918518    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:29.920233 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:29.920208    2560 generic.go:356] "Generic (PLEG): container finished" podID="0274e7fb-2731-44a2-8f19-2670eb16afca" containerID="9abdafd11f1cc3c6b34d1ff7cf9dd004bb54f0981f8094ef4cf83029b225c374" exitCode=0
Sep 11 17:54:29.920339 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:29.920312    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9p7p" event={"ID":"0274e7fb-2731-44a2-8f19-2670eb16afca","Type":"ContainerDied","Data":"9abdafd11f1cc3c6b34d1ff7cf9dd004bb54f0981f8094ef4cf83029b225c374"}
Sep 11 17:54:29.920490 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:29.920457    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v4zkn"
Sep 11 17:54:29.921065 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:29.920947    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v4zkn"
Sep 11 17:54:29.934209 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:29.934188    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:29.934290 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:29.934271    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:54:29.989564 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:29.989523    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd" podStartSLOduration=8.767528755 podStartE2EDuration="26.989510099s" totalImagesPullingTime="18.221981344s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:29.960508067 +0000 UTC m=+27.803743508" watchObservedRunningTime="2026-09-11 17:54:29.989510099 +0000 UTC m=+27.832745831"
Sep 11 17:54:30.681703 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:30.681657    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:30.682565 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:30.681827    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xg8d" podUID="8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf"
Sep 11 17:54:30.682565 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:30.682286    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:30.682565 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:30.682374    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfh2z" podUID="2c3f9d2a-8267-48f4-9c2e-d06ff6f15485"
Sep 11 17:54:30.682565 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:30.682531    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:30.682800 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:30.682633    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7cvn" podUID="c5702497-d40f-4a13-b5b8-0be72c784011"
Sep 11 17:54:31.666494 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:31.666329    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r7cvn"]
Sep 11 17:54:31.666670 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:31.666576    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:31.666782 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:31.666684    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7cvn" podUID="c5702497-d40f-4a13-b5b8-0be72c784011"
Sep 11 17:54:31.669990 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:31.669969    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xfh2z"]
Sep 11 17:54:31.670124 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:31.670053    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:31.670186 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:31.670142    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfh2z" podUID="2c3f9d2a-8267-48f4-9c2e-d06ff6f15485"
Sep 11 17:54:31.670465 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:31.670441    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8xg8d"]
Sep 11 17:54:31.670568 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:31.670517    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:31.670626 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:31.670607    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xg8d" podUID="8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf"
Sep 11 17:54:31.929377 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:31.929342    2560 generic.go:356] "Generic (PLEG): container finished" podID="0274e7fb-2731-44a2-8f19-2670eb16afca" containerID="0887a9ad0cdbc0167feade185c562b1f558b711e4a54cfb41f0f3c386de9190a" exitCode=0
Sep 11 17:54:31.929853 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:31.929431    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9p7p" event={"ID":"0274e7fb-2731-44a2-8f19-2670eb16afca","Type":"ContainerDied","Data":"0887a9ad0cdbc0167feade185c562b1f558b711e4a54cfb41f0f3c386de9190a"}
Sep 11 17:54:32.934257 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:32.934160    2560 generic.go:356] "Generic (PLEG): container finished" podID="0274e7fb-2731-44a2-8f19-2670eb16afca" containerID="9c278934bdc8884a3af9e0863bf268e5e1499ef29103a96da9332b501bca174e" exitCode=0
Sep 11 17:54:32.934257 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:32.934214    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9p7p" event={"ID":"0274e7fb-2731-44a2-8f19-2670eb16afca","Type":"ContainerDied","Data":"9c278934bdc8884a3af9e0863bf268e5e1499ef29103a96da9332b501bca174e"}
Sep 11 17:54:33.681175 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:33.681138    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:33.681354 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:33.681138    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:33.681354 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:33.681260    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7cvn" podUID="c5702497-d40f-4a13-b5b8-0be72c784011"
Sep 11 17:54:33.681354 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:33.681145    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:33.681512 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:33.681363    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xg8d" podUID="8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf"
Sep 11 17:54:33.681568 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:33.681506    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfh2z" podUID="2c3f9d2a-8267-48f4-9c2e-d06ff6f15485"
Sep 11 17:54:35.681111 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:35.681072    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:35.681549 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:35.681080    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:35.681549 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:35.681192    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xfh2z" podUID="2c3f9d2a-8267-48f4-9c2e-d06ff6f15485"
Sep 11 17:54:35.681549 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:35.681222    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:35.681549 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:35.681333    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r7cvn" podUID="c5702497-d40f-4a13-b5b8-0be72c784011"
Sep 11 17:54:35.681549 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:35.681436    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8xg8d" podUID="8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf"
Sep 11 17:54:36.324474 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:36.324439    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs\") pod \"network-metrics-daemon-8xg8d\" (UID: \"8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf\") " pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:36.324633 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:36.324497    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret\") pod \"global-pull-secret-syncer-r7cvn\" (UID: \"c5702497-d40f-4a13-b5b8-0be72c784011\") " pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:36.324633 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:36.324604    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:36.324730 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:36.324681    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs podName:8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf nodeName:}" failed. No retries permitted until 2026-09-11 17:55:08.324661178 +0000 UTC m=+66.167896599 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs") pod "network-metrics-daemon-8xg8d" (UID: "8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 17:54:36.324730 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:36.324610    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:36.324865 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:36.324774    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret podName:c5702497-d40f-4a13-b5b8-0be72c784011 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:08.324760339 +0000 UTC m=+66.167995770 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret") pod "global-pull-secret-syncer-r7cvn" (UID: "c5702497-d40f-4a13-b5b8-0be72c784011") : object "kube-system"/"original-pull-secret" not registered
Sep 11 17:54:36.425283 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:36.425246    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb5bc\" (UniqueName: \"kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc\") pod \"network-check-target-xfh2z\" (UID: \"2c3f9d2a-8267-48f4-9c2e-d06ff6f15485\") " pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:36.425441 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:36.425425    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 17:54:36.425487 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:36.425453    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 17:54:36.425487 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:36.425468    2560 projected.go:196] Error preparing data for projected volume kube-api-access-zb5bc for pod openshift-network-diagnostics/network-check-target-xfh2z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:36.425571 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:36.425524    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc podName:2c3f9d2a-8267-48f4-9c2e-d06ff6f15485 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:08.425509658 +0000 UTC m=+66.268745076 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zb5bc" (UniqueName: "kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc") pod "network-check-target-xfh2z" (UID: "2c3f9d2a-8267-48f4-9c2e-d06ff6f15485") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 17:54:36.953656 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:36.953593    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-29.ec2.internal" event="NodeReady"
Sep 11 17:54:36.954000 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:36.953725    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 17:54:37.005123 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.005097    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-45lfw"]
Sep 11 17:54:37.024924 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.024896    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dfv62"]
Sep 11 17:54:37.025060 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.025004    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-45lfw"
Sep 11 17:54:37.027767 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.027710    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 17:54:37.027881 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.027825    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cj7w5\""
Sep 11 17:54:37.027975 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.027956    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 17:54:37.037013 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.036995    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-45lfw"]
Sep 11 17:54:37.037105 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.037026    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dfv62"]
Sep 11 17:54:37.037105 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.037093    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dfv62"
Sep 11 17:54:37.039980 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.039961    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 17:54:37.040581 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.040565    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 17:54:37.040659 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.040614    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h7qbb\""
Sep 11 17:54:37.040659 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.040643    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 17:54:37.129611 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.129579    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls\") pod \"dns-default-45lfw\" (UID: \"be400dcd-4cff-4bc9-b572-b714103a5c46\") " pod="openshift-dns/dns-default-45lfw"
Sep 11 17:54:37.129611 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.129616    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/be400dcd-4cff-4bc9-b572-b714103a5c46-config-volume\") pod \"dns-default-45lfw\" (UID: \"be400dcd-4cff-4bc9-b572-b714103a5c46\") " pod="openshift-dns/dns-default-45lfw"
Sep 11 17:54:37.129867 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.129639    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3ca3ce1-7383-4588-a129-d69fedda85b3-cert\") pod \"ingress-canary-dfv62\" (UID: \"c3ca3ce1-7383-4588-a129-d69fedda85b3\") " pod="openshift-ingress-canary/ingress-canary-dfv62"
Sep 11 17:54:37.129867 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.129775    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t4tp\" (UniqueName: \"kubernetes.io/projected/c3ca3ce1-7383-4588-a129-d69fedda85b3-kube-api-access-5t4tp\") pod \"ingress-canary-dfv62\" (UID: \"c3ca3ce1-7383-4588-a129-d69fedda85b3\") " pod="openshift-ingress-canary/ingress-canary-dfv62"
Sep 11 17:54:37.129867 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.129841    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be400dcd-4cff-4bc9-b572-b714103a5c46-tmp-dir\") pod \"dns-default-45lfw\" (UID: \"be400dcd-4cff-4bc9-b572-b714103a5c46\") " pod="openshift-dns/dns-default-45lfw"
Sep 11 17:54:37.129978 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.129879    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpt9b\" (UniqueName: \"kubernetes.io/projected/be400dcd-4cff-4bc9-b572-b714103a5c46-kube-api-access-tpt9b\") pod \"dns-default-45lfw\" (UID: \"be400dcd-4cff-4bc9-b572-b714103a5c46\") " pod="openshift-dns/dns-default-45lfw"
Sep 11 17:54:37.230927 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.230896    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls\") pod \"dns-default-45lfw\" (UID: \"be400dcd-4cff-4bc9-b572-b714103a5c46\") " pod="openshift-dns/dns-default-45lfw"
Sep 11 17:54:37.231088 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.230939    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/be400dcd-4cff-4bc9-b572-b714103a5c46-config-volume\") pod \"dns-default-45lfw\" (UID: \"be400dcd-4cff-4bc9-b572-b714103a5c46\") " pod="openshift-dns/dns-default-45lfw"
Sep 11 17:54:37.231088 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.230955    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3ca3ce1-7383-4588-a129-d69fedda85b3-cert\") pod \"ingress-canary-dfv62\" (UID: \"c3ca3ce1-7383-4588-a129-d69fedda85b3\") " pod="openshift-ingress-canary/ingress-canary-dfv62"
Sep 11 17:54:37.231088 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.230989    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t4tp\" (UniqueName: \"kubernetes.io/projected/c3ca3ce1-7383-4588-a129-d69fedda85b3-kube-api-access-5t4tp\") pod \"ingress-canary-dfv62\" (UID: \"c3ca3ce1-7383-4588-a129-d69fedda85b3\") " pod="openshift-ingress-canary/ingress-canary-dfv62"
Sep 11 17:54:37.231088 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.231017    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be400dcd-4cff-4bc9-b572-b714103a5c46-tmp-dir\") pod \"dns-default-45lfw\" (UID: \"be400dcd-4cff-4bc9-b572-b714103a5c46\") " pod="openshift-dns/dns-default-45lfw"
Sep 11 17:54:37.231088 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.231039    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpt9b\" (UniqueName: \"kubernetes.io/projected/be400dcd-4cff-4bc9-b572-b714103a5c46-kube-api-access-tpt9b\") pod \"dns-default-45lfw\" (UID: \"be400dcd-4cff-4bc9-b572-b714103a5c46\") " pod="openshift-dns/dns-default-45lfw"
Sep 11 17:54:37.231088 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:37.231068    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:54:37.231352 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:37.231170    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls podName:be400dcd-4cff-4bc9-b572-b714103a5c46 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:37.731149135 +0000 UTC m=+35.574384614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls") pod "dns-default-45lfw" (UID: "be400dcd-4cff-4bc9-b572-b714103a5c46") : secret "dns-default-metrics-tls" not found
Sep 11 17:54:37.231352 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.231337    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be400dcd-4cff-4bc9-b572-b714103a5c46-tmp-dir\") pod \"dns-default-45lfw\" (UID: \"be400dcd-4cff-4bc9-b572-b714103a5c46\") " pod="openshift-dns/dns-default-45lfw"
Sep 11 17:54:37.231502 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.231480    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/be400dcd-4cff-4bc9-b572-b714103a5c46-config-volume\") pod \"dns-default-45lfw\" (UID: \"be400dcd-4cff-4bc9-b572-b714103a5c46\") " pod="openshift-dns/dns-default-45lfw"
Sep 11 17:54:37.235326 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.235204    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c3ca3ce1-7383-4588-a129-d69fedda85b3-cert\") pod \"ingress-canary-dfv62\" (UID: \"c3ca3ce1-7383-4588-a129-d69fedda85b3\") " pod="openshift-ingress-canary/ingress-canary-dfv62"
Sep 11 17:54:37.240337 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.240310    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t4tp\" (UniqueName: \"kubernetes.io/projected/c3ca3ce1-7383-4588-a129-d69fedda85b3-kube-api-access-5t4tp\") pod \"ingress-canary-dfv62\" (UID: \"c3ca3ce1-7383-4588-a129-d69fedda85b3\") " pod="openshift-ingress-canary/ingress-canary-dfv62"
Sep 11 17:54:37.250078 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.250059    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpt9b\" (UniqueName: \"kubernetes.io/projected/be400dcd-4cff-4bc9-b572-b714103a5c46-kube-api-access-tpt9b\") pod \"dns-default-45lfw\" (UID: \"be400dcd-4cff-4bc9-b572-b714103a5c46\") " pod="openshift-dns/dns-default-45lfw"
Sep 11 17:54:37.344670 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.344631    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dfv62"
Sep 11 17:54:37.647615 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.647584    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dfv62"]
Sep 11 17:54:37.650846 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:37.650815    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ca3ce1_7383_4588_a129_d69fedda85b3.slice/crio-6a8767546bd8a09702d5e0487562301219c5630043ae35516e66838956cf8690 WatchSource:0}: Error finding container 6a8767546bd8a09702d5e0487562301219c5630043ae35516e66838956cf8690: Status 404 returned error can't find the container with id 6a8767546bd8a09702d5e0487562301219c5630043ae35516e66838956cf8690
Sep 11 17:54:37.681545 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.681508    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:54:37.681545 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.681528    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:54:37.681713 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.681512    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:54:37.686444 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.686422    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 17:54:37.686576 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.686466    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vhttf\""
Sep 11 17:54:37.686576 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.686509    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 17:54:37.687000 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.686978    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 17:54:37.687093 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.687057    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-64859\""
Sep 11 17:54:37.687230 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.687217    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 17:54:37.735591 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.735558    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls\") pod \"dns-default-45lfw\" (UID: \"be400dcd-4cff-4bc9-b572-b714103a5c46\") " pod="openshift-dns/dns-default-45lfw"
Sep 11 17:54:37.735745 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:37.735681    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:54:37.735831 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:37.735770    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls podName:be400dcd-4cff-4bc9-b572-b714103a5c46 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:38.735735956 +0000 UTC m=+36.578971388 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls") pod "dns-default-45lfw" (UID: "be400dcd-4cff-4bc9-b572-b714103a5c46") : secret "dns-default-metrics-tls" not found
Sep 11 17:54:37.948473 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.948383    2560 generic.go:356] "Generic (PLEG): container finished" podID="0274e7fb-2731-44a2-8f19-2670eb16afca" containerID="8035e251ee5295fc545459f154dab345b08c84037c95bfefed3e788735742d31" exitCode=0
Sep 11 17:54:37.948646 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.948467    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9p7p" event={"ID":"0274e7fb-2731-44a2-8f19-2670eb16afca","Type":"ContainerDied","Data":"8035e251ee5295fc545459f154dab345b08c84037c95bfefed3e788735742d31"}
Sep 11 17:54:37.949820 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:37.949796    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dfv62" event={"ID":"c3ca3ce1-7383-4588-a129-d69fedda85b3","Type":"ContainerStarted","Data":"6a8767546bd8a09702d5e0487562301219c5630043ae35516e66838956cf8690"}
Sep 11 17:54:38.743391 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:38.743352    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls\") pod \"dns-default-45lfw\" (UID: \"be400dcd-4cff-4bc9-b572-b714103a5c46\") " pod="openshift-dns/dns-default-45lfw"
Sep 11 17:54:38.744145 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:38.743503    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:54:38.744145 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:38.743574    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls podName:be400dcd-4cff-4bc9-b572-b714103a5c46 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:40.743557062 +0000 UTC m=+38.586792499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls") pod "dns-default-45lfw" (UID: "be400dcd-4cff-4bc9-b572-b714103a5c46") : secret "dns-default-metrics-tls" not found
Sep 11 17:54:38.955882 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:38.955850    2560 generic.go:356] "Generic (PLEG): container finished" podID="0274e7fb-2731-44a2-8f19-2670eb16afca" containerID="52fa4af0d31d288491b7e0591870a5d2f407f1d8bc8456f671ff29bfcb36a93c" exitCode=0
Sep 11 17:54:38.956009 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:38.955901    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9p7p" event={"ID":"0274e7fb-2731-44a2-8f19-2670eb16afca","Type":"ContainerDied","Data":"52fa4af0d31d288491b7e0591870a5d2f407f1d8bc8456f671ff29bfcb36a93c"}
Sep 11 17:54:39.961591 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:39.961412    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z9p7p" event={"ID":"0274e7fb-2731-44a2-8f19-2670eb16afca","Type":"ContainerStarted","Data":"1ddcf4dd55af37d64eb4401014faa7fac19f54ac52f4ff7b6cc0ffbe176e572a"}
Sep 11 17:54:39.988987 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:39.988932    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-z9p7p" podStartSLOduration=8.10483307 podStartE2EDuration="36.988915434s" totalImagesPullingTime="28.884082364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:39.987434121 +0000 UTC m=+37.830669561" watchObservedRunningTime="2026-09-11 17:54:39.988915434 +0000 UTC m=+37.832150871"
Sep 11 17:54:40.757866 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:40.757834    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls\") pod \"dns-default-45lfw\" (UID: \"be400dcd-4cff-4bc9-b572-b714103a5c46\") " pod="openshift-dns/dns-default-45lfw"
Sep 11 17:54:40.758037 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:40.757952    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:54:40.758037 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:40.758006    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls podName:be400dcd-4cff-4bc9-b572-b714103a5c46 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:44.75799322 +0000 UTC m=+42.601228638 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls") pod "dns-default-45lfw" (UID: "be400dcd-4cff-4bc9-b572-b714103a5c46") : secret "dns-default-metrics-tls" not found
Sep 11 17:54:40.965324 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:40.965290    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dfv62" event={"ID":"c3ca3ce1-7383-4588-a129-d69fedda85b3","Type":"ContainerStarted","Data":"daf30c1c693e65d9ad39637232fb8c49380304bda816f423d22179ced7bebea9"}
Sep 11 17:54:40.982589 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:40.982532    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dfv62" podStartSLOduration=2.564317784 podStartE2EDuration="4.982511701s" totalImagesPullingTime="2.418193917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:40.981454099 +0000 UTC m=+38.824689577" watchObservedRunningTime="2026-09-11 17:54:40.982511701 +0000 UTC m=+38.825747140"
Sep 11 17:54:44.785488 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:44.785448    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls\") pod \"dns-default-45lfw\" (UID: \"be400dcd-4cff-4bc9-b572-b714103a5c46\") " pod="openshift-dns/dns-default-45lfw"
Sep 11 17:54:44.785871 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:44.785600    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:54:44.785871 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:44.785670    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls podName:be400dcd-4cff-4bc9-b572-b714103a5c46 nodeName:}" failed. No retries permitted until 2026-09-11 17:54:52.785653398 +0000 UTC m=+50.628888822 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls") pod "dns-default-45lfw" (UID: "be400dcd-4cff-4bc9-b572-b714103a5c46") : secret "dns-default-metrics-tls" not found
Sep 11 17:54:52.838292 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:52.838255    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls\") pod \"dns-default-45lfw\" (UID: \"be400dcd-4cff-4bc9-b572-b714103a5c46\") " pod="openshift-dns/dns-default-45lfw"
Sep 11 17:54:52.838690 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:52.838363    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:54:52.838690 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:52.838426    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls podName:be400dcd-4cff-4bc9-b572-b714103a5c46 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:08.838412173 +0000 UTC m=+66.681647593 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls") pod "dns-default-45lfw" (UID: "be400dcd-4cff-4bc9-b572-b714103a5c46") : secret "dns-default-metrics-tls" not found
Sep 11 17:54:53.790900 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:53.790866    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4"]
Sep 11 17:54:53.812688 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:53.812663    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4"]
Sep 11 17:54:53.812847 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:53.812778    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4"
Sep 11 17:54:53.816874 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:53.816851    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 11 17:54:53.817013 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:53.816984    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 17:54:53.818057 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:53.818035    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m8zdr\""
Sep 11 17:54:53.818237 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:53.818136    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 11 17:54:53.818312 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:53.818130    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 17:54:53.895971 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:53.895939    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-26crr"]
Sep 11 17:54:53.916547 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:53.916523    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-26crr"]
Sep 11 17:54:53.916693 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:53.916626    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-26crr"
Sep 11 17:54:53.919225 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:53.919200    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 11 17:54:53.921442 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:53.919421    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 11 17:54:53.921442 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:53.919566    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 17:54:53.921987 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:53.921967    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 17:54:53.922059 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:53.921967    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-j4q5p\""
Sep 11 17:54:53.925147 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:53.925131    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 11 17:54:53.945762 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:53.945721    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e99b97af-cf7c-4b4e-8fac-374449322c1e-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-g6dh4\" (UID: \"e99b97af-cf7c-4b4e-8fac-374449322c1e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4"
Sep 11 17:54:53.945869 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:53.945770    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl7jz\" (UniqueName: \"kubernetes.io/projected/e99b97af-cf7c-4b4e-8fac-374449322c1e-kube-api-access-dl7jz\") pod \"cluster-monitoring-operator-6cd954b897-g6dh4\" (UID: \"e99b97af-cf7c-4b4e-8fac-374449322c1e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4"
Sep 11 17:54:53.945869 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:53.945825    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-g6dh4\" (UID: \"e99b97af-cf7c-4b4e-8fac-374449322c1e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4"
Sep 11 17:54:54.046355 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.046271    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05d87087-b55d-48cd-863f-1b1e6e7e4e72-serving-cert\") pod \"insights-operator-5844b856db-26crr\" (UID: \"05d87087-b55d-48cd-863f-1b1e6e7e4e72\") " pod="openshift-insights/insights-operator-5844b856db-26crr"
Sep 11 17:54:54.046355 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.046316    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05d87087-b55d-48cd-863f-1b1e6e7e4e72-service-ca-bundle\") pod \"insights-operator-5844b856db-26crr\" (UID: \"05d87087-b55d-48cd-863f-1b1e6e7e4e72\") " pod="openshift-insights/insights-operator-5844b856db-26crr"
Sep 11 17:54:54.046570 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.046388    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e99b97af-cf7c-4b4e-8fac-374449322c1e-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-g6dh4\" (UID: \"e99b97af-cf7c-4b4e-8fac-374449322c1e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4"
Sep 11 17:54:54.046570 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.046405    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05d87087-b55d-48cd-863f-1b1e6e7e4e72-trusted-ca-bundle\") pod \"insights-operator-5844b856db-26crr\" (UID: \"05d87087-b55d-48cd-863f-1b1e6e7e4e72\") " pod="openshift-insights/insights-operator-5844b856db-26crr"
Sep 11 17:54:54.046570 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.046441    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl7jz\" (UniqueName: \"kubernetes.io/projected/e99b97af-cf7c-4b4e-8fac-374449322c1e-kube-api-access-dl7jz\") pod \"cluster-monitoring-operator-6cd954b897-g6dh4\" (UID: \"e99b97af-cf7c-4b4e-8fac-374449322c1e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4"
Sep 11 17:54:54.046570 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.046491    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czmws\" (UniqueName: \"kubernetes.io/projected/05d87087-b55d-48cd-863f-1b1e6e7e4e72-kube-api-access-czmws\") pod \"insights-operator-5844b856db-26crr\" (UID: \"05d87087-b55d-48cd-863f-1b1e6e7e4e72\") " pod="openshift-insights/insights-operator-5844b856db-26crr"
Sep 11 17:54:54.046570 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.046517    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-g6dh4\" (UID: \"e99b97af-cf7c-4b4e-8fac-374449322c1e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4"
Sep 11 17:54:54.046570 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.046533    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05d87087-b55d-48cd-863f-1b1e6e7e4e72-tmp\") pod \"insights-operator-5844b856db-26crr\" (UID: \"05d87087-b55d-48cd-863f-1b1e6e7e4e72\") " pod="openshift-insights/insights-operator-5844b856db-26crr"
Sep 11 17:54:54.046570 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.046573    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/05d87087-b55d-48cd-863f-1b1e6e7e4e72-snapshots\") pod \"insights-operator-5844b856db-26crr\" (UID: \"05d87087-b55d-48cd-863f-1b1e6e7e4e72\") " pod="openshift-insights/insights-operator-5844b856db-26crr"
Sep 11 17:54:54.046959 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:54.046661    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 17:54:54.046959 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:54.046744    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls podName:e99b97af-cf7c-4b4e-8fac-374449322c1e nodeName:}" failed. No retries permitted until 2026-09-11 17:54:54.546723326 +0000 UTC m=+52.389958749 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-g6dh4" (UID: "e99b97af-cf7c-4b4e-8fac-374449322c1e") : secret "cluster-monitoring-operator-tls" not found
Sep 11 17:54:54.047250 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.047227    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e99b97af-cf7c-4b4e-8fac-374449322c1e-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-g6dh4\" (UID: \"e99b97af-cf7c-4b4e-8fac-374449322c1e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4"
Sep 11 17:54:54.056725 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.056700    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl7jz\" (UniqueName: \"kubernetes.io/projected/e99b97af-cf7c-4b4e-8fac-374449322c1e-kube-api-access-dl7jz\") pod \"cluster-monitoring-operator-6cd954b897-g6dh4\" (UID: \"e99b97af-cf7c-4b4e-8fac-374449322c1e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4"
Sep 11 17:54:54.147006 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.146966    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05d87087-b55d-48cd-863f-1b1e6e7e4e72-service-ca-bundle\") pod \"insights-operator-5844b856db-26crr\" (UID: \"05d87087-b55d-48cd-863f-1b1e6e7e4e72\") " pod="openshift-insights/insights-operator-5844b856db-26crr"
Sep 11 17:54:54.147150 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.147035    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05d87087-b55d-48cd-863f-1b1e6e7e4e72-trusted-ca-bundle\") pod \"insights-operator-5844b856db-26crr\" (UID: \"05d87087-b55d-48cd-863f-1b1e6e7e4e72\") " pod="openshift-insights/insights-operator-5844b856db-26crr"
Sep 11 17:54:54.147150 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.147057    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czmws\" (UniqueName: \"kubernetes.io/projected/05d87087-b55d-48cd-863f-1b1e6e7e4e72-kube-api-access-czmws\") pod \"insights-operator-5844b856db-26crr\" (UID: \"05d87087-b55d-48cd-863f-1b1e6e7e4e72\") " pod="openshift-insights/insights-operator-5844b856db-26crr"
Sep 11 17:54:54.147150 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.147084    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05d87087-b55d-48cd-863f-1b1e6e7e4e72-tmp\") pod \"insights-operator-5844b856db-26crr\" (UID: \"05d87087-b55d-48cd-863f-1b1e6e7e4e72\") " pod="openshift-insights/insights-operator-5844b856db-26crr"
Sep 11 17:54:54.147150 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.147109    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/05d87087-b55d-48cd-863f-1b1e6e7e4e72-snapshots\") pod \"insights-operator-5844b856db-26crr\" (UID: \"05d87087-b55d-48cd-863f-1b1e6e7e4e72\") " pod="openshift-insights/insights-operator-5844b856db-26crr"
Sep 11 17:54:54.147150 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.147129    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05d87087-b55d-48cd-863f-1b1e6e7e4e72-serving-cert\") pod \"insights-operator-5844b856db-26crr\" (UID: \"05d87087-b55d-48cd-863f-1b1e6e7e4e72\") " pod="openshift-insights/insights-operator-5844b856db-26crr"
Sep 11 17:54:54.147672 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.147643    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05d87087-b55d-48cd-863f-1b1e6e7e4e72-tmp\") pod \"insights-operator-5844b856db-26crr\" (UID: \"05d87087-b55d-48cd-863f-1b1e6e7e4e72\") " pod="openshift-insights/insights-operator-5844b856db-26crr"
Sep 11 17:54:54.147733 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.147695    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05d87087-b55d-48cd-863f-1b1e6e7e4e72-service-ca-bundle\") pod \"insights-operator-5844b856db-26crr\" (UID: \"05d87087-b55d-48cd-863f-1b1e6e7e4e72\") " pod="openshift-insights/insights-operator-5844b856db-26crr"
Sep 11 17:54:54.147820 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.147699    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/05d87087-b55d-48cd-863f-1b1e6e7e4e72-snapshots\") pod \"insights-operator-5844b856db-26crr\" (UID: \"05d87087-b55d-48cd-863f-1b1e6e7e4e72\") " pod="openshift-insights/insights-operator-5844b856db-26crr"
Sep 11 17:54:54.147988 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.147933    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05d87087-b55d-48cd-863f-1b1e6e7e4e72-trusted-ca-bundle\") pod \"insights-operator-5844b856db-26crr\" (UID: \"05d87087-b55d-48cd-863f-1b1e6e7e4e72\") " pod="openshift-insights/insights-operator-5844b856db-26crr"
Sep 11 17:54:54.149559 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.149539    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05d87087-b55d-48cd-863f-1b1e6e7e4e72-serving-cert\") pod \"insights-operator-5844b856db-26crr\" (UID: \"05d87087-b55d-48cd-863f-1b1e6e7e4e72\") " pod="openshift-insights/insights-operator-5844b856db-26crr"
Sep 11 17:54:54.156083 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.156064    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czmws\" (UniqueName: \"kubernetes.io/projected/05d87087-b55d-48cd-863f-1b1e6e7e4e72-kube-api-access-czmws\") pod \"insights-operator-5844b856db-26crr\" (UID: \"05d87087-b55d-48cd-863f-1b1e6e7e4e72\") " pod="openshift-insights/insights-operator-5844b856db-26crr"
Sep 11 17:54:54.227009 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.226973    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-26crr"
Sep 11 17:54:54.497195 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.497164    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-26crr"]
Sep 11 17:54:54.499728 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:54:54.499702    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d87087_b55d_48cd_863f_1b1e6e7e4e72.slice/crio-532bfbd5dc60cceba00b55f25d503bca4cc5d9ce9fddb2959bc0980bd2b05f09 WatchSource:0}: Error finding container 532bfbd5dc60cceba00b55f25d503bca4cc5d9ce9fddb2959bc0980bd2b05f09: Status 404 returned error can't find the container with id 532bfbd5dc60cceba00b55f25d503bca4cc5d9ce9fddb2959bc0980bd2b05f09
Sep 11 17:54:54.549656 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.549626    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-g6dh4\" (UID: \"e99b97af-cf7c-4b4e-8fac-374449322c1e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4"
Sep 11 17:54:54.549779 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:54.549730    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 17:54:54.549822 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:54.549805    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls podName:e99b97af-cf7c-4b4e-8fac-374449322c1e nodeName:}" failed. No retries permitted until 2026-09-11 17:54:55.549786991 +0000 UTC m=+53.393022411 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-g6dh4" (UID: "e99b97af-cf7c-4b4e-8fac-374449322c1e") : secret "cluster-monitoring-operator-tls" not found
Sep 11 17:54:54.997729 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:54.997699    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-26crr" event={"ID":"05d87087-b55d-48cd-863f-1b1e6e7e4e72","Type":"ContainerStarted","Data":"532bfbd5dc60cceba00b55f25d503bca4cc5d9ce9fddb2959bc0980bd2b05f09"}
Sep 11 17:54:55.557559 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:55.557523    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-g6dh4\" (UID: \"e99b97af-cf7c-4b4e-8fac-374449322c1e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4"
Sep 11 17:54:55.557826 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:55.557678    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 17:54:55.557826 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:55.557741    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls podName:e99b97af-cf7c-4b4e-8fac-374449322c1e nodeName:}" failed. No retries permitted until 2026-09-11 17:54:57.557727011 +0000 UTC m=+55.400962430 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-g6dh4" (UID: "e99b97af-cf7c-4b4e-8fac-374449322c1e") : secret "cluster-monitoring-operator-tls" not found
Sep 11 17:54:57.004572 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.004538    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-26crr" event={"ID":"05d87087-b55d-48cd-863f-1b1e6e7e4e72","Type":"ContainerStarted","Data":"b162a5bcfe4002bd01aaa754106ba2800588023035dfed5de06c70e7f7a4731f"}
Sep 11 17:54:57.028956 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.028901    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-26crr" podStartSLOduration=2.236800976 podStartE2EDuration="4.028884108s" totalImagesPullingTime="1.792083132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:54:57.028366709 +0000 UTC m=+54.871602152" watchObservedRunningTime="2026-09-11 17:54:57.028884108 +0000 UTC m=+54.872119548"
Sep 11 17:54:57.572571 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.572530    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-g6dh4\" (UID: \"e99b97af-cf7c-4b4e-8fac-374449322c1e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4"
Sep 11 17:54:57.572784 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:57.572728    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 17:54:57.572866 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:57.572853    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls podName:e99b97af-cf7c-4b4e-8fac-374449322c1e nodeName:}" failed. No retries permitted until 2026-09-11 17:55:01.572830235 +0000 UTC m=+59.416065658 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-g6dh4" (UID: "e99b97af-cf7c-4b4e-8fac-374449322c1e") : secret "cluster-monitoring-operator-tls" not found
Sep 11 17:54:57.649692 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.649663    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-chg9t"]
Sep 11 17:54:57.651847 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.651821    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:54:57.654717 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.654695    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 17:54:57.654865 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.654841    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ntvnk\""
Sep 11 17:54:57.654928 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.654847    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 17:54:57.666112 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.666089    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-chg9t"]
Sep 11 17:54:57.774560 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.774520    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-data-volume\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:54:57.774560 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.774565    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flcl5\" (UniqueName: \"kubernetes.io/projected/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-api-access-flcl5\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:54:57.774790 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.774649    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-crio-socket\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:54:57.774790 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.774723    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:54:57.774873 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.774789    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:54:57.875354 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.875262    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:54:57.875354 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.875324    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-data-volume\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:54:57.875354 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.875345    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flcl5\" (UniqueName: \"kubernetes.io/projected/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-api-access-flcl5\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:54:57.875578 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.875378    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-crio-socket\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:54:57.875578 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:57.875408    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:54:57.875578 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.875420    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:54:57.875578 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:57.875506    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-rbac-proxy-cm podName:025d8e60-0c15-47a4-bd38-7c29f2e30f9e nodeName:}" failed. No retries permitted until 2026-09-11 17:54:58.375484587 +0000 UTC m=+56.218720022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-chg9t" (UID: "025d8e60-0c15-47a4-bd38-7c29f2e30f9e") : configmap "kube-rbac-proxy" not found
Sep 11 17:54:57.875578 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.875505    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-crio-socket\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:54:57.875578 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:57.875523    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:54:57.875578 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:57.875571    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls podName:025d8e60-0c15-47a4-bd38-7c29f2e30f9e nodeName:}" failed. No retries permitted until 2026-09-11 17:54:58.375556656 +0000 UTC m=+56.218792090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-chg9t" (UID: "025d8e60-0c15-47a4-bd38-7c29f2e30f9e") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:54:57.875887 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.875802    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-data-volume\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:54:57.885126 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:57.885102    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flcl5\" (UniqueName: \"kubernetes.io/projected/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-api-access-flcl5\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:54:58.379598 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:58.379554    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:54:58.379598 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:58.379600    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:54:58.380048 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:58.379705    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:54:58.380048 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:58.379717    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:54:58.380048 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:58.379798    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls podName:025d8e60-0c15-47a4-bd38-7c29f2e30f9e nodeName:}" failed. No retries permitted until 2026-09-11 17:54:59.379778373 +0000 UTC m=+57.223013792 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-chg9t" (UID: "025d8e60-0c15-47a4-bd38-7c29f2e30f9e") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:54:58.380048 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:58.379816    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-rbac-proxy-cm podName:025d8e60-0c15-47a4-bd38-7c29f2e30f9e nodeName:}" failed. No retries permitted until 2026-09-11 17:54:59.3798083 +0000 UTC m=+57.223043719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-chg9t" (UID: "025d8e60-0c15-47a4-bd38-7c29f2e30f9e") : configmap "kube-rbac-proxy" not found
Sep 11 17:54:59.386147 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:59.386104    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:54:59.386623 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:59.386158    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:54:59.386623 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:59.386263    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:54:59.386623 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:59.386276    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:54:59.386623 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:59.386351    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-rbac-proxy-cm podName:025d8e60-0c15-47a4-bd38-7c29f2e30f9e nodeName:}" failed. No retries permitted until 2026-09-11 17:55:01.386328101 +0000 UTC m=+59.229563523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-chg9t" (UID: "025d8e60-0c15-47a4-bd38-7c29f2e30f9e") : configmap "kube-rbac-proxy" not found
Sep 11 17:54:59.386623 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:54:59.386371    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls podName:025d8e60-0c15-47a4-bd38-7c29f2e30f9e nodeName:}" failed. No retries permitted until 2026-09-11 17:55:01.386357301 +0000 UTC m=+59.229592720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-chg9t" (UID: "025d8e60-0c15-47a4-bd38-7c29f2e30f9e") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:54:59.612531 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:54:59.612500    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pjbh8_8970f95d-bfe6-4d2c-97ea-be9489333243/dns-node-resolver/0.log"
Sep 11 17:55:00.212787 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:00.212743    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fsfql_91dbf02a-fc59-4467-9c9f-aafb5b1594c4/node-ca/0.log"
Sep 11 17:55:00.798533 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:00.798495    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2tttk"]
Sep 11 17:55:00.825472 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:00.825435    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2tttk"]
Sep 11 17:55:00.825614 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:00.825546    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2tttk"
Sep 11 17:55:00.828356 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:00.828332    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:55:00.828356 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:00.828349    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 11 17:55:00.829547 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:00.829530    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-csl79\""
Sep 11 17:55:00.899000 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:00.898970    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqx5c"]
Sep 11 17:55:00.914784 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:00.914758    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqx5c"]
Sep 11 17:55:00.914954 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:00.914852    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqx5c"
Sep 11 17:55:00.917538 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:00.917515    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 11 17:55:00.917538 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:00.917534    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:55:00.917721 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:00.917605    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dd6gq\""
Sep 11 17:55:00.917721 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:00.917661    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 11 17:55:00.998681 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:00.998645    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw4ts\" (UniqueName: \"kubernetes.io/projected/4d451b56-86ce-460a-813f-054ec25c4e8b-kube-api-access-tw4ts\") pod \"volume-data-source-validator-86695b7cc7-2tttk\" (UID: \"4d451b56-86ce-460a-813f-054ec25c4e8b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2tttk"
Sep 11 17:55:01.099334 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.099252    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e53386e8-153f-4423-ad30-0486d90e094d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kqx5c\" (UID: \"e53386e8-153f-4423-ad30-0486d90e094d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqx5c"
Sep 11 17:55:01.099476 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.099373    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw4ts\" (UniqueName: \"kubernetes.io/projected/4d451b56-86ce-460a-813f-054ec25c4e8b-kube-api-access-tw4ts\") pod \"volume-data-source-validator-86695b7cc7-2tttk\" (UID: \"4d451b56-86ce-460a-813f-054ec25c4e8b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2tttk"
Sep 11 17:55:01.099476 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.099404    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52hls\" (UniqueName: \"kubernetes.io/projected/e53386e8-153f-4423-ad30-0486d90e094d-kube-api-access-52hls\") pod \"cluster-samples-operator-86499c89c4-kqx5c\" (UID: \"e53386e8-153f-4423-ad30-0486d90e094d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqx5c"
Sep 11 17:55:01.108136 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.108109    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw4ts\" (UniqueName: \"kubernetes.io/projected/4d451b56-86ce-460a-813f-054ec25c4e8b-kube-api-access-tw4ts\") pod \"volume-data-source-validator-86695b7cc7-2tttk\" (UID: \"4d451b56-86ce-460a-813f-054ec25c4e8b\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2tttk"
Sep 11 17:55:01.133913 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.133887    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2tttk"
Sep 11 17:55:01.200860 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.200832    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e53386e8-153f-4423-ad30-0486d90e094d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kqx5c\" (UID: \"e53386e8-153f-4423-ad30-0486d90e094d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqx5c"
Sep 11 17:55:01.201023 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.200912    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52hls\" (UniqueName: \"kubernetes.io/projected/e53386e8-153f-4423-ad30-0486d90e094d-kube-api-access-52hls\") pod \"cluster-samples-operator-86499c89c4-kqx5c\" (UID: \"e53386e8-153f-4423-ad30-0486d90e094d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqx5c"
Sep 11 17:55:01.201092 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:01.201036    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 17:55:01.201152 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:01.201128    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e53386e8-153f-4423-ad30-0486d90e094d-samples-operator-tls podName:e53386e8-153f-4423-ad30-0486d90e094d nodeName:}" failed. No retries permitted until 2026-09-11 17:55:01.701106664 +0000 UTC m=+59.544342082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e53386e8-153f-4423-ad30-0486d90e094d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kqx5c" (UID: "e53386e8-153f-4423-ad30-0486d90e094d") : secret "samples-operator-tls" not found
Sep 11 17:55:01.210016 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.209985    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52hls\" (UniqueName: \"kubernetes.io/projected/e53386e8-153f-4423-ad30-0486d90e094d-kube-api-access-52hls\") pod \"cluster-samples-operator-86499c89c4-kqx5c\" (UID: \"e53386e8-153f-4423-ad30-0486d90e094d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqx5c"
Sep 11 17:55:01.212185 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.212148    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dfv62_c3ca3ce1-7383-4588-a129-d69fedda85b3/serve-healthcheck-canary/0.log"
Sep 11 17:55:01.402113 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.402029    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:55:01.402113 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.402078    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:55:01.402296 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:01.402148    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:55:01.402296 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:01.402162    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:55:01.402296 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:01.402214    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-rbac-proxy-cm podName:025d8e60-0c15-47a4-bd38-7c29f2e30f9e nodeName:}" failed. No retries permitted until 2026-09-11 17:55:05.402200734 +0000 UTC m=+63.245436152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-chg9t" (UID: "025d8e60-0c15-47a4-bd38-7c29f2e30f9e") : configmap "kube-rbac-proxy" not found
Sep 11 17:55:01.402296 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:01.402226    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls podName:025d8e60-0c15-47a4-bd38-7c29f2e30f9e nodeName:}" failed. No retries permitted until 2026-09-11 17:55:05.402220535 +0000 UTC m=+63.245455954 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-chg9t" (UID: "025d8e60-0c15-47a4-bd38-7c29f2e30f9e") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:55:01.422880 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.422851    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2tttk"]
Sep 11 17:55:01.425850 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:55:01.425828    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d451b56_86ce_460a_813f_054ec25c4e8b.slice/crio-337a0012074934b6e129381fcdbe81e2e36ae6ce8c1631247c900ce70bb28752 WatchSource:0}: Error finding container 337a0012074934b6e129381fcdbe81e2e36ae6ce8c1631247c900ce70bb28752: Status 404 returned error can't find the container with id 337a0012074934b6e129381fcdbe81e2e36ae6ce8c1631247c900ce70bb28752
Sep 11 17:55:01.604148 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.604110    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-g6dh4\" (UID: \"e99b97af-cf7c-4b4e-8fac-374449322c1e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4"
Sep 11 17:55:01.604330 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:01.604271    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 17:55:01.604386 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:01.604333    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls podName:e99b97af-cf7c-4b4e-8fac-374449322c1e nodeName:}" failed. No retries permitted until 2026-09-11 17:55:09.604317741 +0000 UTC m=+67.447553164 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-g6dh4" (UID: "e99b97af-cf7c-4b4e-8fac-374449322c1e") : secret "cluster-monitoring-operator-tls" not found
Sep 11 17:55:01.705306 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.705239    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e53386e8-153f-4423-ad30-0486d90e094d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kqx5c\" (UID: \"e53386e8-153f-4423-ad30-0486d90e094d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqx5c"
Sep 11 17:55:01.705424 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:01.705345    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 17:55:01.705424 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:01.705393    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e53386e8-153f-4423-ad30-0486d90e094d-samples-operator-tls podName:e53386e8-153f-4423-ad30-0486d90e094d nodeName:}" failed. No retries permitted until 2026-09-11 17:55:02.705381064 +0000 UTC m=+60.548616483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e53386e8-153f-4423-ad30-0486d90e094d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kqx5c" (UID: "e53386e8-153f-4423-ad30-0486d90e094d") : secret "samples-operator-tls" not found
Sep 11 17:55:01.813674 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.813645    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-k8qz2"]
Sep 11 17:55:01.816036 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.816017    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-k8qz2"
Sep 11 17:55:01.818648 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.818629    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 11 17:55:01.818780 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.818666    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-r7bpg\""
Sep 11 17:55:01.818932 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.818917    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 11 17:55:01.819009 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.818934    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 11 17:55:01.819068 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.819024    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 11 17:55:01.823594 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.823559    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 11 17:55:01.824657 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.824632    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-k8qz2"]
Sep 11 17:55:01.940199 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:01.940175    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vw8qd"
Sep 11 17:55:02.008081 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:02.008056    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/84f48f44-bac5-4f61-bcc5-7dc52f0d1628-config\") pod \"console-operator-696884fb9c-k8qz2\" (UID: \"84f48f44-bac5-4f61-bcc5-7dc52f0d1628\") " pod="openshift-console-operator/console-operator-696884fb9c-k8qz2"
Sep 11 17:55:02.008264 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:02.008134    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84f48f44-bac5-4f61-bcc5-7dc52f0d1628-serving-cert\") pod \"console-operator-696884fb9c-k8qz2\" (UID: \"84f48f44-bac5-4f61-bcc5-7dc52f0d1628\") " pod="openshift-console-operator/console-operator-696884fb9c-k8qz2"
Sep 11 17:55:02.008264 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:02.008162    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rblkd\" (UniqueName: \"kubernetes.io/projected/84f48f44-bac5-4f61-bcc5-7dc52f0d1628-kube-api-access-rblkd\") pod \"console-operator-696884fb9c-k8qz2\" (UID: \"84f48f44-bac5-4f61-bcc5-7dc52f0d1628\") " pod="openshift-console-operator/console-operator-696884fb9c-k8qz2"
Sep 11 17:55:02.008264 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:02.008210    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84f48f44-bac5-4f61-bcc5-7dc52f0d1628-trusted-ca\") pod \"console-operator-696884fb9c-k8qz2\" (UID: \"84f48f44-bac5-4f61-bcc5-7dc52f0d1628\") " pod="openshift-console-operator/console-operator-696884fb9c-k8qz2"
Sep 11 17:55:02.018672 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:02.018640    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2tttk" event={"ID":"4d451b56-86ce-460a-813f-054ec25c4e8b","Type":"ContainerStarted","Data":"337a0012074934b6e129381fcdbe81e2e36ae6ce8c1631247c900ce70bb28752"}
Sep 11 17:55:02.109147 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:02.109106    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84f48f44-bac5-4f61-bcc5-7dc52f0d1628-serving-cert\") pod \"console-operator-696884fb9c-k8qz2\" (UID: \"84f48f44-bac5-4f61-bcc5-7dc52f0d1628\") " pod="openshift-console-operator/console-operator-696884fb9c-k8qz2"
Sep 11 17:55:02.109322 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:02.109156    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rblkd\" (UniqueName: \"kubernetes.io/projected/84f48f44-bac5-4f61-bcc5-7dc52f0d1628-kube-api-access-rblkd\") pod \"console-operator-696884fb9c-k8qz2\" (UID: \"84f48f44-bac5-4f61-bcc5-7dc52f0d1628\") " pod="openshift-console-operator/console-operator-696884fb9c-k8qz2"
Sep 11 17:55:02.109322 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:02.109255    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84f48f44-bac5-4f61-bcc5-7dc52f0d1628-trusted-ca\") pod \"console-operator-696884fb9c-k8qz2\" (UID: \"84f48f44-bac5-4f61-bcc5-7dc52f0d1628\") " pod="openshift-console-operator/console-operator-696884fb9c-k8qz2"
Sep 11 17:55:02.109425 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:02.109345    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/84f48f44-bac5-4f61-bcc5-7dc52f0d1628-config\") pod \"console-operator-696884fb9c-k8qz2\" (UID: \"84f48f44-bac5-4f61-bcc5-7dc52f0d1628\") " pod="openshift-console-operator/console-operator-696884fb9c-k8qz2"
Sep 11 17:55:02.110048 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:02.110005    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/84f48f44-bac5-4f61-bcc5-7dc52f0d1628-config\") pod \"console-operator-696884fb9c-k8qz2\" (UID: \"84f48f44-bac5-4f61-bcc5-7dc52f0d1628\") " pod="openshift-console-operator/console-operator-696884fb9c-k8qz2"
Sep 11 17:55:02.110258 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:02.110232    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/84f48f44-bac5-4f61-bcc5-7dc52f0d1628-trusted-ca\") pod \"console-operator-696884fb9c-k8qz2\" (UID: \"84f48f44-bac5-4f61-bcc5-7dc52f0d1628\") " pod="openshift-console-operator/console-operator-696884fb9c-k8qz2"
Sep 11 17:55:02.111854 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:02.111830    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/84f48f44-bac5-4f61-bcc5-7dc52f0d1628-serving-cert\") pod \"console-operator-696884fb9c-k8qz2\" (UID: \"84f48f44-bac5-4f61-bcc5-7dc52f0d1628\") " pod="openshift-console-operator/console-operator-696884fb9c-k8qz2"
Sep 11 17:55:02.118451 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:02.118429    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rblkd\" (UniqueName: \"kubernetes.io/projected/84f48f44-bac5-4f61-bcc5-7dc52f0d1628-kube-api-access-rblkd\") pod \"console-operator-696884fb9c-k8qz2\" (UID: \"84f48f44-bac5-4f61-bcc5-7dc52f0d1628\") " pod="openshift-console-operator/console-operator-696884fb9c-k8qz2"
Sep 11 17:55:02.126272 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:02.126251    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-k8qz2"
Sep 11 17:55:02.405349 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:02.404979    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-k8qz2"]
Sep 11 17:55:02.452811 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:55:02.452776    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f48f44_bac5_4f61_bcc5_7dc52f0d1628.slice/crio-2a1ff67f8753ec358b82c9e9fd3dc4d951fa2ed761c48d12097855bc2a9245dd WatchSource:0}: Error finding container 2a1ff67f8753ec358b82c9e9fd3dc4d951fa2ed761c48d12097855bc2a9245dd: Status 404 returned error can't find the container with id 2a1ff67f8753ec358b82c9e9fd3dc4d951fa2ed761c48d12097855bc2a9245dd
Sep 11 17:55:02.713215 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:02.713190    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e53386e8-153f-4423-ad30-0486d90e094d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kqx5c\" (UID: \"e53386e8-153f-4423-ad30-0486d90e094d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqx5c"
Sep 11 17:55:02.713386 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:02.713309    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 17:55:02.713386 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:02.713366    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e53386e8-153f-4423-ad30-0486d90e094d-samples-operator-tls podName:e53386e8-153f-4423-ad30-0486d90e094d nodeName:}" failed. No retries permitted until 2026-09-11 17:55:04.713351924 +0000 UTC m=+62.556587342 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e53386e8-153f-4423-ad30-0486d90e094d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kqx5c" (UID: "e53386e8-153f-4423-ad30-0486d90e094d") : secret "samples-operator-tls" not found
Sep 11 17:55:03.022389 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:03.022290    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2tttk" event={"ID":"4d451b56-86ce-460a-813f-054ec25c4e8b","Type":"ContainerStarted","Data":"1cf570b16f96737fd20dc814966441a2ce64eb0a3d6d66cfa54b01e733fe05dc"}
Sep 11 17:55:03.023430 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:03.023407    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-k8qz2" event={"ID":"84f48f44-bac5-4f61-bcc5-7dc52f0d1628","Type":"ContainerStarted","Data":"2a1ff67f8753ec358b82c9e9fd3dc4d951fa2ed761c48d12097855bc2a9245dd"}
Sep 11 17:55:03.041947 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:03.041901    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-2tttk" podStartSLOduration=1.9941776820000001 podStartE2EDuration="3.041886809s" totalImagesPullingTime="1.047709127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:03.040889075 +0000 UTC m=+60.884124516" watchObservedRunningTime="2026-09-11 17:55:03.041886809 +0000 UTC m=+60.885122250"
Sep 11 17:55:04.732162 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:04.732127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e53386e8-153f-4423-ad30-0486d90e094d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kqx5c\" (UID: \"e53386e8-153f-4423-ad30-0486d90e094d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqx5c"
Sep 11 17:55:04.732513 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:04.732282    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 17:55:04.732513 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:04.732342    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e53386e8-153f-4423-ad30-0486d90e094d-samples-operator-tls podName:e53386e8-153f-4423-ad30-0486d90e094d nodeName:}" failed. No retries permitted until 2026-09-11 17:55:08.732328493 +0000 UTC m=+66.575563912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e53386e8-153f-4423-ad30-0486d90e094d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kqx5c" (UID: "e53386e8-153f-4423-ad30-0486d90e094d") : secret "samples-operator-tls" not found
Sep 11 17:55:05.030641 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:05.030569    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/0.log"
Sep 11 17:55:05.030641 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:05.030607    2560 generic.go:356] "Generic (PLEG): container finished" podID="84f48f44-bac5-4f61-bcc5-7dc52f0d1628" containerID="20349424cf85f60a995bc0241adb4dea166b44a56353f6260efbd652c02bd415" exitCode=255
Sep 11 17:55:05.030641 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:05.030633    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-k8qz2" event={"ID":"84f48f44-bac5-4f61-bcc5-7dc52f0d1628","Type":"ContainerDied","Data":"20349424cf85f60a995bc0241adb4dea166b44a56353f6260efbd652c02bd415"}
Sep 11 17:55:05.030890 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:05.030875    2560 scope.go:117] "RemoveContainer" containerID="20349424cf85f60a995bc0241adb4dea166b44a56353f6260efbd652c02bd415"
Sep 11 17:55:05.437959 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:05.437863    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:55:05.437959 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:05.437907    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:55:05.438174 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:05.437985    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 17:55:05.438174 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:05.438033    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:55:05.438174 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:05.438052    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-rbac-proxy-cm podName:025d8e60-0c15-47a4-bd38-7c29f2e30f9e nodeName:}" failed. No retries permitted until 2026-09-11 17:55:13.438032386 +0000 UTC m=+71.281267805 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-chg9t" (UID: "025d8e60-0c15-47a4-bd38-7c29f2e30f9e") : configmap "kube-rbac-proxy" not found
Sep 11 17:55:05.438174 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:05.438101    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls podName:025d8e60-0c15-47a4-bd38-7c29f2e30f9e nodeName:}" failed. No retries permitted until 2026-09-11 17:55:13.438084212 +0000 UTC m=+71.281319637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-chg9t" (UID: "025d8e60-0c15-47a4-bd38-7c29f2e30f9e") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:55:06.034809 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:06.034783    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/1.log"
Sep 11 17:55:06.035220 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:06.035202    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/0.log"
Sep 11 17:55:06.035267 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:06.035240    2560 generic.go:356] "Generic (PLEG): container finished" podID="84f48f44-bac5-4f61-bcc5-7dc52f0d1628" containerID="fb5e5c8333437edfd1471830c251303e8ade96ab0fa4fab4e485b501a4e5f5d7" exitCode=255
Sep 11 17:55:06.035298 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:06.035284    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-k8qz2" event={"ID":"84f48f44-bac5-4f61-bcc5-7dc52f0d1628","Type":"ContainerDied","Data":"fb5e5c8333437edfd1471830c251303e8ade96ab0fa4fab4e485b501a4e5f5d7"}
Sep 11 17:55:06.035381 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:06.035371    2560 scope.go:117] "RemoveContainer" containerID="20349424cf85f60a995bc0241adb4dea166b44a56353f6260efbd652c02bd415"
Sep 11 17:55:06.035600 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:06.035581    2560 scope.go:117] "RemoveContainer" containerID="fb5e5c8333437edfd1471830c251303e8ade96ab0fa4fab4e485b501a4e5f5d7"
Sep 11 17:55:06.035818 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:06.035799    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-k8qz2_openshift-console-operator(84f48f44-bac5-4f61-bcc5-7dc52f0d1628)\"" pod="openshift-console-operator/console-operator-696884fb9c-k8qz2" podUID="84f48f44-bac5-4f61-bcc5-7dc52f0d1628"
Sep 11 17:55:07.039426 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:07.039396    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/1.log"
Sep 11 17:55:07.039826 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:07.039704    2560 scope.go:117] "RemoveContainer" containerID="fb5e5c8333437edfd1471830c251303e8ade96ab0fa4fab4e485b501a4e5f5d7"
Sep 11 17:55:07.039880 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:07.039861    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-k8qz2_openshift-console-operator(84f48f44-bac5-4f61-bcc5-7dc52f0d1628)\"" pod="openshift-console-operator/console-operator-696884fb9c-k8qz2" podUID="84f48f44-bac5-4f61-bcc5-7dc52f0d1628"
Sep 11 17:55:08.360608 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:08.360575    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs\") pod \"network-metrics-daemon-8xg8d\" (UID: \"8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf\") " pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:55:08.361073 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:08.360616    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret\") pod \"global-pull-secret-syncer-r7cvn\" (UID: \"c5702497-d40f-4a13-b5b8-0be72c784011\") " pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:55:08.363864 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:08.363843    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 17:55:08.363959 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:08.363847    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 17:55:08.371255 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:08.371238    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 17:55:08.371310 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:08.371295    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs podName:8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf nodeName:}" failed. No retries permitted until 2026-09-11 17:56:12.37128029 +0000 UTC m=+130.214515709 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs") pod "network-metrics-daemon-8xg8d" (UID: "8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf") : secret "metrics-daemon-secret" not found
Sep 11 17:55:08.373492 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:08.373472    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5702497-d40f-4a13-b5b8-0be72c784011-original-pull-secret\") pod \"global-pull-secret-syncer-r7cvn\" (UID: \"c5702497-d40f-4a13-b5b8-0be72c784011\") " pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:55:08.461477 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:08.461445    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb5bc\" (UniqueName: \"kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc\") pod \"network-check-target-xfh2z\" (UID: \"2c3f9d2a-8267-48f4-9c2e-d06ff6f15485\") " pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:55:08.464433 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:08.464412    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 17:55:08.475028 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:08.475004    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 17:55:08.485016 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:08.484986    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb5bc\" (UniqueName: \"kubernetes.io/projected/2c3f9d2a-8267-48f4-9c2e-d06ff6f15485-kube-api-access-zb5bc\") pod \"network-check-target-xfh2z\" (UID: \"2c3f9d2a-8267-48f4-9c2e-d06ff6f15485\") " pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:55:08.592270 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:08.592228    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r7cvn"
Sep 11 17:55:08.600399 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:08.600362    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vhttf\""
Sep 11 17:55:08.608683 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:08.608662    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:55:08.764083 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:08.764041    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e53386e8-153f-4423-ad30-0486d90e094d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kqx5c\" (UID: \"e53386e8-153f-4423-ad30-0486d90e094d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqx5c"
Sep 11 17:55:08.764244 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:08.764164    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 11 17:55:08.764244 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:08.764226    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e53386e8-153f-4423-ad30-0486d90e094d-samples-operator-tls podName:e53386e8-153f-4423-ad30-0486d90e094d nodeName:}" failed. No retries permitted until 2026-09-11 17:55:16.764204506 +0000 UTC m=+74.607439926 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e53386e8-153f-4423-ad30-0486d90e094d-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kqx5c" (UID: "e53386e8-153f-4423-ad30-0486d90e094d") : secret "samples-operator-tls" not found
Sep 11 17:55:08.864701 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:08.864678    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls\") pod \"dns-default-45lfw\" (UID: \"be400dcd-4cff-4bc9-b572-b714103a5c46\") " pod="openshift-dns/dns-default-45lfw"
Sep 11 17:55:08.864794 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:08.864688    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r7cvn"]
Sep 11 17:55:08.864838 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:08.864796    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 17:55:08.864886 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:08.864875    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls podName:be400dcd-4cff-4bc9-b572-b714103a5c46 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:40.86485498 +0000 UTC m=+98.708090399 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls") pod "dns-default-45lfw" (UID: "be400dcd-4cff-4bc9-b572-b714103a5c46") : secret "dns-default-metrics-tls" not found
Sep 11 17:55:08.867047 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:55:08.867022    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5702497_d40f_4a13_b5b8_0be72c784011.slice/crio-a151cefb32bca3ca2841a3c7d169afcf644614c4a81d74af6066e0145c794b33 WatchSource:0}: Error finding container a151cefb32bca3ca2841a3c7d169afcf644614c4a81d74af6066e0145c794b33: Status 404 returned error can't find the container with id a151cefb32bca3ca2841a3c7d169afcf644614c4a81d74af6066e0145c794b33
Sep 11 17:55:08.884268 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:08.884242    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xfh2z"]
Sep 11 17:55:08.887215 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:55:08.887192    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c3f9d2a_8267_48f4_9c2e_d06ff6f15485.slice/crio-526a466f28970d594b31d3cc460f7887494cb546e547aedc9d95bf34b09f3cb3 WatchSource:0}: Error finding container 526a466f28970d594b31d3cc460f7887494cb546e547aedc9d95bf34b09f3cb3: Status 404 returned error can't find the container with id 526a466f28970d594b31d3cc460f7887494cb546e547aedc9d95bf34b09f3cb3
Sep 11 17:55:09.045193 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:09.045156    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r7cvn" event={"ID":"c5702497-d40f-4a13-b5b8-0be72c784011","Type":"ContainerStarted","Data":"a151cefb32bca3ca2841a3c7d169afcf644614c4a81d74af6066e0145c794b33"}
Sep 11 17:55:09.046254 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:09.046236    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xfh2z" event={"ID":"2c3f9d2a-8267-48f4-9c2e-d06ff6f15485","Type":"ContainerStarted","Data":"526a466f28970d594b31d3cc460f7887494cb546e547aedc9d95bf34b09f3cb3"}
Sep 11 17:55:09.671540 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:09.671500    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-g6dh4\" (UID: \"e99b97af-cf7c-4b4e-8fac-374449322c1e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4"
Sep 11 17:55:09.672149 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:09.671629    2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 11 17:55:09.672149 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:09.671691    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls podName:e99b97af-cf7c-4b4e-8fac-374449322c1e nodeName:}" failed. No retries permitted until 2026-09-11 17:55:25.671671303 +0000 UTC m=+83.514906722 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-g6dh4" (UID: "e99b97af-cf7c-4b4e-8fac-374449322c1e") : secret "cluster-monitoring-operator-tls" not found
Sep 11 17:55:12.127239 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:12.127203    2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-k8qz2"
Sep 11 17:55:12.127738 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:12.127277    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-k8qz2"
Sep 11 17:55:12.127738 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:12.127664    2560 scope.go:117] "RemoveContainer" containerID="fb5e5c8333437edfd1471830c251303e8ade96ab0fa4fab4e485b501a4e5f5d7"
Sep 11 17:55:12.127868 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:12.127847    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-k8qz2_openshift-console-operator(84f48f44-bac5-4f61-bcc5-7dc52f0d1628)\"" pod="openshift-console-operator/console-operator-696884fb9c-k8qz2" podUID="84f48f44-bac5-4f61-bcc5-7dc52f0d1628"
Sep 11 17:55:13.501253 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:13.501227    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:55:13.501518 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:13.501272    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:55:13.501518 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:13.501372    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 17:55:13.501518 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:13.501432    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls podName:025d8e60-0c15-47a4-bd38-7c29f2e30f9e nodeName:}" failed. No retries permitted until 2026-09-11 17:55:29.50141558 +0000 UTC m=+87.344651012 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-chg9t" (UID: "025d8e60-0c15-47a4-bd38-7c29f2e30f9e") : secret "insights-runtime-extractor-tls" not found
Sep 11 17:55:13.501703 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:13.501686    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:55:14.062990 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:14.062948    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xfh2z" event={"ID":"2c3f9d2a-8267-48f4-9c2e-d06ff6f15485","Type":"ContainerStarted","Data":"74e23a1013006020c5461aad2abc4be752f19eded543ea1e72b1476b2e1b1103"}
Sep 11 17:55:14.063202 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:14.063022    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:55:14.064458 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:14.064438    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r7cvn" event={"ID":"c5702497-d40f-4a13-b5b8-0be72c784011","Type":"ContainerStarted","Data":"66b7ebe691b3bcec30d9ad7c392c4c8f23dd8f03834ea19d3c074f76ca0ad2d0"}
Sep 11 17:55:14.081177 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:14.081127    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xfh2z" podStartSLOduration=66.445450899 podStartE2EDuration="1m11.081110463s" totalImagesPullingTime="4.635659564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:14.08050065 +0000 UTC m=+71.923736104" watchObservedRunningTime="2026-09-11 17:55:14.081110463 +0000 UTC m=+71.924345903"
Sep 11 17:55:14.098145 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:14.098095    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-r7cvn" podStartSLOduration=66.4301172 podStartE2EDuration="1m11.098079105s" totalImagesPullingTime="4.667961905s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:14.097523111 +0000 UTC m=+71.940758552" watchObservedRunningTime="2026-09-11 17:55:14.098079105 +0000 UTC m=+71.941314545"
Sep 11 17:55:16.830627 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:16.830576    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e53386e8-153f-4423-ad30-0486d90e094d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kqx5c\" (UID: \"e53386e8-153f-4423-ad30-0486d90e094d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqx5c"
Sep 11 17:55:16.833193 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:16.833171    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e53386e8-153f-4423-ad30-0486d90e094d-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kqx5c\" (UID: \"e53386e8-153f-4423-ad30-0486d90e094d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqx5c"
Sep 11 17:55:17.129282 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:17.129201    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dd6gq\""
Sep 11 17:55:17.137232 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:17.137198    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqx5c"
Sep 11 17:55:17.406970 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:17.406868    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqx5c"]
Sep 11 17:55:18.080938 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:18.080905    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqx5c" event={"ID":"e53386e8-153f-4423-ad30-0486d90e094d","Type":"ContainerStarted","Data":"5a7ba97b575bc1a57c1b638336d0c70bd14cc633c0fbe896f2a7cdb5807ed345"}
Sep 11 17:55:20.089040 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:20.089003    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqx5c" event={"ID":"e53386e8-153f-4423-ad30-0486d90e094d","Type":"ContainerStarted","Data":"7b24f46c84b424e5cb342ce43433f3182f4dd9175bdc9ee9828cabb29a1306a5"}
Sep 11 17:55:20.089504 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:20.089051    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqx5c" event={"ID":"e53386e8-153f-4423-ad30-0486d90e094d","Type":"ContainerStarted","Data":"e88c7e06e7ccd91a316c37b37bdaa0f7923c80137a60ab512530b15590e42c0e"}
Sep 11 17:55:20.109549 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:20.109499    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqx5c" podStartSLOduration=18.045340169 podStartE2EDuration="20.109483498s" totalImagesPullingTime="2.064143329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:20.108540985 +0000 UTC m=+77.951776422" watchObservedRunningTime="2026-09-11 17:55:20.109483498 +0000 UTC m=+77.952718938"
Sep 11 17:55:24.681729 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:24.681623    2560 scope.go:117] "RemoveContainer" containerID="fb5e5c8333437edfd1471830c251303e8ade96ab0fa4fab4e485b501a4e5f5d7"
Sep 11 17:55:25.105878 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:25.105850    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/2.log"
Sep 11 17:55:25.106281 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:25.106267    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/1.log"
Sep 11 17:55:25.106329 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:25.106298    2560 generic.go:356] "Generic (PLEG): container finished" podID="84f48f44-bac5-4f61-bcc5-7dc52f0d1628" containerID="87c24ee7f701a1b9a4910aef928ade09cf0b1443f21d2d9a18e5c6d8d0fc460c" exitCode=255
Sep 11 17:55:25.106363 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:25.106334    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-k8qz2" event={"ID":"84f48f44-bac5-4f61-bcc5-7dc52f0d1628","Type":"ContainerDied","Data":"87c24ee7f701a1b9a4910aef928ade09cf0b1443f21d2d9a18e5c6d8d0fc460c"}
Sep 11 17:55:25.106408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:25.106398    2560 scope.go:117] "RemoveContainer" containerID="fb5e5c8333437edfd1471830c251303e8ade96ab0fa4fab4e485b501a4e5f5d7"
Sep 11 17:55:25.106668 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:25.106651    2560 scope.go:117] "RemoveContainer" containerID="87c24ee7f701a1b9a4910aef928ade09cf0b1443f21d2d9a18e5c6d8d0fc460c"
Sep 11 17:55:25.106857 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:25.106839    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-k8qz2_openshift-console-operator(84f48f44-bac5-4f61-bcc5-7dc52f0d1628)\"" pod="openshift-console-operator/console-operator-696884fb9c-k8qz2" podUID="84f48f44-bac5-4f61-bcc5-7dc52f0d1628"
Sep 11 17:55:25.697605 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:25.697570    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-g6dh4\" (UID: \"e99b97af-cf7c-4b4e-8fac-374449322c1e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4"
Sep 11 17:55:25.700031 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:25.700011    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e99b97af-cf7c-4b4e-8fac-374449322c1e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-g6dh4\" (UID: \"e99b97af-cf7c-4b4e-8fac-374449322c1e\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4"
Sep 11 17:55:25.924274 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:25.924241    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m8zdr\""
Sep 11 17:55:25.931723 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:25.931704    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4"
Sep 11 17:55:26.110827 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:26.110800    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/2.log"
Sep 11 17:55:26.201866 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:26.201835    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4"]
Sep 11 17:55:26.204737 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:55:26.204711    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99b97af_cf7c_4b4e_8fac_374449322c1e.slice/crio-a95859baaf2ddf524c0be455213f82847380aa69889bccc9e43150da4054c547 WatchSource:0}: Error finding container a95859baaf2ddf524c0be455213f82847380aa69889bccc9e43150da4054c547: Status 404 returned error can't find the container with id a95859baaf2ddf524c0be455213f82847380aa69889bccc9e43150da4054c547
Sep 11 17:55:27.114464 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:27.114431    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4" event={"ID":"e99b97af-cf7c-4b4e-8fac-374449322c1e","Type":"ContainerStarted","Data":"a95859baaf2ddf524c0be455213f82847380aa69889bccc9e43150da4054c547"}
Sep 11 17:55:29.122329 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:29.122294    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4" event={"ID":"e99b97af-cf7c-4b4e-8fac-374449322c1e","Type":"ContainerStarted","Data":"af4ddb629d156a13c22e2136c0a2342861b6f738df3fb3471f85e27497f76eac"}
Sep 11 17:55:29.140927 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:29.140880    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-g6dh4" podStartSLOduration=34.150347724 podStartE2EDuration="36.140864756s" totalImagesPullingTime="1.990517032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:29.140342379 +0000 UTC m=+86.983577834" watchObservedRunningTime="2026-09-11 17:55:29.140864756 +0000 UTC m=+86.984100197"
Sep 11 17:55:29.526123 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:29.526084    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:55:29.528435 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:29.528412    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/025d8e60-0c15-47a4-bd38-7c29f2e30f9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-chg9t\" (UID: \"025d8e60-0c15-47a4-bd38-7c29f2e30f9e\") " pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:55:29.764691 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:29.764653    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ntvnk\""
Sep 11 17:55:29.772179 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:29.772159    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-chg9t"
Sep 11 17:55:30.045614 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:30.045589    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-chg9t"]
Sep 11 17:55:30.048004 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:55:30.047972    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod025d8e60_0c15_47a4_bd38_7c29f2e30f9e.slice/crio-33630f1a9c2b2f3bb29318b83844b8c21477b671ce164727f71df7d80d76b40f WatchSource:0}: Error finding container 33630f1a9c2b2f3bb29318b83844b8c21477b671ce164727f71df7d80d76b40f: Status 404 returned error can't find the container with id 33630f1a9c2b2f3bb29318b83844b8c21477b671ce164727f71df7d80d76b40f
Sep 11 17:55:30.128112 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:30.128081    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-chg9t" event={"ID":"025d8e60-0c15-47a4-bd38-7c29f2e30f9e","Type":"ContainerStarted","Data":"991a6706f0ee799cb6ade1988769b7a4112064667455efa4a47d93eb7ca1436b"}
Sep 11 17:55:30.128439 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:30.128140    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-chg9t" event={"ID":"025d8e60-0c15-47a4-bd38-7c29f2e30f9e","Type":"ContainerStarted","Data":"33630f1a9c2b2f3bb29318b83844b8c21477b671ce164727f71df7d80d76b40f"}
Sep 11 17:55:31.134869 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:31.134829    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-chg9t" event={"ID":"025d8e60-0c15-47a4-bd38-7c29f2e30f9e","Type":"ContainerStarted","Data":"63ec31ffad15a6e723ae177794074c3dfb69ad8abadf21d45559ea965fc360a4"}
Sep 11 17:55:32.126533 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:32.126500    2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-k8qz2"
Sep 11 17:55:32.126715 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:32.126564    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-k8qz2"
Sep 11 17:55:32.126937 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:32.126919    2560 scope.go:117] "RemoveContainer" containerID="87c24ee7f701a1b9a4910aef928ade09cf0b1443f21d2d9a18e5c6d8d0fc460c"
Sep 11 17:55:32.127101 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:32.127081    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-k8qz2_openshift-console-operator(84f48f44-bac5-4f61-bcc5-7dc52f0d1628)\"" pod="openshift-console-operator/console-operator-696884fb9c-k8qz2" podUID="84f48f44-bac5-4f61-bcc5-7dc52f0d1628"
Sep 11 17:55:33.144191 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:33.144155    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-chg9t" event={"ID":"025d8e60-0c15-47a4-bd38-7c29f2e30f9e","Type":"ContainerStarted","Data":"7806613fa518c36e9537434f06a0e36069afbf6f475cc8e32cf7d0c1b00eaa4d"}
Sep 11 17:55:33.170213 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:33.170164    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-chg9t" podStartSLOduration=34.134050893 podStartE2EDuration="36.170147401s" totalImagesPullingTime="2.036096508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 17:54:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:33.169586676 +0000 UTC m=+91.012822175" watchObservedRunningTime="2026-09-11 17:55:33.170147401 +0000 UTC m=+91.013382833"
Sep 11 17:55:34.475891 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.475859    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d446b885-2nk2p"]
Sep 11 17:55:34.479034 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.479019    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d446b885-2nk2p"
Sep 11 17:55:34.482636 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.482613    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 17:55:34.483919 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.483898    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 17:55:34.484172 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.484157    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 17:55:34.485353 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.485332    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 17:55:34.497990 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.497966    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d446b885-2nk2p"]
Sep 11 17:55:34.559644 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.559608    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/18c95a40-c603-432d-854a-5c95f887a9bf-klusterlet-config\") pod \"klusterlet-addon-workmgr-54d446b885-2nk2p\" (UID: \"18c95a40-c603-432d-854a-5c95f887a9bf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d446b885-2nk2p"
Sep 11 17:55:34.559644 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.559646    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5prlt\" (UniqueName: \"kubernetes.io/projected/18c95a40-c603-432d-854a-5c95f887a9bf-kube-api-access-5prlt\") pod \"klusterlet-addon-workmgr-54d446b885-2nk2p\" (UID: \"18c95a40-c603-432d-854a-5c95f887a9bf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d446b885-2nk2p"
Sep 11 17:55:34.559901 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.559684    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18c95a40-c603-432d-854a-5c95f887a9bf-tmp\") pod \"klusterlet-addon-workmgr-54d446b885-2nk2p\" (UID: \"18c95a40-c603-432d-854a-5c95f887a9bf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d446b885-2nk2p"
Sep 11 17:55:34.605351 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.605316    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-74tpj"]
Sep 11 17:55:34.608419 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.608401    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-74tpj"
Sep 11 17:55:34.612611 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.612591    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-799b9b8bf5-z8cnm"]
Sep 11 17:55:34.618488 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.618467    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.618605 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.618489    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 11 17:55:34.618605 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.618555    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7k94f\""
Sep 11 17:55:34.618605 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.618582    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 11 17:55:34.634713 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.634693    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 17:55:34.635289 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.635268    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-74tpj"]
Sep 11 17:55:34.635943 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.635929    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5487c\""
Sep 11 17:55:34.636627 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.636611    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 17:55:34.637344 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.637323    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 17:55:34.654194 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.654174    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-799b9b8bf5-z8cnm"]
Sep 11 17:55:34.657635 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.657614    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 17:55:34.660148 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.660131    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a95d5df4-396b-404d-af3f-3787a83dea92-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-74tpj\" (UID: \"a95d5df4-396b-404d-af3f-3787a83dea92\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-74tpj"
Sep 11 17:55:34.660235 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.660161    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-ca-trust-extracted\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.660235 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.660181    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-bound-sa-token\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.660235 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.660199    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj6dt\" (UniqueName: \"kubernetes.io/projected/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-kube-api-access-gj6dt\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.660235 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.660217    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-image-registry-private-configuration\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.660235 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.660234    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-registry-certificates\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.660412 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.660291    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-trusted-ca\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.660412 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.660318    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/18c95a40-c603-432d-854a-5c95f887a9bf-klusterlet-config\") pod \"klusterlet-addon-workmgr-54d446b885-2nk2p\" (UID: \"18c95a40-c603-432d-854a-5c95f887a9bf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d446b885-2nk2p"
Sep 11 17:55:34.660412 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.660337    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5prlt\" (UniqueName: \"kubernetes.io/projected/18c95a40-c603-432d-854a-5c95f887a9bf-kube-api-access-5prlt\") pod \"klusterlet-addon-workmgr-54d446b885-2nk2p\" (UID: \"18c95a40-c603-432d-854a-5c95f887a9bf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d446b885-2nk2p"
Sep 11 17:55:34.660412 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.660362    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a95d5df4-396b-404d-af3f-3787a83dea92-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-74tpj\" (UID: \"a95d5df4-396b-404d-af3f-3787a83dea92\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-74tpj"
Sep 11 17:55:34.660412 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.660379    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18c95a40-c603-432d-854a-5c95f887a9bf-tmp\") pod \"klusterlet-addon-workmgr-54d446b885-2nk2p\" (UID: \"18c95a40-c603-432d-854a-5c95f887a9bf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d446b885-2nk2p"
Sep 11 17:55:34.660412 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.660397    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-registry-tls\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.660618 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.660411    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-installation-pull-secrets\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.660851 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.660834    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18c95a40-c603-432d-854a-5c95f887a9bf-tmp\") pod \"klusterlet-addon-workmgr-54d446b885-2nk2p\" (UID: \"18c95a40-c603-432d-854a-5c95f887a9bf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d446b885-2nk2p"
Sep 11 17:55:34.662917 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.662900    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/18c95a40-c603-432d-854a-5c95f887a9bf-klusterlet-config\") pod \"klusterlet-addon-workmgr-54d446b885-2nk2p\" (UID: \"18c95a40-c603-432d-854a-5c95f887a9bf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d446b885-2nk2p"
Sep 11 17:55:34.680860 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.680834    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5prlt\" (UniqueName: \"kubernetes.io/projected/18c95a40-c603-432d-854a-5c95f887a9bf-kube-api-access-5prlt\") pod \"klusterlet-addon-workmgr-54d446b885-2nk2p\" (UID: \"18c95a40-c603-432d-854a-5c95f887a9bf\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d446b885-2nk2p"
Sep 11 17:55:34.721716 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.721680    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-799b9b8bf5-z8cnm"]
Sep 11 17:55:34.721943 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:34.721917    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-gj6dt registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm" podUID="c0cf0e3e-be34-431b-9308-6ae7e8d9a009"
Sep 11 17:55:34.761345 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.761261    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a95d5df4-396b-404d-af3f-3787a83dea92-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-74tpj\" (UID: \"a95d5df4-396b-404d-af3f-3787a83dea92\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-74tpj"
Sep 11 17:55:34.761345 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.761327    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-ca-trust-extracted\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.761598 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.761575    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-bound-sa-token\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.761655 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.761620    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj6dt\" (UniqueName: \"kubernetes.io/projected/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-kube-api-access-gj6dt\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.761711 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.761652    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-image-registry-private-configuration\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.761711 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.761679    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-registry-certificates\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.762293 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.761711    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-ca-trust-extracted\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.762293 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.761722    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-trusted-ca\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.762293 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.761811    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a95d5df4-396b-404d-af3f-3787a83dea92-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-74tpj\" (UID: \"a95d5df4-396b-404d-af3f-3787a83dea92\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-74tpj"
Sep 11 17:55:34.762293 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.761863    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-registry-tls\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.762293 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.761888    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-installation-pull-secrets\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.762563 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.762402    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a95d5df4-396b-404d-af3f-3787a83dea92-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-74tpj\" (UID: \"a95d5df4-396b-404d-af3f-3787a83dea92\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-74tpj"
Sep 11 17:55:34.762898 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.762873    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-registry-certificates\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.762986 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.762911    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-trusted-ca\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.764546 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.764524    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-image-registry-private-configuration\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.764546 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.764531    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-registry-tls\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.764692 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.764583    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-installation-pull-secrets\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.764692 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.764650    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a95d5df4-396b-404d-af3f-3787a83dea92-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-74tpj\" (UID: \"a95d5df4-396b-404d-af3f-3787a83dea92\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-74tpj"
Sep 11 17:55:34.782043 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.782020    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-bound-sa-token\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.783640 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.783622    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj6dt\" (UniqueName: \"kubernetes.io/projected/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-kube-api-access-gj6dt\") pod \"image-registry-799b9b8bf5-z8cnm\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") " pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:34.788505 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.788485    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d446b885-2nk2p"
Sep 11 17:55:34.920129 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:34.920101    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-74tpj"
Sep 11 17:55:35.073727 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:55:35.073700    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c95a40_c603_432d_854a_5c95f887a9bf.slice/crio-fef3c75c4f9dae64880b1c9ca6dbb90960d95599d201ff81817f251ca91e3d81 WatchSource:0}: Error finding container fef3c75c4f9dae64880b1c9ca6dbb90960d95599d201ff81817f251ca91e3d81: Status 404 returned error can't find the container with id fef3c75c4f9dae64880b1c9ca6dbb90960d95599d201ff81817f251ca91e3d81
Sep 11 17:55:35.074906 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.074886    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d446b885-2nk2p"]
Sep 11 17:55:35.153296 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.153266    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:35.153296 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.153290    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d446b885-2nk2p" event={"ID":"18c95a40-c603-432d-854a-5c95f887a9bf","Type":"ContainerStarted","Data":"fef3c75c4f9dae64880b1c9ca6dbb90960d95599d201ff81817f251ca91e3d81"}
Sep 11 17:55:35.162925 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.162893    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:35.192540 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.192505    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-74tpj"]
Sep 11 17:55:35.195295 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:55:35.195272    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95d5df4_396b_404d_af3f_3787a83dea92.slice/crio-1b1bfba846f3029c27638a50231d0bf082553ca6400efb9296e86a4fe7f618d4 WatchSource:0}: Error finding container 1b1bfba846f3029c27638a50231d0bf082553ca6400efb9296e86a4fe7f618d4: Status 404 returned error can't find the container with id 1b1bfba846f3029c27638a50231d0bf082553ca6400efb9296e86a4fe7f618d4
Sep 11 17:55:35.265515 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.265480    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-image-registry-private-configuration\") pod \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") "
Sep 11 17:55:35.265515 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.265515    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-bound-sa-token\") pod \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") "
Sep 11 17:55:35.265694 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.265554    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-installation-pull-secrets\") pod \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") "
Sep 11 17:55:35.265694 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.265586    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-registry-certificates\") pod \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") "
Sep 11 17:55:35.265694 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.265614    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gj6dt\" (UniqueName: \"kubernetes.io/projected/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-kube-api-access-gj6dt\") pod \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") "
Sep 11 17:55:35.265694 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.265638    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-trusted-ca\") pod \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") "
Sep 11 17:55:35.265694 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.265660    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-registry-tls\") pod \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") "
Sep 11 17:55:35.265694 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.265692    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-ca-trust-extracted\") pod \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\" (UID: \"c0cf0e3e-be34-431b-9308-6ae7e8d9a009\") "
Sep 11 17:55:35.267889 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.267862    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c0cf0e3e-be34-431b-9308-6ae7e8d9a009" (UID: "c0cf0e3e-be34-431b-9308-6ae7e8d9a009"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:55:35.268642 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.268623    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c0cf0e3e-be34-431b-9308-6ae7e8d9a009" (UID: "c0cf0e3e-be34-431b-9308-6ae7e8d9a009"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:55:35.269037 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.269011    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c0cf0e3e-be34-431b-9308-6ae7e8d9a009" (UID: "c0cf0e3e-be34-431b-9308-6ae7e8d9a009"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:55:35.269152 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.269073    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c0cf0e3e-be34-431b-9308-6ae7e8d9a009" (UID: "c0cf0e3e-be34-431b-9308-6ae7e8d9a009"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:55:35.269152 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.269103    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c0cf0e3e-be34-431b-9308-6ae7e8d9a009" (UID: "c0cf0e3e-be34-431b-9308-6ae7e8d9a009"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:55:35.270706 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.270682    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c0cf0e3e-be34-431b-9308-6ae7e8d9a009" (UID: "c0cf0e3e-be34-431b-9308-6ae7e8d9a009"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:55:35.270811 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.270714    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-kube-api-access-gj6dt" (OuterVolumeSpecName: "kube-api-access-gj6dt") pod "c0cf0e3e-be34-431b-9308-6ae7e8d9a009" (UID: "c0cf0e3e-be34-431b-9308-6ae7e8d9a009"). InnerVolumeSpecName "kube-api-access-gj6dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:55:35.270811 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.270786    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c0cf0e3e-be34-431b-9308-6ae7e8d9a009" (UID: "c0cf0e3e-be34-431b-9308-6ae7e8d9a009"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:55:35.366462 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.366393    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj6dt\" (UniqueName: \"kubernetes.io/projected/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-kube-api-access-gj6dt\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:55:35.366462 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.366415    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-trusted-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:55:35.366462 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.366425    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-registry-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:55:35.366462 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.366433    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-ca-trust-extracted\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:55:35.366462 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.366442    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-image-registry-private-configuration\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:55:35.366462 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.366450    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-bound-sa-token\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:55:35.366462 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.366458    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-installation-pull-secrets\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:55:35.366462 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:35.366467    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c0cf0e3e-be34-431b-9308-6ae7e8d9a009-registry-certificates\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:55:36.160130 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:36.160097    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-799b9b8bf5-z8cnm"
Sep 11 17:55:36.162491 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:36.162458    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-74tpj" event={"ID":"a95d5df4-396b-404d-af3f-3787a83dea92","Type":"ContainerStarted","Data":"1b1bfba846f3029c27638a50231d0bf082553ca6400efb9296e86a4fe7f618d4"}
Sep 11 17:55:36.201109 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:36.201071    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-799b9b8bf5-z8cnm"]
Sep 11 17:55:36.205691 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:36.205660    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-799b9b8bf5-z8cnm"]
Sep 11 17:55:36.689918 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:36.688338    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0cf0e3e-be34-431b-9308-6ae7e8d9a009" path="/var/lib/kubelet/pods/c0cf0e3e-be34-431b-9308-6ae7e8d9a009/volumes"
Sep 11 17:55:37.166411 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.166376    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-74tpj" event={"ID":"a95d5df4-396b-404d-af3f-3787a83dea92","Type":"ContainerStarted","Data":"32b2dd1e96126150c7560a333233b8f920fef6c89f6e298fd18c3bc316b39854"}
Sep 11 17:55:37.183412 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.183357    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-74tpj" podStartSLOduration=2.032107266 podStartE2EDuration="3.183337792s" totalImagesPullingTime="1.151230526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:37.182979103 +0000 UTC m=+95.026214545" watchObservedRunningTime="2026-09-11 17:55:37.183337792 +0000 UTC m=+95.026573232"
Sep 11 17:55:37.832248 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.832209    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-s7225"]
Sep 11 17:55:37.835777 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.835718    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7225"
Sep 11 17:55:37.838939 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.838913    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 17:55:37.840035 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.840018    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 17:55:37.840154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.840119    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bg7vx\""
Sep 11 17:55:37.840154 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.840120    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 17:55:37.849414 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.849391    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-s7225"]
Sep 11 17:55:37.887378 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.887339    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00cc7737-711d-4e6f-87ce-da84f6bc1684-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-s7225\" (UID: \"00cc7737-711d-4e6f-87ce-da84f6bc1684\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7225"
Sep 11 17:55:37.887524 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.887431    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h58q\" (UniqueName: \"kubernetes.io/projected/00cc7737-711d-4e6f-87ce-da84f6bc1684-kube-api-access-6h58q\") pod \"prometheus-operator-68bcd7f987-s7225\" (UID: \"00cc7737-711d-4e6f-87ce-da84f6bc1684\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7225"
Sep 11 17:55:37.887524 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.887486    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/00cc7737-711d-4e6f-87ce-da84f6bc1684-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-s7225\" (UID: \"00cc7737-711d-4e6f-87ce-da84f6bc1684\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7225"
Sep 11 17:55:37.887647 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.887523    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00cc7737-711d-4e6f-87ce-da84f6bc1684-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-s7225\" (UID: \"00cc7737-711d-4e6f-87ce-da84f6bc1684\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7225"
Sep 11 17:55:37.988873 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.988836    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/00cc7737-711d-4e6f-87ce-da84f6bc1684-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-s7225\" (UID: \"00cc7737-711d-4e6f-87ce-da84f6bc1684\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7225"
Sep 11 17:55:37.989051 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.988901    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00cc7737-711d-4e6f-87ce-da84f6bc1684-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-s7225\" (UID: \"00cc7737-711d-4e6f-87ce-da84f6bc1684\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7225"
Sep 11 17:55:37.989051 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.988942    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00cc7737-711d-4e6f-87ce-da84f6bc1684-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-s7225\" (UID: \"00cc7737-711d-4e6f-87ce-da84f6bc1684\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7225"
Sep 11 17:55:37.989051 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.989015    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h58q\" (UniqueName: \"kubernetes.io/projected/00cc7737-711d-4e6f-87ce-da84f6bc1684-kube-api-access-6h58q\") pod \"prometheus-operator-68bcd7f987-s7225\" (UID: \"00cc7737-711d-4e6f-87ce-da84f6bc1684\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7225"
Sep 11 17:55:37.990002 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.989971    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00cc7737-711d-4e6f-87ce-da84f6bc1684-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-s7225\" (UID: \"00cc7737-711d-4e6f-87ce-da84f6bc1684\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7225"
Sep 11 17:55:37.991813 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.991785    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/00cc7737-711d-4e6f-87ce-da84f6bc1684-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-s7225\" (UID: \"00cc7737-711d-4e6f-87ce-da84f6bc1684\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7225"
Sep 11 17:55:37.992110 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.992083    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00cc7737-711d-4e6f-87ce-da84f6bc1684-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-s7225\" (UID: \"00cc7737-711d-4e6f-87ce-da84f6bc1684\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7225"
Sep 11 17:55:37.998051 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:37.998026    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h58q\" (UniqueName: \"kubernetes.io/projected/00cc7737-711d-4e6f-87ce-da84f6bc1684-kube-api-access-6h58q\") pod \"prometheus-operator-68bcd7f987-s7225\" (UID: \"00cc7737-711d-4e6f-87ce-da84f6bc1684\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7225"
Sep 11 17:55:38.146499 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:38.146409    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7225"
Sep 11 17:55:39.016274 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:39.016189    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-s7225"]
Sep 11 17:55:39.020313 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:55:39.020283    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00cc7737_711d_4e6f_87ce_da84f6bc1684.slice/crio-3c570b6baf379b98dc657bd75b2c9323ba870d166f2858a79f30e8664c3a4ab8 WatchSource:0}: Error finding container 3c570b6baf379b98dc657bd75b2c9323ba870d166f2858a79f30e8664c3a4ab8: Status 404 returned error can't find the container with id 3c570b6baf379b98dc657bd75b2c9323ba870d166f2858a79f30e8664c3a4ab8
Sep 11 17:55:39.175389 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:39.175352    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7225" event={"ID":"00cc7737-711d-4e6f-87ce-da84f6bc1684","Type":"ContainerStarted","Data":"3c570b6baf379b98dc657bd75b2c9323ba870d166f2858a79f30e8664c3a4ab8"}
Sep 11 17:55:39.176783 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:39.176742    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d446b885-2nk2p" event={"ID":"18c95a40-c603-432d-854a-5c95f887a9bf","Type":"ContainerStarted","Data":"08a87f112f3527e4b8f11a956ddbc94d8cbc79bdc9a73970432381be913260cd"}
Sep 11 17:55:39.176981 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:39.176963    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d446b885-2nk2p"
Sep 11 17:55:39.178583 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:39.178565    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d446b885-2nk2p"
Sep 11 17:55:39.195575 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:39.195532    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-54d446b885-2nk2p" podStartSLOduration=1.451798769 podStartE2EDuration="5.195519349s" totalImagesPullingTime="3.74372058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:39.19477737 +0000 UTC m=+97.038012805" watchObservedRunningTime="2026-09-11 17:55:39.195519349 +0000 UTC m=+97.038754791"
Sep 11 17:55:40.911615 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:40.911576    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls\") pod \"dns-default-45lfw\" (UID: \"be400dcd-4cff-4bc9-b572-b714103a5c46\") " pod="openshift-dns/dns-default-45lfw"
Sep 11 17:55:40.914095 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:40.914070    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/be400dcd-4cff-4bc9-b572-b714103a5c46-metrics-tls\") pod \"dns-default-45lfw\" (UID: \"be400dcd-4cff-4bc9-b572-b714103a5c46\") " pod="openshift-dns/dns-default-45lfw"
Sep 11 17:55:40.937569 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:40.937540    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cj7w5\""
Sep 11 17:55:40.945058 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:40.945043    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-45lfw"
Sep 11 17:55:41.185709 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:41.185628    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7225" event={"ID":"00cc7737-711d-4e6f-87ce-da84f6bc1684","Type":"ContainerStarted","Data":"4caaa08a402c8cf51e15da57f98e499abcd0d1dc3c9dd4daa4ad0fa90338aa03"}
Sep 11 17:55:41.185709 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:41.185678    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7225" event={"ID":"00cc7737-711d-4e6f-87ce-da84f6bc1684","Type":"ContainerStarted","Data":"c9675a567b4e29423286707c46711994e48c3b5e5c4a746b968991f956b1a690"}
Sep 11 17:55:41.206267 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:41.206223    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-s7225" podStartSLOduration=2.983525143 podStartE2EDuration="4.206208185s" totalImagesPullingTime="1.222683042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:41.204997562 +0000 UTC m=+99.048233003" watchObservedRunningTime="2026-09-11 17:55:41.206208185 +0000 UTC m=+99.049443651"
Sep 11 17:55:41.214838 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:41.214808    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-45lfw"]
Sep 11 17:55:41.217539 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:55:41.217514    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe400dcd_4cff_4bc9_b572_b714103a5c46.slice/crio-bc5dadd031e95f5724d8e4568da0dd7311b2154175f361adae6354d3ed67c6fa WatchSource:0}: Error finding container bc5dadd031e95f5724d8e4568da0dd7311b2154175f361adae6354d3ed67c6fa: Status 404 returned error can't find the container with id bc5dadd031e95f5724d8e4568da0dd7311b2154175f361adae6354d3ed67c6fa
Sep 11 17:55:42.190071 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:42.190028    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-45lfw" event={"ID":"be400dcd-4cff-4bc9-b572-b714103a5c46","Type":"ContainerStarted","Data":"bc5dadd031e95f5724d8e4568da0dd7311b2154175f361adae6354d3ed67c6fa"}
Sep 11 17:55:43.195676 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.195573    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-45lfw" event={"ID":"be400dcd-4cff-4bc9-b572-b714103a5c46","Type":"ContainerStarted","Data":"dbe921326ad1a937c243a4b5e3e77e6c472d5a5728835753bfd95dc82d8a8051"}
Sep 11 17:55:43.195676 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.195640    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-45lfw" event={"ID":"be400dcd-4cff-4bc9-b572-b714103a5c46","Type":"ContainerStarted","Data":"54b36181b6c7e5d9fdab0b55fcf809ccca913d0d40e28318774d78ae45e94485"}
Sep 11 17:55:43.196178 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.195703    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-45lfw"
Sep 11 17:55:43.238853 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.238794    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-45lfw" podStartSLOduration=65.589696293 podStartE2EDuration="1m7.23877591s" totalImagesPullingTime="1.649079617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:43.237772072 +0000 UTC m=+101.081007510" watchObservedRunningTime="2026-09-11 17:55:43.23877591 +0000 UTC m=+101.082011343"
Sep 11 17:55:43.266635 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.266599    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw"]
Sep 11 17:55:43.269800 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.269778    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw"
Sep 11 17:55:43.273331 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.273312    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rlc7n\""
Sep 11 17:55:43.273911 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.273884    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 11 17:55:43.274002 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.273893    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 11 17:55:43.309556 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.309521    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw"]
Sep 11 17:55:43.329528 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.329495    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6f650bd3-55b5-4762-9ff2-772e71d1e5e7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-kdnvw\" (UID: \"6f650bd3-55b5-4762-9ff2-772e71d1e5e7\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw"
Sep 11 17:55:43.329720 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.329590    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f650bd3-55b5-4762-9ff2-772e71d1e5e7-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-kdnvw\" (UID: \"6f650bd3-55b5-4762-9ff2-772e71d1e5e7\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw"
Sep 11 17:55:43.329720 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.329646    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f650bd3-55b5-4762-9ff2-772e71d1e5e7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-kdnvw\" (UID: \"6f650bd3-55b5-4762-9ff2-772e71d1e5e7\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw"
Sep 11 17:55:43.329848 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.329711    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97j6s\" (UniqueName: \"kubernetes.io/projected/6f650bd3-55b5-4762-9ff2-772e71d1e5e7-kube-api-access-97j6s\") pod \"openshift-state-metrics-67dcf8c979-kdnvw\" (UID: \"6f650bd3-55b5-4762-9ff2-772e71d1e5e7\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw"
Sep 11 17:55:43.336537 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.336507    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p65z7"]
Sep 11 17:55:43.341726 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.341684    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.347446 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.346765    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 17:55:43.347446 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.347274    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 17:55:43.347657 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.347506    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 17:55:43.348374 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.348303    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-97vtp\""
Sep 11 17:55:43.430301 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.430263    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f650bd3-55b5-4762-9ff2-772e71d1e5e7-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-kdnvw\" (UID: \"6f650bd3-55b5-4762-9ff2-772e71d1e5e7\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw"
Sep 11 17:55:43.430451 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.430311    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-metrics-client-ca\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.430451 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.430333    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-sys\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.430451 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.430363    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.430451 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.430402    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f650bd3-55b5-4762-9ff2-772e71d1e5e7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-kdnvw\" (UID: \"6f650bd3-55b5-4762-9ff2-772e71d1e5e7\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw"
Sep 11 17:55:43.430451 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.430425    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-node-exporter-textfile\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.430451 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.430450    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97j6s\" (UniqueName: \"kubernetes.io/projected/6f650bd3-55b5-4762-9ff2-772e71d1e5e7-kube-api-access-97j6s\") pod \"openshift-state-metrics-67dcf8c979-kdnvw\" (UID: \"6f650bd3-55b5-4762-9ff2-772e71d1e5e7\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw"
Sep 11 17:55:43.430782 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.430468    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-root\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.430782 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.430483    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twk56\" (UniqueName: \"kubernetes.io/projected/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-kube-api-access-twk56\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.430782 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.430504    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.430782 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.430553    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6f650bd3-55b5-4762-9ff2-772e71d1e5e7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-kdnvw\" (UID: \"6f650bd3-55b5-4762-9ff2-772e71d1e5e7\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw"
Sep 11 17:55:43.430782 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.430574    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-node-exporter-tls\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.430782 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.430602    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-node-exporter-wtmp\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.430782 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:43.430636    2560 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 11 17:55:43.430782 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:43.430712    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f650bd3-55b5-4762-9ff2-772e71d1e5e7-openshift-state-metrics-tls podName:6f650bd3-55b5-4762-9ff2-772e71d1e5e7 nodeName:}" failed. No retries permitted until 2026-09-11 17:55:43.930691699 +0000 UTC m=+101.773927142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6f650bd3-55b5-4762-9ff2-772e71d1e5e7-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-kdnvw" (UID: "6f650bd3-55b5-4762-9ff2-772e71d1e5e7") : secret "openshift-state-metrics-tls" not found
Sep 11 17:55:43.431059 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.431021    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f650bd3-55b5-4762-9ff2-772e71d1e5e7-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-kdnvw\" (UID: \"6f650bd3-55b5-4762-9ff2-772e71d1e5e7\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw"
Sep 11 17:55:43.433091 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.433068    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6f650bd3-55b5-4762-9ff2-772e71d1e5e7-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-kdnvw\" (UID: \"6f650bd3-55b5-4762-9ff2-772e71d1e5e7\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw"
Sep 11 17:55:43.441201 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.441171    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97j6s\" (UniqueName: \"kubernetes.io/projected/6f650bd3-55b5-4762-9ff2-772e71d1e5e7-kube-api-access-97j6s\") pod \"openshift-state-metrics-67dcf8c979-kdnvw\" (UID: \"6f650bd3-55b5-4762-9ff2-772e71d1e5e7\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw"
Sep 11 17:55:43.531099 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.531064    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-node-exporter-textfile\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.531099 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.531103    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-root\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.531365 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.531125    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twk56\" (UniqueName: \"kubernetes.io/projected/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-kube-api-access-twk56\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.531365 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.531153    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.531365 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.531200    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-root\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.531365 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.531220    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-node-exporter-tls\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.531365 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.531261    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-node-exporter-wtmp\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.531365 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.531318    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-metrics-client-ca\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.531365 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.531347    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-sys\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.531365 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.531363    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.531736 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.531439    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-node-exporter-wtmp\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.531736 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.531487    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-node-exporter-textfile\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.531736 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.531507    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-sys\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.531921 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.531863    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-node-exporter-accelerators-collector-config\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.532009 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.531987    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-metrics-client-ca\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.533652 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.533637    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-node-exporter-tls\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.533811 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.533795    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.540647 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.540622    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twk56\" (UniqueName: \"kubernetes.io/projected/f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6-kube-api-access-twk56\") pod \"node-exporter-p65z7\" (UID: \"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6\") " pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.651724 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.651682    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p65z7"
Sep 11 17:55:43.663788 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:55:43.663738    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70f4cdd_df7e_4704_8c9b_280fc6f8d5c6.slice/crio-5352e69c595e0cd24043924269c6961797449fec53f116a00de1385ec5c6b461 WatchSource:0}: Error finding container 5352e69c595e0cd24043924269c6961797449fec53f116a00de1385ec5c6b461: Status 404 returned error can't find the container with id 5352e69c595e0cd24043924269c6961797449fec53f116a00de1385ec5c6b461
Sep 11 17:55:43.935868 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.935774    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6f650bd3-55b5-4762-9ff2-772e71d1e5e7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-kdnvw\" (UID: \"6f650bd3-55b5-4762-9ff2-772e71d1e5e7\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw"
Sep 11 17:55:43.938368 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:43.938339    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6f650bd3-55b5-4762-9ff2-772e71d1e5e7-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-kdnvw\" (UID: \"6f650bd3-55b5-4762-9ff2-772e71d1e5e7\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw"
Sep 11 17:55:44.178894 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.178853    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw"
Sep 11 17:55:44.202164 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.202110    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p65z7" event={"ID":"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6","Type":"ContainerStarted","Data":"5352e69c595e0cd24043924269c6961797449fec53f116a00de1385ec5c6b461"}
Sep 11 17:55:44.254468 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.249816    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:55:44.262732 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.262685    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.265887 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.265863    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 17:55:44.266567 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.266530    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 17:55:44.266804 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.266788    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 17:55:44.267100 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.266940    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 17:55:44.267286 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.267242    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 17:55:44.267962 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.267442    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 17:55:44.267962 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.267579    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 17:55:44.271097 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.270084    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-l7xsf\""
Sep 11 17:55:44.271097 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.270248    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 17:55:44.271097 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.270642    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 17:55:44.272996 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.272919    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:55:44.342934 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.342533    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.342934 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.342579    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6457x\" (UniqueName: \"kubernetes.io/projected/489228ee-7c94-4f55-9133-055b34d3c5ed-kube-api-access-6457x\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.342934 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.342636    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.342934 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.342665    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/489228ee-7c94-4f55-9133-055b34d3c5ed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.342934 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.342691    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.342934 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.342730    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/489228ee-7c94-4f55-9133-055b34d3c5ed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.342934 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.342795    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/489228ee-7c94-4f55-9133-055b34d3c5ed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.342934 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.342831    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.342934 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.342862    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-config-volume\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.342934 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.342890    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/489228ee-7c94-4f55-9133-055b34d3c5ed-config-out\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.343372 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.342953    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.343372 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.342998    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-web-config\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.343372 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.343038    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/489228ee-7c94-4f55-9133-055b34d3c5ed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.444966 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.443504    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.444966 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.443548    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6457x\" (UniqueName: \"kubernetes.io/projected/489228ee-7c94-4f55-9133-055b34d3c5ed-kube-api-access-6457x\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.444966 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.443589    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.444966 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.443619    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/489228ee-7c94-4f55-9133-055b34d3c5ed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.444966 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.443642    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.444966 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.443675    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/489228ee-7c94-4f55-9133-055b34d3c5ed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.444966 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.443722    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/489228ee-7c94-4f55-9133-055b34d3c5ed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.444966 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.443790    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.444966 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.443818    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-config-volume\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.444966 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.443840    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/489228ee-7c94-4f55-9133-055b34d3c5ed-config-out\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.444966 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:44.444229    2560 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 11 17:55:44.444966 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:44.444305    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-main-tls podName:489228ee-7c94-4f55-9133-055b34d3c5ed nodeName:}" failed. No retries permitted until 2026-09-11 17:55:44.9442834 +0000 UTC m=+102.787518840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "489228ee-7c94-4f55-9133-055b34d3c5ed") : secret "alertmanager-main-tls" not found
Sep 11 17:55:44.444966 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.444592    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/489228ee-7c94-4f55-9133-055b34d3c5ed-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.445803 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.445775    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.445899 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.445820    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-web-config\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.445899 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.445852    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/489228ee-7c94-4f55-9133-055b34d3c5ed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.445899 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.445863    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/489228ee-7c94-4f55-9133-055b34d3c5ed-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.446650 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:55:44.446610    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/489228ee-7c94-4f55-9133-055b34d3c5ed-alertmanager-trusted-ca-bundle podName:489228ee-7c94-4f55-9133-055b34d3c5ed nodeName:}" failed. No retries permitted until 2026-09-11 17:55:44.946592349 +0000 UTC m=+102.789827785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/489228ee-7c94-4f55-9133-055b34d3c5ed-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "489228ee-7c94-4f55-9133-055b34d3c5ed") : configmap references non-existent config key: ca-bundle.crt
Sep 11 17:55:44.447194 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.447169    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/489228ee-7c94-4f55-9133-055b34d3c5ed-config-out\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.451831 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.451784    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-web-config\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.452538 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.452511    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.454233 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.454208    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.454323 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.454270    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.454323 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.454273    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-config-volume\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.454492 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.454476    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/489228ee-7c94-4f55-9133-055b34d3c5ed-tls-assets\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.454854 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.454684    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.460325 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.460301    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6457x\" (UniqueName: \"kubernetes.io/projected/489228ee-7c94-4f55-9133-055b34d3c5ed-kube-api-access-6457x\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.472718 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.472680    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw"]
Sep 11 17:55:44.556638 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:55:44.556605    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f650bd3_55b5_4762_9ff2_772e71d1e5e7.slice/crio-4f5e23ab1bc485c1b7d8dde153713dc3515ce8294a2019b6733d8a77b6b17db4 WatchSource:0}: Error finding container 4f5e23ab1bc485c1b7d8dde153713dc3515ce8294a2019b6733d8a77b6b17db4: Status 404 returned error can't find the container with id 4f5e23ab1bc485c1b7d8dde153713dc3515ce8294a2019b6733d8a77b6b17db4
Sep 11 17:55:44.950117 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.950081    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/489228ee-7c94-4f55-9133-055b34d3c5ed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.950283 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.950138    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.951008 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.950979    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/489228ee-7c94-4f55-9133-055b34d3c5ed-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:44.952802 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:44.952780    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:45.071547 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:45.071478    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xfh2z"
Sep 11 17:55:45.182357 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:45.182196    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:55:45.209273 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:45.209234    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw" event={"ID":"6f650bd3-55b5-4762-9ff2-772e71d1e5e7","Type":"ContainerStarted","Data":"d4416a4d1504520bf94b021226a21984a782321b0c60a06ebd58e7e28bda296d"}
Sep 11 17:55:45.214390 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:45.209315    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw" event={"ID":"6f650bd3-55b5-4762-9ff2-772e71d1e5e7","Type":"ContainerStarted","Data":"1837a33592967d1ba957bdd862dfc5edab5f481f414e93ff53f6adc30d6b701a"}
Sep 11 17:55:45.214390 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:45.209346    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw" event={"ID":"6f650bd3-55b5-4762-9ff2-772e71d1e5e7","Type":"ContainerStarted","Data":"4f5e23ab1bc485c1b7d8dde153713dc3515ce8294a2019b6733d8a77b6b17db4"}
Sep 11 17:55:45.214390 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:45.211253    2560 generic.go:356] "Generic (PLEG): container finished" podID="f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6" containerID="86412b532b35d5a7612591b646a9cf9eb4d7d66737e535b41cd73db21b782c51" exitCode=0
Sep 11 17:55:45.214390 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:45.211320    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p65z7" event={"ID":"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6","Type":"ContainerDied","Data":"86412b532b35d5a7612591b646a9cf9eb4d7d66737e535b41cd73db21b782c51"}
Sep 11 17:55:45.480872 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:45.480839    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:55:45.560389 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:55:45.560352    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489228ee_7c94_4f55_9133_055b34d3c5ed.slice/crio-44d53b232ebd9ca7bf060e022b85c0e34d9ae63109fed9fd0820ace3d178d192 WatchSource:0}: Error finding container 44d53b232ebd9ca7bf060e022b85c0e34d9ae63109fed9fd0820ace3d178d192: Status 404 returned error can't find the container with id 44d53b232ebd9ca7bf060e022b85c0e34d9ae63109fed9fd0820ace3d178d192
Sep 11 17:55:45.682167 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:45.682144    2560 scope.go:117] "RemoveContainer" containerID="87c24ee7f701a1b9a4910aef928ade09cf0b1443f21d2d9a18e5c6d8d0fc460c"
Sep 11 17:55:46.220601 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:46.220496    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw" event={"ID":"6f650bd3-55b5-4762-9ff2-772e71d1e5e7","Type":"ContainerStarted","Data":"ebfc70235719ccc2363aead87fc45ab77a964c7f3e1abae460fb155c8b9a5762"}
Sep 11 17:55:46.223305 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:46.223273    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"489228ee-7c94-4f55-9133-055b34d3c5ed","Type":"ContainerStarted","Data":"44d53b232ebd9ca7bf060e022b85c0e34d9ae63109fed9fd0820ace3d178d192"}
Sep 11 17:55:46.225706 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:46.225683    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/2.log"
Sep 11 17:55:46.225861 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:46.225786    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-k8qz2" event={"ID":"84f48f44-bac5-4f61-bcc5-7dc52f0d1628","Type":"ContainerStarted","Data":"af6287769b6ff9b6e7074e6ca5b1ce03eaa059141fe85f575d6558060fe730ea"}
Sep 11 17:55:46.227952 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:46.227901    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-k8qz2"
Sep 11 17:55:46.232136 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:46.232109    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p65z7" event={"ID":"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6","Type":"ContainerStarted","Data":"e651459655f8373b1cf8a588ca9c81e6878db4c0ad3baaba72bd88591a9b3261"}
Sep 11 17:55:46.232237 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:46.232167    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p65z7" event={"ID":"f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6","Type":"ContainerStarted","Data":"fe80291f65690bff1483258f8ee73087d187ac49e839f533d0043a8331858934"}
Sep 11 17:55:46.246064 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:46.246014    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-kdnvw" podStartSLOduration=2.30948071 podStartE2EDuration="3.245996907s" totalImagesPullingTime="936.516197ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:46.24392441 +0000 UTC m=+104.087159852" watchObservedRunningTime="2026-09-11 17:55:46.245996907 +0000 UTC m=+104.089232351"
Sep 11 17:55:46.264112 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:46.264059    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-k8qz2" podStartSLOduration=43.386809009 podStartE2EDuration="45.264040845s" totalImagesPullingTime="1.877231836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:46.262177808 +0000 UTC m=+104.105413269" watchObservedRunningTime="2026-09-11 17:55:46.264040845 +0000 UTC m=+104.107276290"
Sep 11 17:55:46.290249 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:46.290191    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p65z7" podStartSLOduration=2.283533131 podStartE2EDuration="3.290174505s" totalImagesPullingTime="937.028947ms" totalInitContainerRuntime="69.612427ms" isStatefulPod=true podCreationTimestamp="2026-09-11 17:55:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:46.289978667 +0000 UTC m=+104.133214120" watchObservedRunningTime="2026-09-11 17:55:46.290174505 +0000 UTC m=+104.133409947"
Sep 11 17:55:46.517202 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:46.517125    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-k8qz2"
Sep 11 17:55:46.724405 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:46.724371    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-nhkpk"]
Sep 11 17:55:46.743201 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:46.743170    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-nhkpk"]
Sep 11 17:55:46.743379 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:46.743346    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-nhkpk"
Sep 11 17:55:46.746026 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:46.745999    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-wfhj6\""
Sep 11 17:55:46.746190 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:46.746104    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 17:55:46.746190 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:46.746104    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 17:55:46.868086 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:46.868013    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncxkt\" (UniqueName: \"kubernetes.io/projected/9222f535-e0de-48b8-9026-ff6b962d0e22-kube-api-access-ncxkt\") pod \"downloads-76f99584f4-nhkpk\" (UID: \"9222f535-e0de-48b8-9026-ff6b962d0e22\") " pod="openshift-console/downloads-76f99584f4-nhkpk"
Sep 11 17:55:46.968861 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:46.968826    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncxkt\" (UniqueName: \"kubernetes.io/projected/9222f535-e0de-48b8-9026-ff6b962d0e22-kube-api-access-ncxkt\") pod \"downloads-76f99584f4-nhkpk\" (UID: \"9222f535-e0de-48b8-9026-ff6b962d0e22\") " pod="openshift-console/downloads-76f99584f4-nhkpk"
Sep 11 17:55:46.986235 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:46.986206    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncxkt\" (UniqueName: \"kubernetes.io/projected/9222f535-e0de-48b8-9026-ff6b962d0e22-kube-api-access-ncxkt\") pod \"downloads-76f99584f4-nhkpk\" (UID: \"9222f535-e0de-48b8-9026-ff6b962d0e22\") " pod="openshift-console/downloads-76f99584f4-nhkpk"
Sep 11 17:55:47.054014 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:47.053980    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-nhkpk"
Sep 11 17:55:47.384099 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:47.384031    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-nhkpk"]
Sep 11 17:55:47.387284 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:55:47.387252    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9222f535_e0de_48b8_9026_ff6b962d0e22.slice/crio-4bb2ca6212d4060810e0580772e291c59c4b6f9728bfdc3591e2638f1a062f1d WatchSource:0}: Error finding container 4bb2ca6212d4060810e0580772e291c59c4b6f9728bfdc3591e2638f1a062f1d: Status 404 returned error can't find the container with id 4bb2ca6212d4060810e0580772e291c59c4b6f9728bfdc3591e2638f1a062f1d
Sep 11 17:55:47.801307 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:47.801272    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"]
Sep 11 17:55:47.819882 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:47.819855    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:47.824983 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:47.824952    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 17:55:47.825324 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:47.825237    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4z6vc\""
Sep 11 17:55:47.825535 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:47.825498    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"]
Sep 11 17:55:47.826891 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:47.826696    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 17:55:47.826891 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:47.826709    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 17:55:47.826891 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:47.826728    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6fdaphimf3l3u\""
Sep 11 17:55:47.826891 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:47.826864    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 17:55:47.979552 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:47.979514    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mmlb\" (UniqueName: \"kubernetes.io/projected/2183bc68-cb24-4a71-99f8-40cd8177e13e-kube-api-access-7mmlb\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:47.979724 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:47.979563    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2183bc68-cb24-4a71-99f8-40cd8177e13e-client-ca-bundle\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:47.979724 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:47.979609    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2183bc68-cb24-4a71-99f8-40cd8177e13e-audit-log\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:47.979724 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:47.979635    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2183bc68-cb24-4a71-99f8-40cd8177e13e-metrics-server-audit-profiles\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:47.979724 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:47.979668    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2183bc68-cb24-4a71-99f8-40cd8177e13e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:47.979930 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:47.979742    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2183bc68-cb24-4a71-99f8-40cd8177e13e-secret-metrics-server-client-certs\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:47.979930 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:47.979827    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2183bc68-cb24-4a71-99f8-40cd8177e13e-secret-metrics-server-tls\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:47.999565 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:47.999541    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-kngjs"]
Sep 11 17:55:48.021075 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.021046    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-kngjs"]
Sep 11 17:55:48.021183 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.021149    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kngjs"
Sep 11 17:55:48.024008 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.023976    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-bl2q4\""
Sep 11 17:55:48.024140 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.024123    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 11 17:55:48.080429 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.080344    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2183bc68-cb24-4a71-99f8-40cd8177e13e-secret-metrics-server-tls\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:48.080429 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.080416    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mmlb\" (UniqueName: \"kubernetes.io/projected/2183bc68-cb24-4a71-99f8-40cd8177e13e-kube-api-access-7mmlb\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:48.080650 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.080454    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2183bc68-cb24-4a71-99f8-40cd8177e13e-client-ca-bundle\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:48.080650 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.080500    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2183bc68-cb24-4a71-99f8-40cd8177e13e-audit-log\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:48.080650 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.080521    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2183bc68-cb24-4a71-99f8-40cd8177e13e-metrics-server-audit-profiles\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:48.080650 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.080554    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2183bc68-cb24-4a71-99f8-40cd8177e13e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:48.080650 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.080589    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2183bc68-cb24-4a71-99f8-40cd8177e13e-secret-metrics-server-client-certs\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:48.080952 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.080933    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/2183bc68-cb24-4a71-99f8-40cd8177e13e-audit-log\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:48.081886 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.081861    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2183bc68-cb24-4a71-99f8-40cd8177e13e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:48.081995 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.081892    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/2183bc68-cb24-4a71-99f8-40cd8177e13e-metrics-server-audit-profiles\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:48.083142 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.083123    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/2183bc68-cb24-4a71-99f8-40cd8177e13e-client-ca-bundle\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:48.083230 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.083214    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/2183bc68-cb24-4a71-99f8-40cd8177e13e-secret-metrics-server-client-certs\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:48.083304 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.083286    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/2183bc68-cb24-4a71-99f8-40cd8177e13e-secret-metrics-server-tls\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:48.088854 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.088826    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mmlb\" (UniqueName: \"kubernetes.io/projected/2183bc68-cb24-4a71-99f8-40cd8177e13e-kube-api-access-7mmlb\") pod \"metrics-server-6d8c99dcb-b5j7r\" (UID: \"2183bc68-cb24-4a71-99f8-40cd8177e13e\") " pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:48.129512 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.129488    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:55:48.181548 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.181459    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4fbe05f0-4cdc-4d26-ada7-b23de0eb0dcf-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-kngjs\" (UID: \"4fbe05f0-4cdc-4d26-ada7-b23de0eb0dcf\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kngjs"
Sep 11 17:55:48.181548 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.181535    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5lw2\" (UniqueName: \"kubernetes.io/projected/4fbe05f0-4cdc-4d26-ada7-b23de0eb0dcf-kube-api-access-c5lw2\") pod \"monitoring-plugin-5d85778b9d-kngjs\" (UID: \"4fbe05f0-4cdc-4d26-ada7-b23de0eb0dcf\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kngjs"
Sep 11 17:55:48.246096 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.246025    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-nhkpk" event={"ID":"9222f535-e0de-48b8-9026-ff6b962d0e22","Type":"ContainerStarted","Data":"4bb2ca6212d4060810e0580772e291c59c4b6f9728bfdc3591e2638f1a062f1d"}
Sep 11 17:55:48.247977 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.247950    2560 generic.go:356] "Generic (PLEG): container finished" podID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerID="d8834326f19fb7a47c2ce279defb678792a4d7eecc6e1ff4000ef8413503ff88" exitCode=0
Sep 11 17:55:48.248113 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.248016    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"489228ee-7c94-4f55-9133-055b34d3c5ed","Type":"ContainerDied","Data":"d8834326f19fb7a47c2ce279defb678792a4d7eecc6e1ff4000ef8413503ff88"}
Sep 11 17:55:48.282830 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.282792    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4fbe05f0-4cdc-4d26-ada7-b23de0eb0dcf-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-kngjs\" (UID: \"4fbe05f0-4cdc-4d26-ada7-b23de0eb0dcf\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kngjs"
Sep 11 17:55:48.283032 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.282856    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5lw2\" (UniqueName: \"kubernetes.io/projected/4fbe05f0-4cdc-4d26-ada7-b23de0eb0dcf-kube-api-access-c5lw2\") pod \"monitoring-plugin-5d85778b9d-kngjs\" (UID: \"4fbe05f0-4cdc-4d26-ada7-b23de0eb0dcf\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kngjs"
Sep 11 17:55:48.285960 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.285933    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4fbe05f0-4cdc-4d26-ada7-b23de0eb0dcf-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-kngjs\" (UID: \"4fbe05f0-4cdc-4d26-ada7-b23de0eb0dcf\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kngjs"
Sep 11 17:55:48.298612 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.298583    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5lw2\" (UniqueName: \"kubernetes.io/projected/4fbe05f0-4cdc-4d26-ada7-b23de0eb0dcf-kube-api-access-c5lw2\") pod \"monitoring-plugin-5d85778b9d-kngjs\" (UID: \"4fbe05f0-4cdc-4d26-ada7-b23de0eb0dcf\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kngjs"
Sep 11 17:55:48.330569 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.330478    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kngjs"
Sep 11 17:55:48.440147 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.440124    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"]
Sep 11 17:55:48.442560 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:55:48.442528    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2183bc68_cb24_4a71_99f8_40cd8177e13e.slice/crio-e9142a258d206bee38c55ea3e230a71cc61f5b2fc19b55bed2ff9c5e411611eb WatchSource:0}: Error finding container e9142a258d206bee38c55ea3e230a71cc61f5b2fc19b55bed2ff9c5e411611eb: Status 404 returned error can't find the container with id e9142a258d206bee38c55ea3e230a71cc61f5b2fc19b55bed2ff9c5e411611eb
Sep 11 17:55:48.631799 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:48.631692    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-kngjs"]
Sep 11 17:55:48.635952 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:55:48.635924    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbe05f0_4cdc_4d26_ada7_b23de0eb0dcf.slice/crio-7ee8cca6e828c539dc2fdf6f683f671dc706d641112d3eb9c429a4e2b6556a06 WatchSource:0}: Error finding container 7ee8cca6e828c539dc2fdf6f683f671dc706d641112d3eb9c429a4e2b6556a06: Status 404 returned error can't find the container with id 7ee8cca6e828c539dc2fdf6f683f671dc706d641112d3eb9c429a4e2b6556a06
Sep 11 17:55:49.255510 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:49.255406    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r" event={"ID":"2183bc68-cb24-4a71-99f8-40cd8177e13e","Type":"ContainerStarted","Data":"e9142a258d206bee38c55ea3e230a71cc61f5b2fc19b55bed2ff9c5e411611eb"}
Sep 11 17:55:49.257223 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:49.257192    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kngjs" event={"ID":"4fbe05f0-4cdc-4d26-ada7-b23de0eb0dcf","Type":"ContainerStarted","Data":"7ee8cca6e828c539dc2fdf6f683f671dc706d641112d3eb9c429a4e2b6556a06"}
Sep 11 17:55:51.269855 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:51.269629    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kngjs" event={"ID":"4fbe05f0-4cdc-4d26-ada7-b23de0eb0dcf","Type":"ContainerStarted","Data":"e1a91e724e803bc6dda5c4c8ba52999bbf036385a5bfe3ed7518b69a1222a4f8"}
Sep 11 17:55:51.270239 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:51.270151    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kngjs"
Sep 11 17:55:51.272768 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:51.272724    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r" event={"ID":"2183bc68-cb24-4a71-99f8-40cd8177e13e","Type":"ContainerStarted","Data":"2f27b154822d61efacebf3d00de93929054658647d0d6da40f1654d3fd6ac35f"}
Sep 11 17:55:51.275074 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:51.275047    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"489228ee-7c94-4f55-9133-055b34d3c5ed","Type":"ContainerStarted","Data":"7dda3b41dcf7703fb5248e41377f379e78764f25024974230dd7ac302db4e5ad"}
Sep 11 17:55:51.276192 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:51.276173    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kngjs"
Sep 11 17:55:51.287034 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:51.286989    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-kngjs" podStartSLOduration=1.802032202 podStartE2EDuration="4.286972005s" totalImagesPullingTime="2.484939803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:51.286385753 +0000 UTC m=+109.129621207" watchObservedRunningTime="2026-09-11 17:55:51.286972005 +0000 UTC m=+109.130207447"
Sep 11 17:55:51.327071 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:51.326506    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r" podStartSLOduration=1.649536187 podStartE2EDuration="4.326482939s" totalImagesPullingTime="2.676946752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:51.326290104 +0000 UTC m=+109.169525548" watchObservedRunningTime="2026-09-11 17:55:51.326482939 +0000 UTC m=+109.169718381"
Sep 11 17:55:52.284213 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:52.284180    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"489228ee-7c94-4f55-9133-055b34d3c5ed","Type":"ContainerStarted","Data":"844f88c2ac4e5f7da38b56caee39ac6dadeb63c0298c5f5ac5e8b864d1a8a935"}
Sep 11 17:55:52.284662 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:52.284264    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"489228ee-7c94-4f55-9133-055b34d3c5ed","Type":"ContainerStarted","Data":"509b4ce8f7dcca23204135c983096073625ef6ddda40bded02a19a49c6fc5e08"}
Sep 11 17:55:52.284662 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:52.284307    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"489228ee-7c94-4f55-9133-055b34d3c5ed","Type":"ContainerStarted","Data":"768ee5b8b6a9cb542b472e6755c4a8229e3c27775e0cfe3ab17eb22f5fc65943"}
Sep 11 17:55:52.284662 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:52.284351    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"489228ee-7c94-4f55-9133-055b34d3c5ed","Type":"ContainerStarted","Data":"d3ee61d6eaffa746351d4b782032c3429ed77ccd621191aab1b1aa578244d84e"}
Sep 11 17:55:53.204769 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:53.204719    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-45lfw"
Sep 11 17:55:53.293688 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:53.293649    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"489228ee-7c94-4f55-9133-055b34d3c5ed","Type":"ContainerStarted","Data":"e2f4d60e236e9937f2c8ac88fa4efe3e19b0d993acd8393ba0df0fdb4e84498e"}
Sep 11 17:55:53.321852 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:53.321793    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.044051833 podStartE2EDuration="9.321771506s" totalImagesPullingTime="5.247404509s" totalInitContainerRuntime="30.315164ms" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:55:53.320414839 +0000 UTC m=+111.163650292" watchObservedRunningTime="2026-09-11 17:55:53.321771506 +0000 UTC m=+111.165006946"
Sep 11 17:55:55.871382 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:55.871345    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d758cbfd6-nhqs4"]
Sep 11 17:55:55.873793 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:55.873765    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:55:55.876899 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:55.876873    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 17:55:55.877244 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:55.877221    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-n9trt\""
Sep 11 17:55:55.877997 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:55.877971    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 17:55:55.878101 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:55.878029    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 17:55:55.878101 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:55.878096    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 17:55:55.878232 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:55.878096    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 17:55:55.886091 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:55.886070    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d758cbfd6-nhqs4"]
Sep 11 17:55:55.963226 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:55.963195    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76h9v\" (UniqueName: \"kubernetes.io/projected/334845ea-f617-45a3-bbc4-e3370e941451-kube-api-access-76h9v\") pod \"console-d758cbfd6-nhqs4\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") " pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:55:55.963399 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:55.963239    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/334845ea-f617-45a3-bbc4-e3370e941451-console-config\") pod \"console-d758cbfd6-nhqs4\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") " pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:55:55.963399 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:55.963266    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/334845ea-f617-45a3-bbc4-e3370e941451-service-ca\") pod \"console-d758cbfd6-nhqs4\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") " pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:55:55.963399 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:55.963314    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/334845ea-f617-45a3-bbc4-e3370e941451-oauth-serving-cert\") pod \"console-d758cbfd6-nhqs4\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") " pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:55:55.963561 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:55.963446    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/334845ea-f617-45a3-bbc4-e3370e941451-console-oauth-config\") pod \"console-d758cbfd6-nhqs4\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") " pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:55:55.963561 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:55.963501    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/334845ea-f617-45a3-bbc4-e3370e941451-console-serving-cert\") pod \"console-d758cbfd6-nhqs4\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") " pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:55:56.064957 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:56.064925    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/334845ea-f617-45a3-bbc4-e3370e941451-console-oauth-config\") pod \"console-d758cbfd6-nhqs4\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") " pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:55:56.065134 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:56.064982    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/334845ea-f617-45a3-bbc4-e3370e941451-console-serving-cert\") pod \"console-d758cbfd6-nhqs4\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") " pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:55:56.065134 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:56.065058    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76h9v\" (UniqueName: \"kubernetes.io/projected/334845ea-f617-45a3-bbc4-e3370e941451-kube-api-access-76h9v\") pod \"console-d758cbfd6-nhqs4\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") " pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:55:56.065134 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:56.065087    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/334845ea-f617-45a3-bbc4-e3370e941451-console-config\") pod \"console-d758cbfd6-nhqs4\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") " pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:55:56.065134 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:56.065108    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/334845ea-f617-45a3-bbc4-e3370e941451-service-ca\") pod \"console-d758cbfd6-nhqs4\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") " pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:55:56.065357 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:56.065158    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/334845ea-f617-45a3-bbc4-e3370e941451-oauth-serving-cert\") pod \"console-d758cbfd6-nhqs4\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") " pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:55:56.065912 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:56.065880    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/334845ea-f617-45a3-bbc4-e3370e941451-console-config\") pod \"console-d758cbfd6-nhqs4\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") " pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:55:56.066041 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:56.065945    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/334845ea-f617-45a3-bbc4-e3370e941451-oauth-serving-cert\") pod \"console-d758cbfd6-nhqs4\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") " pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:55:56.066041 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:56.065989    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/334845ea-f617-45a3-bbc4-e3370e941451-service-ca\") pod \"console-d758cbfd6-nhqs4\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") " pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:55:56.068017 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:56.067987    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/334845ea-f617-45a3-bbc4-e3370e941451-console-serving-cert\") pod \"console-d758cbfd6-nhqs4\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") " pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:55:56.068220 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:56.068204    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/334845ea-f617-45a3-bbc4-e3370e941451-console-oauth-config\") pod \"console-d758cbfd6-nhqs4\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") " pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:55:56.082286 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:56.082261    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76h9v\" (UniqueName: \"kubernetes.io/projected/334845ea-f617-45a3-bbc4-e3370e941451-kube-api-access-76h9v\") pod \"console-d758cbfd6-nhqs4\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") " pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:55:56.185288 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:56.185211    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:55:56.468100 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:56.467859    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d758cbfd6-nhqs4"]
Sep 11 17:55:56.470186 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:55:56.470164    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334845ea_f617_45a3_bbc4_e3370e941451.slice/crio-8d385b5e6bac0a51ea66d6d56c1f69cfc88eefcdb02bcdffc6f39660309dd23c WatchSource:0}: Error finding container 8d385b5e6bac0a51ea66d6d56c1f69cfc88eefcdb02bcdffc6f39660309dd23c: Status 404 returned error can't find the container with id 8d385b5e6bac0a51ea66d6d56c1f69cfc88eefcdb02bcdffc6f39660309dd23c
Sep 11 17:55:57.317140 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:55:57.317052    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d758cbfd6-nhqs4" event={"ID":"334845ea-f617-45a3-bbc4-e3370e941451","Type":"ContainerStarted","Data":"8d385b5e6bac0a51ea66d6d56c1f69cfc88eefcdb02bcdffc6f39660309dd23c"}
Sep 11 17:56:02.823960 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:02.823922    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c5c7bc68d-gvr8q"]
Sep 11 17:56:02.827260 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:02.827238    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:02.837036 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:02.836380    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 17:56:02.838674 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:02.838650    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c5c7bc68d-gvr8q"]
Sep 11 17:56:02.928702 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:02.928658    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-trusted-ca-bundle\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:02.928877 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:02.928717    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-console-config\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:02.928877 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:02.928774    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/130c74da-fa1e-4951-bfee-4b14dcc15939-console-serving-cert\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:02.928877 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:02.928801    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/130c74da-fa1e-4951-bfee-4b14dcc15939-console-oauth-config\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:02.928877 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:02.928867    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjzw6\" (UniqueName: \"kubernetes.io/projected/130c74da-fa1e-4951-bfee-4b14dcc15939-kube-api-access-gjzw6\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:02.929060 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:02.928894    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-service-ca\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:02.929060 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:02.928917    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-oauth-serving-cert\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:03.029945 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:03.029909    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjzw6\" (UniqueName: \"kubernetes.io/projected/130c74da-fa1e-4951-bfee-4b14dcc15939-kube-api-access-gjzw6\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:03.030120 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:03.029950    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-service-ca\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:03.030120 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:03.029995    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-oauth-serving-cert\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:03.030120 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:03.030071    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-trusted-ca-bundle\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:03.030120 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:03.030103    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-console-config\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:03.030314 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:03.030136    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/130c74da-fa1e-4951-bfee-4b14dcc15939-console-serving-cert\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:03.030314 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:03.030164    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/130c74da-fa1e-4951-bfee-4b14dcc15939-console-oauth-config\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:03.030885 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:03.030854    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-oauth-serving-cert\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:03.031171 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:03.030997    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-console-config\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:03.031171 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:03.031056    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-service-ca\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:03.031287 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:03.031165    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-trusted-ca-bundle\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:03.033052 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:03.033028    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/130c74da-fa1e-4951-bfee-4b14dcc15939-console-oauth-config\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:03.033322 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:03.033303    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/130c74da-fa1e-4951-bfee-4b14dcc15939-console-serving-cert\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:03.039483 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:03.039462    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjzw6\" (UniqueName: \"kubernetes.io/projected/130c74da-fa1e-4951-bfee-4b14dcc15939-kube-api-access-gjzw6\") pod \"console-5c5c7bc68d-gvr8q\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") " pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:03.140800 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:03.140714    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:06.881350 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:06.881321    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c5c7bc68d-gvr8q"]
Sep 11 17:56:06.882327 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:56:06.882305    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130c74da_fa1e_4951_bfee_4b14dcc15939.slice/crio-dc6b9179c2fb9ffdaf10cd0ac7aca44f1e20cb325ca9a4331360322710a54008 WatchSource:0}: Error finding container dc6b9179c2fb9ffdaf10cd0ac7aca44f1e20cb325ca9a4331360322710a54008: Status 404 returned error can't find the container with id dc6b9179c2fb9ffdaf10cd0ac7aca44f1e20cb325ca9a4331360322710a54008
Sep 11 17:56:07.364397 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:07.364360    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-nhkpk" event={"ID":"9222f535-e0de-48b8-9026-ff6b962d0e22","Type":"ContainerStarted","Data":"091595f28325ee81bead34a60e7b66b004939d7ec06fd822413c41c08822ffb2"}
Sep 11 17:56:07.364671 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:07.364647    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-nhkpk"
Sep 11 17:56:07.366428 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:07.366370    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d758cbfd6-nhqs4" event={"ID":"334845ea-f617-45a3-bbc4-e3370e941451","Type":"ContainerStarted","Data":"236a65cf1c38bb6c8cf2e98a1a0ae06a90af151d531df2ee6b9003f852f3426b"}
Sep 11 17:56:07.368463 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:07.368435    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5c7bc68d-gvr8q" event={"ID":"130c74da-fa1e-4951-bfee-4b14dcc15939","Type":"ContainerStarted","Data":"2ef90e1a97c1ba414bd721b35209809d72ca2515bc4ddfad787ef620577e6676"}
Sep 11 17:56:07.368577 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:07.368479    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5c7bc68d-gvr8q" event={"ID":"130c74da-fa1e-4951-bfee-4b14dcc15939","Type":"ContainerStarted","Data":"dc6b9179c2fb9ffdaf10cd0ac7aca44f1e20cb325ca9a4331360322710a54008"}
Sep 11 17:56:07.383657 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:07.383580    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-nhkpk" podStartSLOduration=2.166585155 podStartE2EDuration="21.383562066s" totalImagesPullingTime="19.216976911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:56:07.382370297 +0000 UTC m=+125.225605742" watchObservedRunningTime="2026-09-11 17:56:07.383562066 +0000 UTC m=+125.226797508"
Sep 11 17:56:07.384440 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:07.384413    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-nhkpk"
Sep 11 17:56:07.398599 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:07.398544    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d758cbfd6-nhqs4" podStartSLOduration=2.268713983 podStartE2EDuration="12.398529738s" totalImagesPullingTime="10.129815755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:56:07.398036574 +0000 UTC m=+125.241272015" watchObservedRunningTime="2026-09-11 17:56:07.398529738 +0000 UTC m=+125.241765179"
Sep 11 17:56:07.416522 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:07.416464    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c5c7bc68d-gvr8q" podStartSLOduration=5.416446071 podStartE2EDuration="5.416446071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:56:07.414883078 +0000 UTC m=+125.258118518" watchObservedRunningTime="2026-09-11 17:56:07.416446071 +0000 UTC m=+125.259681512"
Sep 11 17:56:08.129709 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:08.129654    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:56:08.130181 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:08.129801    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:56:12.423364 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:12.423322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs\") pod \"network-metrics-daemon-8xg8d\" (UID: \"8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf\") " pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:56:12.426051 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:12.426019    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf-metrics-certs\") pod \"network-metrics-daemon-8xg8d\" (UID: \"8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf\") " pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:56:12.505393 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:12.505357    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-64859\""
Sep 11 17:56:12.512649 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:12.512615    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8xg8d"
Sep 11 17:56:12.817353 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:12.817318    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8xg8d"]
Sep 11 17:56:12.821541 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:56:12.821514    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8be05bfc_59ee_43d6_a7cf_cafea9f9a7cf.slice/crio-d03675148fe8418931a32ad3a22234261b88ffc68233ddef0d169509d24ece5b WatchSource:0}: Error finding container d03675148fe8418931a32ad3a22234261b88ffc68233ddef0d169509d24ece5b: Status 404 returned error can't find the container with id d03675148fe8418931a32ad3a22234261b88ffc68233ddef0d169509d24ece5b
Sep 11 17:56:13.141563 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:13.141488    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:13.141961 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:13.141936    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:13.146969 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:13.146945    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:13.399016 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:13.398889    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8xg8d" event={"ID":"8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf","Type":"ContainerStarted","Data":"d03675148fe8418931a32ad3a22234261b88ffc68233ddef0d169509d24ece5b"}
Sep 11 17:56:13.404069 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:13.403962    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:56:13.454686 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:13.454643    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d758cbfd6-nhqs4"]
Sep 11 17:56:15.413152 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:15.413107    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8xg8d" event={"ID":"8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf","Type":"ContainerStarted","Data":"2922c24a35901841313dff2a85910559f3d42b210096026d5b9b52c5de108acb"}
Sep 11 17:56:15.413574 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:15.413178    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8xg8d" event={"ID":"8be05bfc-59ee-43d6-a7cf-cafea9f9a7cf","Type":"ContainerStarted","Data":"34bf700bef87720e5ada907eee4e47d5ae306ecbee4be97463eaa71cd5fc9dae"}
Sep 11 17:56:15.432452 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:15.432407    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8xg8d" podStartSLOduration=130.839786415 podStartE2EDuration="2m12.432391825s" totalImagesPullingTime="1.59260541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:54:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:56:15.431490116 +0000 UTC m=+133.274725555" watchObservedRunningTime="2026-09-11 17:56:15.432391825 +0000 UTC m=+133.275627248"
Sep 11 17:56:16.185705 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:16.185671    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:56:17.422940 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:17.422857    2560 generic.go:356] "Generic (PLEG): container finished" podID="05d87087-b55d-48cd-863f-1b1e6e7e4e72" containerID="b162a5bcfe4002bd01aaa754106ba2800588023035dfed5de06c70e7f7a4731f" exitCode=0
Sep 11 17:56:17.422940 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:17.422921    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-26crr" event={"ID":"05d87087-b55d-48cd-863f-1b1e6e7e4e72","Type":"ContainerDied","Data":"b162a5bcfe4002bd01aaa754106ba2800588023035dfed5de06c70e7f7a4731f"}
Sep 11 17:56:17.423304 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:17.423230    2560 scope.go:117] "RemoveContainer" containerID="b162a5bcfe4002bd01aaa754106ba2800588023035dfed5de06c70e7f7a4731f"
Sep 11 17:56:18.115220 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:18.115165    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dfv62_c3ca3ce1-7383-4588-a129-d69fedda85b3/serve-healthcheck-canary/0.log"
Sep 11 17:56:18.429231 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:18.429135    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-26crr" event={"ID":"05d87087-b55d-48cd-863f-1b1e6e7e4e72","Type":"ContainerStarted","Data":"3d0d423d900094fd6181b78fbdd9b45a12c4f6d9dd471071e176c515fcf8e491"}
Sep 11 17:56:28.135050 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:28.135014    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:56:28.138968 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:28.138944    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6d8c99dcb-b5j7r"
Sep 11 17:56:38.486524 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:38.486484    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d758cbfd6-nhqs4" podUID="334845ea-f617-45a3-bbc4-e3370e941451" containerName="console" containerID="cri-o://236a65cf1c38bb6c8cf2e98a1a0ae06a90af151d531df2ee6b9003f852f3426b" gracePeriod=15
Sep 11 17:56:38.773884 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:38.773858    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:56:38.850432 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:38.850402    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/334845ea-f617-45a3-bbc4-e3370e941451-oauth-serving-cert\") pod \"334845ea-f617-45a3-bbc4-e3370e941451\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") "
Sep 11 17:56:38.850432 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:38.850435    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/334845ea-f617-45a3-bbc4-e3370e941451-console-serving-cert\") pod \"334845ea-f617-45a3-bbc4-e3370e941451\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") "
Sep 11 17:56:38.850662 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:38.850467    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/334845ea-f617-45a3-bbc4-e3370e941451-console-config\") pod \"334845ea-f617-45a3-bbc4-e3370e941451\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") "
Sep 11 17:56:38.850662 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:38.850502    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/334845ea-f617-45a3-bbc4-e3370e941451-console-oauth-config\") pod \"334845ea-f617-45a3-bbc4-e3370e941451\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") "
Sep 11 17:56:38.850662 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:38.850537    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76h9v\" (UniqueName: \"kubernetes.io/projected/334845ea-f617-45a3-bbc4-e3370e941451-kube-api-access-76h9v\") pod \"334845ea-f617-45a3-bbc4-e3370e941451\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") "
Sep 11 17:56:38.850662 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:38.850619    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/334845ea-f617-45a3-bbc4-e3370e941451-service-ca\") pod \"334845ea-f617-45a3-bbc4-e3370e941451\" (UID: \"334845ea-f617-45a3-bbc4-e3370e941451\") "
Sep 11 17:56:38.850880 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:38.850841    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334845ea-f617-45a3-bbc4-e3370e941451-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "334845ea-f617-45a3-bbc4-e3370e941451" (UID: "334845ea-f617-45a3-bbc4-e3370e941451"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:56:38.850942 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:38.850920    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334845ea-f617-45a3-bbc4-e3370e941451-console-config" (OuterVolumeSpecName: "console-config") pod "334845ea-f617-45a3-bbc4-e3370e941451" (UID: "334845ea-f617-45a3-bbc4-e3370e941451"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:56:38.851075 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:38.851053    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334845ea-f617-45a3-bbc4-e3370e941451-service-ca" (OuterVolumeSpecName: "service-ca") pod "334845ea-f617-45a3-bbc4-e3370e941451" (UID: "334845ea-f617-45a3-bbc4-e3370e941451"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:56:38.852840 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:38.852810    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/334845ea-f617-45a3-bbc4-e3370e941451-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "334845ea-f617-45a3-bbc4-e3370e941451" (UID: "334845ea-f617-45a3-bbc4-e3370e941451"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:56:38.852840 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:38.852834    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/334845ea-f617-45a3-bbc4-e3370e941451-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "334845ea-f617-45a3-bbc4-e3370e941451" (UID: "334845ea-f617-45a3-bbc4-e3370e941451"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:56:38.852974 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:38.852846    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334845ea-f617-45a3-bbc4-e3370e941451-kube-api-access-76h9v" (OuterVolumeSpecName: "kube-api-access-76h9v") pod "334845ea-f617-45a3-bbc4-e3370e941451" (UID: "334845ea-f617-45a3-bbc4-e3370e941451"). InnerVolumeSpecName "kube-api-access-76h9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:56:38.951167 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:38.951133    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/334845ea-f617-45a3-bbc4-e3370e941451-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:56:38.951167 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:38.951158    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/334845ea-f617-45a3-bbc4-e3370e941451-oauth-serving-cert\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:56:38.951167 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:38.951167    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/334845ea-f617-45a3-bbc4-e3370e941451-console-serving-cert\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:56:38.951167 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:38.951177    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/334845ea-f617-45a3-bbc4-e3370e941451-console-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:56:38.951406 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:38.951185    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/334845ea-f617-45a3-bbc4-e3370e941451-console-oauth-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:56:38.951406 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:38.951193    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76h9v\" (UniqueName: \"kubernetes.io/projected/334845ea-f617-45a3-bbc4-e3370e941451-kube-api-access-76h9v\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:56:39.522020 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:39.521982    2560 generic.go:356] "Generic (PLEG): container finished" podID="334845ea-f617-45a3-bbc4-e3370e941451" containerID="236a65cf1c38bb6c8cf2e98a1a0ae06a90af151d531df2ee6b9003f852f3426b" exitCode=0
Sep 11 17:56:39.522429 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:39.522024    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d758cbfd6-nhqs4"
Sep 11 17:56:39.522429 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:39.522068    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d758cbfd6-nhqs4" event={"ID":"334845ea-f617-45a3-bbc4-e3370e941451","Type":"ContainerDied","Data":"236a65cf1c38bb6c8cf2e98a1a0ae06a90af151d531df2ee6b9003f852f3426b"}
Sep 11 17:56:39.522429 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:39.522115    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d758cbfd6-nhqs4" event={"ID":"334845ea-f617-45a3-bbc4-e3370e941451","Type":"ContainerDied","Data":"8d385b5e6bac0a51ea66d6d56c1f69cfc88eefcdb02bcdffc6f39660309dd23c"}
Sep 11 17:56:39.522429 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:39.522138    2560 scope.go:117] "RemoveContainer" containerID="236a65cf1c38bb6c8cf2e98a1a0ae06a90af151d531df2ee6b9003f852f3426b"
Sep 11 17:56:39.531672 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:39.531654    2560 scope.go:117] "RemoveContainer" containerID="236a65cf1c38bb6c8cf2e98a1a0ae06a90af151d531df2ee6b9003f852f3426b"
Sep 11 17:56:39.531981 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:56:39.531961    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"236a65cf1c38bb6c8cf2e98a1a0ae06a90af151d531df2ee6b9003f852f3426b\": container with ID starting with 236a65cf1c38bb6c8cf2e98a1a0ae06a90af151d531df2ee6b9003f852f3426b not found: ID does not exist" containerID="236a65cf1c38bb6c8cf2e98a1a0ae06a90af151d531df2ee6b9003f852f3426b"
Sep 11 17:56:39.532036 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:39.531992    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"236a65cf1c38bb6c8cf2e98a1a0ae06a90af151d531df2ee6b9003f852f3426b"} err="failed to get container status \"236a65cf1c38bb6c8cf2e98a1a0ae06a90af151d531df2ee6b9003f852f3426b\": rpc error: code = NotFound desc = could not find container \"236a65cf1c38bb6c8cf2e98a1a0ae06a90af151d531df2ee6b9003f852f3426b\": container with ID starting with 236a65cf1c38bb6c8cf2e98a1a0ae06a90af151d531df2ee6b9003f852f3426b not found: ID does not exist"
Sep 11 17:56:39.549454 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:39.549429    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d758cbfd6-nhqs4"]
Sep 11 17:56:39.550857 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:39.550838    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d758cbfd6-nhqs4"]
Sep 11 17:56:40.686882 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:56:40.686845    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="334845ea-f617-45a3-bbc4-e3370e941451" path="/var/lib/kubelet/pods/334845ea-f617-45a3-bbc4-e3370e941451/volumes"
Sep 11 17:57:03.685405 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:03.685372    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:57:03.685840 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:03.685644    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="alertmanager" containerID="cri-o://7dda3b41dcf7703fb5248e41377f379e78764f25024974230dd7ac302db4e5ad" gracePeriod=120
Sep 11 17:57:03.685840 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:03.685704    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="kube-rbac-proxy-web" containerID="cri-o://768ee5b8b6a9cb542b472e6755c4a8229e3c27775e0cfe3ab17eb22f5fc65943" gracePeriod=120
Sep 11 17:57:03.685840 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:03.685716    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="kube-rbac-proxy-metric" containerID="cri-o://844f88c2ac4e5f7da38b56caee39ac6dadeb63c0298c5f5ac5e8b864d1a8a935" gracePeriod=120
Sep 11 17:57:03.685840 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:03.685722    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="kube-rbac-proxy" containerID="cri-o://509b4ce8f7dcca23204135c983096073625ef6ddda40bded02a19a49c6fc5e08" gracePeriod=120
Sep 11 17:57:03.685840 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:03.685794    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="config-reloader" containerID="cri-o://d3ee61d6eaffa746351d4b782032c3429ed77ccd621191aab1b1aa578244d84e" gracePeriod=120
Sep 11 17:57:03.686092 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:03.685810    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="prom-label-proxy" containerID="cri-o://e2f4d60e236e9937f2c8ac88fa4efe3e19b0d993acd8393ba0df0fdb4e84498e" gracePeriod=120
Sep 11 17:57:04.628306 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:04.628278    2560 generic.go:356] "Generic (PLEG): container finished" podID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerID="e2f4d60e236e9937f2c8ac88fa4efe3e19b0d993acd8393ba0df0fdb4e84498e" exitCode=0
Sep 11 17:57:04.628306 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:04.628300    2560 generic.go:356] "Generic (PLEG): container finished" podID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerID="509b4ce8f7dcca23204135c983096073625ef6ddda40bded02a19a49c6fc5e08" exitCode=0
Sep 11 17:57:04.628306 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:04.628307    2560 generic.go:356] "Generic (PLEG): container finished" podID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerID="d3ee61d6eaffa746351d4b782032c3429ed77ccd621191aab1b1aa578244d84e" exitCode=0
Sep 11 17:57:04.628306 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:04.628313    2560 generic.go:356] "Generic (PLEG): container finished" podID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerID="7dda3b41dcf7703fb5248e41377f379e78764f25024974230dd7ac302db4e5ad" exitCode=0
Sep 11 17:57:04.628568 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:04.628350    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"489228ee-7c94-4f55-9133-055b34d3c5ed","Type":"ContainerDied","Data":"e2f4d60e236e9937f2c8ac88fa4efe3e19b0d993acd8393ba0df0fdb4e84498e"}
Sep 11 17:57:04.628568 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:04.628416    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"489228ee-7c94-4f55-9133-055b34d3c5ed","Type":"ContainerDied","Data":"509b4ce8f7dcca23204135c983096073625ef6ddda40bded02a19a49c6fc5e08"}
Sep 11 17:57:04.628568 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:04.628444    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"489228ee-7c94-4f55-9133-055b34d3c5ed","Type":"ContainerDied","Data":"d3ee61d6eaffa746351d4b782032c3429ed77ccd621191aab1b1aa578244d84e"}
Sep 11 17:57:04.628568 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:04.628469    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"489228ee-7c94-4f55-9133-055b34d3c5ed","Type":"ContainerDied","Data":"7dda3b41dcf7703fb5248e41377f379e78764f25024974230dd7ac302db4e5ad"}
Sep 11 17:57:04.934204 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:04.934182    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.069740 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.069704    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/489228ee-7c94-4f55-9133-055b34d3c5ed-alertmanager-trusted-ca-bundle\") pod \"489228ee-7c94-4f55-9133-055b34d3c5ed\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") "
Sep 11 17:57:05.069934 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.069780    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-kube-rbac-proxy-web\") pod \"489228ee-7c94-4f55-9133-055b34d3c5ed\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") "
Sep 11 17:57:05.069934 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.069837    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/489228ee-7c94-4f55-9133-055b34d3c5ed-metrics-client-ca\") pod \"489228ee-7c94-4f55-9133-055b34d3c5ed\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") "
Sep 11 17:57:05.069934 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.069871    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6457x\" (UniqueName: \"kubernetes.io/projected/489228ee-7c94-4f55-9133-055b34d3c5ed-kube-api-access-6457x\") pod \"489228ee-7c94-4f55-9133-055b34d3c5ed\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") "
Sep 11 17:57:05.069934 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.069892    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-kube-rbac-proxy\") pod \"489228ee-7c94-4f55-9133-055b34d3c5ed\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") "
Sep 11 17:57:05.069934 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.069911    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/489228ee-7c94-4f55-9133-055b34d3c5ed-tls-assets\") pod \"489228ee-7c94-4f55-9133-055b34d3c5ed\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") "
Sep 11 17:57:05.069934 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.069933    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/489228ee-7c94-4f55-9133-055b34d3c5ed-alertmanager-main-db\") pod \"489228ee-7c94-4f55-9133-055b34d3c5ed\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") "
Sep 11 17:57:05.070238 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.069972    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/489228ee-7c94-4f55-9133-055b34d3c5ed-config-out\") pod \"489228ee-7c94-4f55-9133-055b34d3c5ed\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") "
Sep 11 17:57:05.070238 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.070010    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-web-config\") pod \"489228ee-7c94-4f55-9133-055b34d3c5ed\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") "
Sep 11 17:57:05.070238 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.070037    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-config-volume\") pod \"489228ee-7c94-4f55-9133-055b34d3c5ed\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") "
Sep 11 17:57:05.070238 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.070068    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-cluster-tls-config\") pod \"489228ee-7c94-4f55-9133-055b34d3c5ed\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") "
Sep 11 17:57:05.070238 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.070092    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-main-tls\") pod \"489228ee-7c94-4f55-9133-055b34d3c5ed\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") "
Sep 11 17:57:05.070238 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.070128    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-kube-rbac-proxy-metric\") pod \"489228ee-7c94-4f55-9133-055b34d3c5ed\" (UID: \"489228ee-7c94-4f55-9133-055b34d3c5ed\") "
Sep 11 17:57:05.070238 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.070133    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/489228ee-7c94-4f55-9133-055b34d3c5ed-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "489228ee-7c94-4f55-9133-055b34d3c5ed" (UID: "489228ee-7c94-4f55-9133-055b34d3c5ed"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:57:05.070238 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.070204    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/489228ee-7c94-4f55-9133-055b34d3c5ed-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "489228ee-7c94-4f55-9133-055b34d3c5ed" (UID: "489228ee-7c94-4f55-9133-055b34d3c5ed"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:57:05.070893 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.070693    2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/489228ee-7c94-4f55-9133-055b34d3c5ed-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:57:05.070893 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.070720    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/489228ee-7c94-4f55-9133-055b34d3c5ed-metrics-client-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:57:05.070893 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.070745    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489228ee-7c94-4f55-9133-055b34d3c5ed-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "489228ee-7c94-4f55-9133-055b34d3c5ed" (UID: "489228ee-7c94-4f55-9133-055b34d3c5ed"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:57:05.073580 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.073538    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489228ee-7c94-4f55-9133-055b34d3c5ed-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "489228ee-7c94-4f55-9133-055b34d3c5ed" (UID: "489228ee-7c94-4f55-9133-055b34d3c5ed"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:57:05.073684 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.073620    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "489228ee-7c94-4f55-9133-055b34d3c5ed" (UID: "489228ee-7c94-4f55-9133-055b34d3c5ed"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:57:05.073770 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.073719    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "489228ee-7c94-4f55-9133-055b34d3c5ed" (UID: "489228ee-7c94-4f55-9133-055b34d3c5ed"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:57:05.073770 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.073730    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "489228ee-7c94-4f55-9133-055b34d3c5ed" (UID: "489228ee-7c94-4f55-9133-055b34d3c5ed"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:57:05.074105 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.074081    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-config-volume" (OuterVolumeSpecName: "config-volume") pod "489228ee-7c94-4f55-9133-055b34d3c5ed" (UID: "489228ee-7c94-4f55-9133-055b34d3c5ed"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:57:05.074577 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.074553    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489228ee-7c94-4f55-9133-055b34d3c5ed-kube-api-access-6457x" (OuterVolumeSpecName: "kube-api-access-6457x") pod "489228ee-7c94-4f55-9133-055b34d3c5ed" (UID: "489228ee-7c94-4f55-9133-055b34d3c5ed"). InnerVolumeSpecName "kube-api-access-6457x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:57:05.074725 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.074703    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "489228ee-7c94-4f55-9133-055b34d3c5ed" (UID: "489228ee-7c94-4f55-9133-055b34d3c5ed"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:57:05.075324 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.075295    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/489228ee-7c94-4f55-9133-055b34d3c5ed-config-out" (OuterVolumeSpecName: "config-out") pod "489228ee-7c94-4f55-9133-055b34d3c5ed" (UID: "489228ee-7c94-4f55-9133-055b34d3c5ed"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 17:57:05.077613 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.077592    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "489228ee-7c94-4f55-9133-055b34d3c5ed" (UID: "489228ee-7c94-4f55-9133-055b34d3c5ed"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:57:05.084947 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.084927    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-web-config" (OuterVolumeSpecName: "web-config") pod "489228ee-7c94-4f55-9133-055b34d3c5ed" (UID: "489228ee-7c94-4f55-9133-055b34d3c5ed"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:57:05.171160 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.171073    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:57:05.171160 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.171099    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:57:05.171160 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.171109    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6457x\" (UniqueName: \"kubernetes.io/projected/489228ee-7c94-4f55-9133-055b34d3c5ed-kube-api-access-6457x\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:57:05.171160 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.171119    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:57:05.171160 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.171129    2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/489228ee-7c94-4f55-9133-055b34d3c5ed-tls-assets\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:57:05.171160 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.171137    2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/489228ee-7c94-4f55-9133-055b34d3c5ed-alertmanager-main-db\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:57:05.171160 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.171145    2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/489228ee-7c94-4f55-9133-055b34d3c5ed-config-out\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:57:05.171160 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.171153    2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-web-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:57:05.171160 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.171160    2560 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-config-volume\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:57:05.171160 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.171168    2560 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-cluster-tls-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:57:05.171695 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.171175    2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/489228ee-7c94-4f55-9133-055b34d3c5ed-secret-alertmanager-main-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:57:05.635382 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.635351    2560 generic.go:356] "Generic (PLEG): container finished" podID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerID="844f88c2ac4e5f7da38b56caee39ac6dadeb63c0298c5f5ac5e8b864d1a8a935" exitCode=0
Sep 11 17:57:05.635382 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.635376    2560 generic.go:356] "Generic (PLEG): container finished" podID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerID="768ee5b8b6a9cb542b472e6755c4a8229e3c27775e0cfe3ab17eb22f5fc65943" exitCode=0
Sep 11 17:57:05.635586 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.635409    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"489228ee-7c94-4f55-9133-055b34d3c5ed","Type":"ContainerDied","Data":"844f88c2ac4e5f7da38b56caee39ac6dadeb63c0298c5f5ac5e8b864d1a8a935"}
Sep 11 17:57:05.635586 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.635434    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.635586 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.635465    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"489228ee-7c94-4f55-9133-055b34d3c5ed","Type":"ContainerDied","Data":"768ee5b8b6a9cb542b472e6755c4a8229e3c27775e0cfe3ab17eb22f5fc65943"}
Sep 11 17:57:05.635586 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.635492    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"489228ee-7c94-4f55-9133-055b34d3c5ed","Type":"ContainerDied","Data":"44d53b232ebd9ca7bf060e022b85c0e34d9ae63109fed9fd0820ace3d178d192"}
Sep 11 17:57:05.635586 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.635522    2560 scope.go:117] "RemoveContainer" containerID="e2f4d60e236e9937f2c8ac88fa4efe3e19b0d993acd8393ba0df0fdb4e84498e"
Sep 11 17:57:05.647117 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.647088    2560 scope.go:117] "RemoveContainer" containerID="844f88c2ac4e5f7da38b56caee39ac6dadeb63c0298c5f5ac5e8b864d1a8a935"
Sep 11 17:57:05.654406 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.654385    2560 scope.go:117] "RemoveContainer" containerID="509b4ce8f7dcca23204135c983096073625ef6ddda40bded02a19a49c6fc5e08"
Sep 11 17:57:05.661177 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.661157    2560 scope.go:117] "RemoveContainer" containerID="768ee5b8b6a9cb542b472e6755c4a8229e3c27775e0cfe3ab17eb22f5fc65943"
Sep 11 17:57:05.665404 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.665381    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:57:05.669934 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.669907    2560 scope.go:117] "RemoveContainer" containerID="d3ee61d6eaffa746351d4b782032c3429ed77ccd621191aab1b1aa578244d84e"
Sep 11 17:57:05.670396 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.670378    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:57:05.676968 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.676951    2560 scope.go:117] "RemoveContainer" containerID="7dda3b41dcf7703fb5248e41377f379e78764f25024974230dd7ac302db4e5ad"
Sep 11 17:57:05.683822 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.683805    2560 scope.go:117] "RemoveContainer" containerID="d8834326f19fb7a47c2ce279defb678792a4d7eecc6e1ff4000ef8413503ff88"
Sep 11 17:57:05.690735 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.690714    2560 scope.go:117] "RemoveContainer" containerID="e2f4d60e236e9937f2c8ac88fa4efe3e19b0d993acd8393ba0df0fdb4e84498e"
Sep 11 17:57:05.690996 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:57:05.690975    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2f4d60e236e9937f2c8ac88fa4efe3e19b0d993acd8393ba0df0fdb4e84498e\": container with ID starting with e2f4d60e236e9937f2c8ac88fa4efe3e19b0d993acd8393ba0df0fdb4e84498e not found: ID does not exist" containerID="e2f4d60e236e9937f2c8ac88fa4efe3e19b0d993acd8393ba0df0fdb4e84498e"
Sep 11 17:57:05.691067 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.691007    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2f4d60e236e9937f2c8ac88fa4efe3e19b0d993acd8393ba0df0fdb4e84498e"} err="failed to get container status \"e2f4d60e236e9937f2c8ac88fa4efe3e19b0d993acd8393ba0df0fdb4e84498e\": rpc error: code = NotFound desc = could not find container \"e2f4d60e236e9937f2c8ac88fa4efe3e19b0d993acd8393ba0df0fdb4e84498e\": container with ID starting with e2f4d60e236e9937f2c8ac88fa4efe3e19b0d993acd8393ba0df0fdb4e84498e not found: ID does not exist"
Sep 11 17:57:05.691067 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.691033    2560 scope.go:117] "RemoveContainer" containerID="844f88c2ac4e5f7da38b56caee39ac6dadeb63c0298c5f5ac5e8b864d1a8a935"
Sep 11 17:57:05.691300 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:57:05.691281    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"844f88c2ac4e5f7da38b56caee39ac6dadeb63c0298c5f5ac5e8b864d1a8a935\": container with ID starting with 844f88c2ac4e5f7da38b56caee39ac6dadeb63c0298c5f5ac5e8b864d1a8a935 not found: ID does not exist" containerID="844f88c2ac4e5f7da38b56caee39ac6dadeb63c0298c5f5ac5e8b864d1a8a935"
Sep 11 17:57:05.691338 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.691311    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"844f88c2ac4e5f7da38b56caee39ac6dadeb63c0298c5f5ac5e8b864d1a8a935"} err="failed to get container status \"844f88c2ac4e5f7da38b56caee39ac6dadeb63c0298c5f5ac5e8b864d1a8a935\": rpc error: code = NotFound desc = could not find container \"844f88c2ac4e5f7da38b56caee39ac6dadeb63c0298c5f5ac5e8b864d1a8a935\": container with ID starting with 844f88c2ac4e5f7da38b56caee39ac6dadeb63c0298c5f5ac5e8b864d1a8a935 not found: ID does not exist"
Sep 11 17:57:05.691338 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.691329    2560 scope.go:117] "RemoveContainer" containerID="509b4ce8f7dcca23204135c983096073625ef6ddda40bded02a19a49c6fc5e08"
Sep 11 17:57:05.691547 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:57:05.691531    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"509b4ce8f7dcca23204135c983096073625ef6ddda40bded02a19a49c6fc5e08\": container with ID starting with 509b4ce8f7dcca23204135c983096073625ef6ddda40bded02a19a49c6fc5e08 not found: ID does not exist" containerID="509b4ce8f7dcca23204135c983096073625ef6ddda40bded02a19a49c6fc5e08"
Sep 11 17:57:05.691587 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.691549    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509b4ce8f7dcca23204135c983096073625ef6ddda40bded02a19a49c6fc5e08"} err="failed to get container status \"509b4ce8f7dcca23204135c983096073625ef6ddda40bded02a19a49c6fc5e08\": rpc error: code = NotFound desc = could not find container \"509b4ce8f7dcca23204135c983096073625ef6ddda40bded02a19a49c6fc5e08\": container with ID starting with 509b4ce8f7dcca23204135c983096073625ef6ddda40bded02a19a49c6fc5e08 not found: ID does not exist"
Sep 11 17:57:05.691587 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.691564    2560 scope.go:117] "RemoveContainer" containerID="768ee5b8b6a9cb542b472e6755c4a8229e3c27775e0cfe3ab17eb22f5fc65943"
Sep 11 17:57:05.691902 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:57:05.691878    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"768ee5b8b6a9cb542b472e6755c4a8229e3c27775e0cfe3ab17eb22f5fc65943\": container with ID starting with 768ee5b8b6a9cb542b472e6755c4a8229e3c27775e0cfe3ab17eb22f5fc65943 not found: ID does not exist" containerID="768ee5b8b6a9cb542b472e6755c4a8229e3c27775e0cfe3ab17eb22f5fc65943"
Sep 11 17:57:05.692042 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.691906    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"768ee5b8b6a9cb542b472e6755c4a8229e3c27775e0cfe3ab17eb22f5fc65943"} err="failed to get container status \"768ee5b8b6a9cb542b472e6755c4a8229e3c27775e0cfe3ab17eb22f5fc65943\": rpc error: code = NotFound desc = could not find container \"768ee5b8b6a9cb542b472e6755c4a8229e3c27775e0cfe3ab17eb22f5fc65943\": container with ID starting with 768ee5b8b6a9cb542b472e6755c4a8229e3c27775e0cfe3ab17eb22f5fc65943 not found: ID does not exist"
Sep 11 17:57:05.692042 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.691927    2560 scope.go:117] "RemoveContainer" containerID="d3ee61d6eaffa746351d4b782032c3429ed77ccd621191aab1b1aa578244d84e"
Sep 11 17:57:05.692196 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:57:05.692179    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3ee61d6eaffa746351d4b782032c3429ed77ccd621191aab1b1aa578244d84e\": container with ID starting with d3ee61d6eaffa746351d4b782032c3429ed77ccd621191aab1b1aa578244d84e not found: ID does not exist" containerID="d3ee61d6eaffa746351d4b782032c3429ed77ccd621191aab1b1aa578244d84e"
Sep 11 17:57:05.692236 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.692200    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3ee61d6eaffa746351d4b782032c3429ed77ccd621191aab1b1aa578244d84e"} err="failed to get container status \"d3ee61d6eaffa746351d4b782032c3429ed77ccd621191aab1b1aa578244d84e\": rpc error: code = NotFound desc = could not find container \"d3ee61d6eaffa746351d4b782032c3429ed77ccd621191aab1b1aa578244d84e\": container with ID starting with d3ee61d6eaffa746351d4b782032c3429ed77ccd621191aab1b1aa578244d84e not found: ID does not exist"
Sep 11 17:57:05.692236 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.692213    2560 scope.go:117] "RemoveContainer" containerID="7dda3b41dcf7703fb5248e41377f379e78764f25024974230dd7ac302db4e5ad"
Sep 11 17:57:05.692439 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:57:05.692425    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dda3b41dcf7703fb5248e41377f379e78764f25024974230dd7ac302db4e5ad\": container with ID starting with 7dda3b41dcf7703fb5248e41377f379e78764f25024974230dd7ac302db4e5ad not found: ID does not exist" containerID="7dda3b41dcf7703fb5248e41377f379e78764f25024974230dd7ac302db4e5ad"
Sep 11 17:57:05.692482 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.692442    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dda3b41dcf7703fb5248e41377f379e78764f25024974230dd7ac302db4e5ad"} err="failed to get container status \"7dda3b41dcf7703fb5248e41377f379e78764f25024974230dd7ac302db4e5ad\": rpc error: code = NotFound desc = could not find container \"7dda3b41dcf7703fb5248e41377f379e78764f25024974230dd7ac302db4e5ad\": container with ID starting with 7dda3b41dcf7703fb5248e41377f379e78764f25024974230dd7ac302db4e5ad not found: ID does not exist"
Sep 11 17:57:05.692482 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.692453    2560 scope.go:117] "RemoveContainer" containerID="d8834326f19fb7a47c2ce279defb678792a4d7eecc6e1ff4000ef8413503ff88"
Sep 11 17:57:05.692670 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:57:05.692654    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8834326f19fb7a47c2ce279defb678792a4d7eecc6e1ff4000ef8413503ff88\": container with ID starting with d8834326f19fb7a47c2ce279defb678792a4d7eecc6e1ff4000ef8413503ff88 not found: ID does not exist" containerID="d8834326f19fb7a47c2ce279defb678792a4d7eecc6e1ff4000ef8413503ff88"
Sep 11 17:57:05.692730 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.692678    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8834326f19fb7a47c2ce279defb678792a4d7eecc6e1ff4000ef8413503ff88"} err="failed to get container status \"d8834326f19fb7a47c2ce279defb678792a4d7eecc6e1ff4000ef8413503ff88\": rpc error: code = NotFound desc = could not find container \"d8834326f19fb7a47c2ce279defb678792a4d7eecc6e1ff4000ef8413503ff88\": container with ID starting with d8834326f19fb7a47c2ce279defb678792a4d7eecc6e1ff4000ef8413503ff88 not found: ID does not exist"
Sep 11 17:57:05.692730 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.692698    2560 scope.go:117] "RemoveContainer" containerID="e2f4d60e236e9937f2c8ac88fa4efe3e19b0d993acd8393ba0df0fdb4e84498e"
Sep 11 17:57:05.692967 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.692948    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2f4d60e236e9937f2c8ac88fa4efe3e19b0d993acd8393ba0df0fdb4e84498e"} err="failed to get container status \"e2f4d60e236e9937f2c8ac88fa4efe3e19b0d993acd8393ba0df0fdb4e84498e\": rpc error: code = NotFound desc = could not find container \"e2f4d60e236e9937f2c8ac88fa4efe3e19b0d993acd8393ba0df0fdb4e84498e\": container with ID starting with e2f4d60e236e9937f2c8ac88fa4efe3e19b0d993acd8393ba0df0fdb4e84498e not found: ID does not exist"
Sep 11 17:57:05.693011 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.692968    2560 scope.go:117] "RemoveContainer" containerID="844f88c2ac4e5f7da38b56caee39ac6dadeb63c0298c5f5ac5e8b864d1a8a935"
Sep 11 17:57:05.693186 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.693170    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"844f88c2ac4e5f7da38b56caee39ac6dadeb63c0298c5f5ac5e8b864d1a8a935"} err="failed to get container status \"844f88c2ac4e5f7da38b56caee39ac6dadeb63c0298c5f5ac5e8b864d1a8a935\": rpc error: code = NotFound desc = could not find container \"844f88c2ac4e5f7da38b56caee39ac6dadeb63c0298c5f5ac5e8b864d1a8a935\": container with ID starting with 844f88c2ac4e5f7da38b56caee39ac6dadeb63c0298c5f5ac5e8b864d1a8a935 not found: ID does not exist"
Sep 11 17:57:05.693241 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.693187    2560 scope.go:117] "RemoveContainer" containerID="509b4ce8f7dcca23204135c983096073625ef6ddda40bded02a19a49c6fc5e08"
Sep 11 17:57:05.693411 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.693393    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509b4ce8f7dcca23204135c983096073625ef6ddda40bded02a19a49c6fc5e08"} err="failed to get container status \"509b4ce8f7dcca23204135c983096073625ef6ddda40bded02a19a49c6fc5e08\": rpc error: code = NotFound desc = could not find container \"509b4ce8f7dcca23204135c983096073625ef6ddda40bded02a19a49c6fc5e08\": container with ID starting with 509b4ce8f7dcca23204135c983096073625ef6ddda40bded02a19a49c6fc5e08 not found: ID does not exist"
Sep 11 17:57:05.693411 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.693411    2560 scope.go:117] "RemoveContainer" containerID="768ee5b8b6a9cb542b472e6755c4a8229e3c27775e0cfe3ab17eb22f5fc65943"
Sep 11 17:57:05.693608 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.693592    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"768ee5b8b6a9cb542b472e6755c4a8229e3c27775e0cfe3ab17eb22f5fc65943"} err="failed to get container status \"768ee5b8b6a9cb542b472e6755c4a8229e3c27775e0cfe3ab17eb22f5fc65943\": rpc error: code = NotFound desc = could not find container \"768ee5b8b6a9cb542b472e6755c4a8229e3c27775e0cfe3ab17eb22f5fc65943\": container with ID starting with 768ee5b8b6a9cb542b472e6755c4a8229e3c27775e0cfe3ab17eb22f5fc65943 not found: ID does not exist"
Sep 11 17:57:05.693649 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.693608    2560 scope.go:117] "RemoveContainer" containerID="d3ee61d6eaffa746351d4b782032c3429ed77ccd621191aab1b1aa578244d84e"
Sep 11 17:57:05.693826 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.693811    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3ee61d6eaffa746351d4b782032c3429ed77ccd621191aab1b1aa578244d84e"} err="failed to get container status \"d3ee61d6eaffa746351d4b782032c3429ed77ccd621191aab1b1aa578244d84e\": rpc error: code = NotFound desc = could not find container \"d3ee61d6eaffa746351d4b782032c3429ed77ccd621191aab1b1aa578244d84e\": container with ID starting with d3ee61d6eaffa746351d4b782032c3429ed77ccd621191aab1b1aa578244d84e not found: ID does not exist"
Sep 11 17:57:05.693826 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.693826    2560 scope.go:117] "RemoveContainer" containerID="7dda3b41dcf7703fb5248e41377f379e78764f25024974230dd7ac302db4e5ad"
Sep 11 17:57:05.694035 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.694018    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dda3b41dcf7703fb5248e41377f379e78764f25024974230dd7ac302db4e5ad"} err="failed to get container status \"7dda3b41dcf7703fb5248e41377f379e78764f25024974230dd7ac302db4e5ad\": rpc error: code = NotFound desc = could not find container \"7dda3b41dcf7703fb5248e41377f379e78764f25024974230dd7ac302db4e5ad\": container with ID starting with 7dda3b41dcf7703fb5248e41377f379e78764f25024974230dd7ac302db4e5ad not found: ID does not exist"
Sep 11 17:57:05.694035 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.694034    2560 scope.go:117] "RemoveContainer" containerID="d8834326f19fb7a47c2ce279defb678792a4d7eecc6e1ff4000ef8413503ff88"
Sep 11 17:57:05.694260 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.694241    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8834326f19fb7a47c2ce279defb678792a4d7eecc6e1ff4000ef8413503ff88"} err="failed to get container status \"d8834326f19fb7a47c2ce279defb678792a4d7eecc6e1ff4000ef8413503ff88\": rpc error: code = NotFound desc = could not find container \"d8834326f19fb7a47c2ce279defb678792a4d7eecc6e1ff4000ef8413503ff88\": container with ID starting with d8834326f19fb7a47c2ce279defb678792a4d7eecc6e1ff4000ef8413503ff88 not found: ID does not exist"
Sep 11 17:57:05.696600 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.696581    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:57:05.696947 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.696931    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="init-config-reloader"
Sep 11 17:57:05.697029 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.696949    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="init-config-reloader"
Sep 11 17:57:05.697029 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.696965    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="kube-rbac-proxy-metric"
Sep 11 17:57:05.697029 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.696973    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="kube-rbac-proxy-metric"
Sep 11 17:57:05.697029 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.696985    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="334845ea-f617-45a3-bbc4-e3370e941451" containerName="console"
Sep 11 17:57:05.697029 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.696994    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="334845ea-f617-45a3-bbc4-e3370e941451" containerName="console"
Sep 11 17:57:05.697029 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.697007    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="alertmanager"
Sep 11 17:57:05.697029 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.697015    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="alertmanager"
Sep 11 17:57:05.697029 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.697024    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="kube-rbac-proxy-web"
Sep 11 17:57:05.697408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.697032    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="kube-rbac-proxy-web"
Sep 11 17:57:05.697408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.697049    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="kube-rbac-proxy"
Sep 11 17:57:05.697408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.697058    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="kube-rbac-proxy"
Sep 11 17:57:05.697408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.697070    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="config-reloader"
Sep 11 17:57:05.697408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.697079    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="config-reloader"
Sep 11 17:57:05.697408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.697087    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="prom-label-proxy"
Sep 11 17:57:05.697408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.697095    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="prom-label-proxy"
Sep 11 17:57:05.697408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.697170    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="kube-rbac-proxy"
Sep 11 17:57:05.697408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.697182    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="kube-rbac-proxy-metric"
Sep 11 17:57:05.697408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.697192    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="alertmanager"
Sep 11 17:57:05.697408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.697202    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="prom-label-proxy"
Sep 11 17:57:05.697408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.697211    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="config-reloader"
Sep 11 17:57:05.697408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.697218    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="334845ea-f617-45a3-bbc4-e3370e941451" containerName="console"
Sep 11 17:57:05.697408 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.697229    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" containerName="kube-rbac-proxy-web"
Sep 11 17:57:05.699647 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.699630    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.702285 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.702264    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 17:57:05.702489 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.702469    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 17:57:05.702741 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.702724    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 17:57:05.702882 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.702789    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 17:57:05.702962 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.702933    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 17:57:05.702962 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.702939    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-l7xsf\""
Sep 11 17:57:05.703119 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.702953    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 17:57:05.703304 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.703275    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 17:57:05.703473 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.703454    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 17:57:05.709321 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.709301    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 17:57:05.716065 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.716045    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:57:05.775634 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.775608    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.775724 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.775638    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.775724 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.775668    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d878d705-9b00-40f5-85f3-16a887c4192d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.775802 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.775769    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d878d705-9b00-40f5-85f3-16a887c4192d-config-out\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.775839 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.775808    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d878d705-9b00-40f5-85f3-16a887c4192d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.775904 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.775887    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d878d705-9b00-40f5-85f3-16a887c4192d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.775936 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.775918    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-config-volume\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.775968 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.775946    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-web-config\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.776004 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.775970    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxdsc\" (UniqueName: \"kubernetes.io/projected/d878d705-9b00-40f5-85f3-16a887c4192d-kube-api-access-fxdsc\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.776004 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.775996    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.776098 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.776012    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.776098 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.776026    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.776098 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.776046    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d878d705-9b00-40f5-85f3-16a887c4192d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.877080 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.877040    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d878d705-9b00-40f5-85f3-16a887c4192d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.877080 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.877083    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-config-volume\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.877324 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.877117    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-web-config\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.877324 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.877140    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxdsc\" (UniqueName: \"kubernetes.io/projected/d878d705-9b00-40f5-85f3-16a887c4192d-kube-api-access-fxdsc\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.877324 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.877172    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.877324 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.877198    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.877324 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.877220    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.877324 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.877241    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d878d705-9b00-40f5-85f3-16a887c4192d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.877324 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.877269    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.877324 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.877299    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.877699 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.877356    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d878d705-9b00-40f5-85f3-16a887c4192d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.877699 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.877404    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d878d705-9b00-40f5-85f3-16a887c4192d-config-out\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.877699 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.877435    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d878d705-9b00-40f5-85f3-16a887c4192d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.878297 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.878206    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d878d705-9b00-40f5-85f3-16a887c4192d-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.879054 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.878838    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d878d705-9b00-40f5-85f3-16a887c4192d-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.879054 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.878848    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d878d705-9b00-40f5-85f3-16a887c4192d-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.880333 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.880307    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d878d705-9b00-40f5-85f3-16a887c4192d-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.880643 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.880581    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.880643 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.880598    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-web-config\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.880858 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.880698    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.881004 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.880983    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.881073 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.881029    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-config-volume\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.881493 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.881466    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.881960 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.881938    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d878d705-9b00-40f5-85f3-16a887c4192d-config-out\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.882304 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.882282    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d878d705-9b00-40f5-85f3-16a887c4192d-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:05.886343 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:05.886294    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxdsc\" (UniqueName: \"kubernetes.io/projected/d878d705-9b00-40f5-85f3-16a887c4192d-kube-api-access-fxdsc\") pod \"alertmanager-main-0\" (UID: \"d878d705-9b00-40f5-85f3-16a887c4192d\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:06.010620 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:06.010586    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 17:57:06.293577 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:06.293551    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 17:57:06.294837 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:57:06.294811    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd878d705_9b00_40f5_85f3_16a887c4192d.slice/crio-c2e6ac6066fe981111a38159962b8b6b1d12acd5aaef12cd7a76a81ceecd0c91 WatchSource:0}: Error finding container c2e6ac6066fe981111a38159962b8b6b1d12acd5aaef12cd7a76a81ceecd0c91: Status 404 returned error can't find the container with id c2e6ac6066fe981111a38159962b8b6b1d12acd5aaef12cd7a76a81ceecd0c91
Sep 11 17:57:06.641621 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:06.641532    2560 generic.go:356] "Generic (PLEG): container finished" podID="d878d705-9b00-40f5-85f3-16a887c4192d" containerID="7000d4adf92064a7843f92b3f4c48e365db18e3d2bb1aaf9afbae75094640bc8" exitCode=0
Sep 11 17:57:06.641781 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:06.641623    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d878d705-9b00-40f5-85f3-16a887c4192d","Type":"ContainerDied","Data":"7000d4adf92064a7843f92b3f4c48e365db18e3d2bb1aaf9afbae75094640bc8"}
Sep 11 17:57:06.641781 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:06.641694    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d878d705-9b00-40f5-85f3-16a887c4192d","Type":"ContainerStarted","Data":"c2e6ac6066fe981111a38159962b8b6b1d12acd5aaef12cd7a76a81ceecd0c91"}
Sep 11 17:57:06.688235 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:06.688212    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="489228ee-7c94-4f55-9133-055b34d3c5ed" path="/var/lib/kubelet/pods/489228ee-7c94-4f55-9133-055b34d3c5ed/volumes"
Sep 11 17:57:07.650158 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.650119    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d878d705-9b00-40f5-85f3-16a887c4192d","Type":"ContainerStarted","Data":"7a1ff2c7330a2c94b59b91b78727be9a588dc1ea3a488cce46a6b77e0573abb4"}
Sep 11 17:57:07.650570 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.650239    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d878d705-9b00-40f5-85f3-16a887c4192d","Type":"ContainerStarted","Data":"99b10ae075685b2334cab63bb38a7293f05695a702ec93e7df51658b09161c65"}
Sep 11 17:57:07.650570 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.650288    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d878d705-9b00-40f5-85f3-16a887c4192d","Type":"ContainerStarted","Data":"ebd0348d8f2b91905c0b6d48a24202b1e923e6cf3dff370e27f7c45e5e41b7a5"}
Sep 11 17:57:07.650570 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.650324    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d878d705-9b00-40f5-85f3-16a887c4192d","Type":"ContainerStarted","Data":"72cf51984d37376025b7b6ba37d74480551686753c52150e1d46a32001a7ec5d"}
Sep 11 17:57:07.650570 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.650359    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d878d705-9b00-40f5-85f3-16a887c4192d","Type":"ContainerStarted","Data":"f4e00a7b6e946f9673cef3df03443841a811da69e3875a6ef26ece0fc7f320b2"}
Sep 11 17:57:07.650570 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.650399    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d878d705-9b00-40f5-85f3-16a887c4192d","Type":"ContainerStarted","Data":"a508eeaa97f778c456b4e0b9209323d202d9e87a1ab789dd556adfa19f7ce590"}
Sep 11 17:57:07.680928 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.680857    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.665126156 podStartE2EDuration="2.680835731s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.709575ms" isStatefulPod=false podCreationTimestamp="2026-09-11 17:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:57:07.679420694 +0000 UTC m=+185.522656159" watchObservedRunningTime="2026-09-11 17:57:07.680835731 +0000 UTC m=+185.524071169"
Sep 11 17:57:07.765446 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.765407    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"]
Sep 11 17:57:07.768065 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.768036    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.770971 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.770946    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 11 17:57:07.771117 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.771058    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-b6h8g\""
Sep 11 17:57:07.771117 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.771120    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 11 17:57:07.771264 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.771230    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 11 17:57:07.771432 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.771412    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 11 17:57:07.771560 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.771446    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 11 17:57:07.778781 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.777962    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 11 17:57:07.783192 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.783165    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"]
Sep 11 17:57:07.895334 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.895278    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/262186c8-449d-4f55-8194-fe7b2e619d63-metrics-client-ca\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.895334 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.895338    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/262186c8-449d-4f55-8194-fe7b2e619d63-telemeter-client-tls\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.895597 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.895367    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slrgx\" (UniqueName: \"kubernetes.io/projected/262186c8-449d-4f55-8194-fe7b2e619d63-kube-api-access-slrgx\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.895597 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.895434    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/262186c8-449d-4f55-8194-fe7b2e619d63-serving-certs-ca-bundle\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.895597 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.895463    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/262186c8-449d-4f55-8194-fe7b2e619d63-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.895597 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.895540    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/262186c8-449d-4f55-8194-fe7b2e619d63-federate-client-tls\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.895597 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.895568    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/262186c8-449d-4f55-8194-fe7b2e619d63-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.895804 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.895601    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/262186c8-449d-4f55-8194-fe7b2e619d63-secret-telemeter-client\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.996819 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.996776    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/262186c8-449d-4f55-8194-fe7b2e619d63-federate-client-tls\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.996819 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.996821    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/262186c8-449d-4f55-8194-fe7b2e619d63-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.997081 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.996848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/262186c8-449d-4f55-8194-fe7b2e619d63-secret-telemeter-client\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.997081 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.996907    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/262186c8-449d-4f55-8194-fe7b2e619d63-metrics-client-ca\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.997081 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.996969    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/262186c8-449d-4f55-8194-fe7b2e619d63-telemeter-client-tls\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.997081 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.997026    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slrgx\" (UniqueName: \"kubernetes.io/projected/262186c8-449d-4f55-8194-fe7b2e619d63-kube-api-access-slrgx\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.997271 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.997092    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/262186c8-449d-4f55-8194-fe7b2e619d63-serving-certs-ca-bundle\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.997271 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.997124    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/262186c8-449d-4f55-8194-fe7b2e619d63-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.997940 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.997909    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/262186c8-449d-4f55-8194-fe7b2e619d63-serving-certs-ca-bundle\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.998084 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.997878    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/262186c8-449d-4f55-8194-fe7b2e619d63-metrics-client-ca\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.998084 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.998023    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/262186c8-449d-4f55-8194-fe7b2e619d63-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.999656 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.999625    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/262186c8-449d-4f55-8194-fe7b2e619d63-federate-client-tls\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:07.999798 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:07.999778    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/262186c8-449d-4f55-8194-fe7b2e619d63-secret-telemeter-client\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:08.000125 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:08.000109    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/262186c8-449d-4f55-8194-fe7b2e619d63-telemeter-client-tls\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:08.000125 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:08.000114    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/262186c8-449d-4f55-8194-fe7b2e619d63-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:08.005592 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:08.005566    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slrgx\" (UniqueName: \"kubernetes.io/projected/262186c8-449d-4f55-8194-fe7b2e619d63-kube-api-access-slrgx\") pod \"telemeter-client-7dbf9f45f6-g9prl\" (UID: \"262186c8-449d-4f55-8194-fe7b2e619d63\") " pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:08.079893 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:08.079846    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"
Sep 11 17:57:08.372879 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:08.372788    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl"]
Sep 11 17:57:08.376424 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:57:08.376390    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262186c8_449d_4f55_8194_fe7b2e619d63.slice/crio-d258682cdd044c2811c549b6b0d5fd144e34b457f9048a40244ba9cadd5d54e0 WatchSource:0}: Error finding container d258682cdd044c2811c549b6b0d5fd144e34b457f9048a40244ba9cadd5d54e0: Status 404 returned error can't find the container with id d258682cdd044c2811c549b6b0d5fd144e34b457f9048a40244ba9cadd5d54e0
Sep 11 17:57:08.656091 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:08.655997    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl" event={"ID":"262186c8-449d-4f55-8194-fe7b2e619d63","Type":"ContainerStarted","Data":"d258682cdd044c2811c549b6b0d5fd144e34b457f9048a40244ba9cadd5d54e0"}
Sep 11 17:57:10.668205 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:10.668121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl" event={"ID":"262186c8-449d-4f55-8194-fe7b2e619d63","Type":"ContainerStarted","Data":"77428c17001bc18569f106ea3bfe121fe5be923dfea6f1881cd0672e0b0a4c22"}
Sep 11 17:57:10.668592 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:10.668207    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl" event={"ID":"262186c8-449d-4f55-8194-fe7b2e619d63","Type":"ContainerStarted","Data":"9f39f90fc2a3738c0cfe688eced62a8ee1d193f5001dbd117b1f9b8c6faf943c"}
Sep 11 17:57:10.668592 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:10.668228    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl" event={"ID":"262186c8-449d-4f55-8194-fe7b2e619d63","Type":"ContainerStarted","Data":"420007fa35c57420a5a9d07312ba774145eec10b5482d0202d21477374266ea9"}
Sep 11 17:57:10.695713 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:10.695657    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7dbf9f45f6-g9prl" podStartSLOduration=1.720915354 podStartE2EDuration="3.695641538s" totalImagesPullingTime="1.974726184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:57:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:57:10.694553555 +0000 UTC m=+188.537788996" watchObservedRunningTime="2026-09-11 17:57:10.695641538 +0000 UTC m=+188.538876978"
Sep 11 17:57:10.787162 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:10.787131    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b4fc54849-tpstd"]
Sep 11 17:57:10.789568 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:10.789538    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:10.803367 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:10.803334    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b4fc54849-tpstd"]
Sep 11 17:57:10.924092 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:10.923991    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-trusted-ca-bundle\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:10.924092 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:10.924058    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5badad16-76ee-454d-90f2-9aca295c867c-console-serving-cert\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:10.924092 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:10.924082    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-console-config\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:10.924361 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:10.924104    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-oauth-serving-cert\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:10.924361 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:10.924128    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5badad16-76ee-454d-90f2-9aca295c867c-console-oauth-config\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:10.924361 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:10.924172    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7hwl\" (UniqueName: \"kubernetes.io/projected/5badad16-76ee-454d-90f2-9aca295c867c-kube-api-access-v7hwl\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:10.924361 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:10.924225    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-service-ca\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:11.024769 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:11.024696    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-service-ca\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:11.024964 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:11.024796    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-trusted-ca-bundle\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:11.024964 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:11.024842    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5badad16-76ee-454d-90f2-9aca295c867c-console-serving-cert\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:11.024964 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:11.024870    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-console-config\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:11.024964 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:11.024894    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-oauth-serving-cert\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:11.024964 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:11.024921    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5badad16-76ee-454d-90f2-9aca295c867c-console-oauth-config\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:11.025606 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:11.025571    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-service-ca\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:11.025713 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:11.025581    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-console-config\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:11.025814 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:11.025711    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7hwl\" (UniqueName: \"kubernetes.io/projected/5badad16-76ee-454d-90f2-9aca295c867c-kube-api-access-v7hwl\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:11.025927 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:11.025869    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-oauth-serving-cert\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:11.025927 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:11.025918    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-trusted-ca-bundle\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:11.027919 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:11.027827    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5badad16-76ee-454d-90f2-9aca295c867c-console-serving-cert\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:11.028348 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:11.028314    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5badad16-76ee-454d-90f2-9aca295c867c-console-oauth-config\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:11.034957 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:11.034932    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7hwl\" (UniqueName: \"kubernetes.io/projected/5badad16-76ee-454d-90f2-9aca295c867c-kube-api-access-v7hwl\") pod \"console-5b4fc54849-tpstd\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") " pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:11.101862 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:11.101825    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:11.384824 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:11.384487    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b4fc54849-tpstd"]
Sep 11 17:57:11.387239 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:57:11.387214    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5badad16_76ee_454d_90f2_9aca295c867c.slice/crio-602cff60d79b4b5447494e43b055939854b366806ebbad2b3d8b83e401cb565d WatchSource:0}: Error finding container 602cff60d79b4b5447494e43b055939854b366806ebbad2b3d8b83e401cb565d: Status 404 returned error can't find the container with id 602cff60d79b4b5447494e43b055939854b366806ebbad2b3d8b83e401cb565d
Sep 11 17:57:11.674904 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:11.674813    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b4fc54849-tpstd" event={"ID":"5badad16-76ee-454d-90f2-9aca295c867c","Type":"ContainerStarted","Data":"5cd3f4f4f0064b594f4b09ecaf754ab0e25d32a1d04d4622d852691d71d4cbc5"}
Sep 11 17:57:11.674904 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:11.674870    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b4fc54849-tpstd" event={"ID":"5badad16-76ee-454d-90f2-9aca295c867c","Type":"ContainerStarted","Data":"602cff60d79b4b5447494e43b055939854b366806ebbad2b3d8b83e401cb565d"}
Sep 11 17:57:11.695635 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:11.695585    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b4fc54849-tpstd" podStartSLOduration=1.6955667349999999 podStartE2EDuration="1.695566735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:57:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:57:11.69419508 +0000 UTC m=+189.537430557" watchObservedRunningTime="2026-09-11 17:57:11.695566735 +0000 UTC m=+189.538802175"
Sep 11 17:57:21.102904 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:21.102849    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:21.103360 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:21.102926    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:21.108085 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:21.108051    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:21.720181 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:21.720142    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:57:21.778884 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:21.778852    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c5c7bc68d-gvr8q"]
Sep 11 17:57:46.799831 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:46.799779    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c5c7bc68d-gvr8q" podUID="130c74da-fa1e-4951-bfee-4b14dcc15939" containerName="console" containerID="cri-o://2ef90e1a97c1ba414bd721b35209809d72ca2515bc4ddfad787ef620577e6676" gracePeriod=15
Sep 11 17:57:47.041974 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.041942    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:57:47.143584 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.143502    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/130c74da-fa1e-4951-bfee-4b14dcc15939-console-serving-cert\") pod \"130c74da-fa1e-4951-bfee-4b14dcc15939\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") "
Sep 11 17:57:47.143584 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.143567    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjzw6\" (UniqueName: \"kubernetes.io/projected/130c74da-fa1e-4951-bfee-4b14dcc15939-kube-api-access-gjzw6\") pod \"130c74da-fa1e-4951-bfee-4b14dcc15939\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") "
Sep 11 17:57:47.143584 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.143586    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-trusted-ca-bundle\") pod \"130c74da-fa1e-4951-bfee-4b14dcc15939\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") "
Sep 11 17:57:47.143857 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.143699    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-oauth-serving-cert\") pod \"130c74da-fa1e-4951-bfee-4b14dcc15939\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") "
Sep 11 17:57:47.143857 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.143775    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-console-config\") pod \"130c74da-fa1e-4951-bfee-4b14dcc15939\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") "
Sep 11 17:57:47.143857 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.143820    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/130c74da-fa1e-4951-bfee-4b14dcc15939-console-oauth-config\") pod \"130c74da-fa1e-4951-bfee-4b14dcc15939\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") "
Sep 11 17:57:47.143857 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.143843    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-service-ca\") pod \"130c74da-fa1e-4951-bfee-4b14dcc15939\" (UID: \"130c74da-fa1e-4951-bfee-4b14dcc15939\") "
Sep 11 17:57:47.144063 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.143980    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "130c74da-fa1e-4951-bfee-4b14dcc15939" (UID: "130c74da-fa1e-4951-bfee-4b14dcc15939"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:57:47.144223 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.144191    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "130c74da-fa1e-4951-bfee-4b14dcc15939" (UID: "130c74da-fa1e-4951-bfee-4b14dcc15939"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:57:47.144223 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.144202    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-console-config" (OuterVolumeSpecName: "console-config") pod "130c74da-fa1e-4951-bfee-4b14dcc15939" (UID: "130c74da-fa1e-4951-bfee-4b14dcc15939"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:57:47.144394 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.144244    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-trusted-ca-bundle\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:57:47.144495 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.144462    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-service-ca" (OuterVolumeSpecName: "service-ca") pod "130c74da-fa1e-4951-bfee-4b14dcc15939" (UID: "130c74da-fa1e-4951-bfee-4b14dcc15939"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:57:47.145937 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.145909    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130c74da-fa1e-4951-bfee-4b14dcc15939-kube-api-access-gjzw6" (OuterVolumeSpecName: "kube-api-access-gjzw6") pod "130c74da-fa1e-4951-bfee-4b14dcc15939" (UID: "130c74da-fa1e-4951-bfee-4b14dcc15939"). InnerVolumeSpecName "kube-api-access-gjzw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:57:47.145937 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.145925    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/130c74da-fa1e-4951-bfee-4b14dcc15939-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "130c74da-fa1e-4951-bfee-4b14dcc15939" (UID: "130c74da-fa1e-4951-bfee-4b14dcc15939"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:57:47.146061 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.145939    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/130c74da-fa1e-4951-bfee-4b14dcc15939-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "130c74da-fa1e-4951-bfee-4b14dcc15939" (UID: "130c74da-fa1e-4951-bfee-4b14dcc15939"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:57:47.245184 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.245152    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-console-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:57:47.245184 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.245174    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/130c74da-fa1e-4951-bfee-4b14dcc15939-console-oauth-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:57:47.245184 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.245184    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:57:47.245384 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.245192    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/130c74da-fa1e-4951-bfee-4b14dcc15939-console-serving-cert\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:57:47.245384 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.245202    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjzw6\" (UniqueName: \"kubernetes.io/projected/130c74da-fa1e-4951-bfee-4b14dcc15939-kube-api-access-gjzw6\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:57:47.245384 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.245211    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/130c74da-fa1e-4951-bfee-4b14dcc15939-oauth-serving-cert\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:57:47.829998 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.829960    2560 generic.go:356] "Generic (PLEG): container finished" podID="130c74da-fa1e-4951-bfee-4b14dcc15939" containerID="2ef90e1a97c1ba414bd721b35209809d72ca2515bc4ddfad787ef620577e6676" exitCode=0
Sep 11 17:57:47.830430 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.830005    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5c7bc68d-gvr8q"
Sep 11 17:57:47.830430 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.830052    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5c7bc68d-gvr8q" event={"ID":"130c74da-fa1e-4951-bfee-4b14dcc15939","Type":"ContainerDied","Data":"2ef90e1a97c1ba414bd721b35209809d72ca2515bc4ddfad787ef620577e6676"}
Sep 11 17:57:47.830430 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.830100    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5c7bc68d-gvr8q" event={"ID":"130c74da-fa1e-4951-bfee-4b14dcc15939","Type":"ContainerDied","Data":"dc6b9179c2fb9ffdaf10cd0ac7aca44f1e20cb325ca9a4331360322710a54008"}
Sep 11 17:57:47.830430 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.830121    2560 scope.go:117] "RemoveContainer" containerID="2ef90e1a97c1ba414bd721b35209809d72ca2515bc4ddfad787ef620577e6676"
Sep 11 17:57:47.839306 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.839289    2560 scope.go:117] "RemoveContainer" containerID="2ef90e1a97c1ba414bd721b35209809d72ca2515bc4ddfad787ef620577e6676"
Sep 11 17:57:47.839584 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:57:47.839565    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef90e1a97c1ba414bd721b35209809d72ca2515bc4ddfad787ef620577e6676\": container with ID starting with 2ef90e1a97c1ba414bd721b35209809d72ca2515bc4ddfad787ef620577e6676 not found: ID does not exist" containerID="2ef90e1a97c1ba414bd721b35209809d72ca2515bc4ddfad787ef620577e6676"
Sep 11 17:57:47.839655 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.839591    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef90e1a97c1ba414bd721b35209809d72ca2515bc4ddfad787ef620577e6676"} err="failed to get container status \"2ef90e1a97c1ba414bd721b35209809d72ca2515bc4ddfad787ef620577e6676\": rpc error: code = NotFound desc = could not find container \"2ef90e1a97c1ba414bd721b35209809d72ca2515bc4ddfad787ef620577e6676\": container with ID starting with 2ef90e1a97c1ba414bd721b35209809d72ca2515bc4ddfad787ef620577e6676 not found: ID does not exist"
Sep 11 17:57:47.869158 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.869085    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c5c7bc68d-gvr8q"]
Sep 11 17:57:47.871230 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:47.871192    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c5c7bc68d-gvr8q"]
Sep 11 17:57:48.688647 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:57:48.688617    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="130c74da-fa1e-4951-bfee-4b14dcc15939" path="/var/lib/kubelet/pods/130c74da-fa1e-4951-bfee-4b14dcc15939/volumes"
Sep 11 17:58:46.938548 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:46.938515    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bf7df85d8-jwv2l"]
Sep 11 17:58:46.938972 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:46.938857    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="130c74da-fa1e-4951-bfee-4b14dcc15939" containerName="console"
Sep 11 17:58:46.938972 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:46.938869    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="130c74da-fa1e-4951-bfee-4b14dcc15939" containerName="console"
Sep 11 17:58:46.938972 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:46.938928    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="130c74da-fa1e-4951-bfee-4b14dcc15939" containerName="console"
Sep 11 17:58:46.941727 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:46.941710    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:46.954539 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:46.954514    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bf7df85d8-jwv2l"]
Sep 11 17:58:47.028605 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.028570    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e846f82b-2faf-460b-9160-360df9136313-console-serving-cert\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.028605 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.028604    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e846f82b-2faf-460b-9160-360df9136313-console-oauth-config\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.028892 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.028624    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-service-ca\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.028892 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.028673    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-trusted-ca-bundle\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.028892 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.028720    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-oauth-serving-cert\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.028892 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.028738    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpm4c\" (UniqueName: \"kubernetes.io/projected/e846f82b-2faf-460b-9160-360df9136313-kube-api-access-kpm4c\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.028892 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.028776    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-console-config\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.129536 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.129498    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e846f82b-2faf-460b-9160-360df9136313-console-serving-cert\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.129536 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.129534    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e846f82b-2faf-460b-9160-360df9136313-console-oauth-config\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.129836 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.129552    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-service-ca\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.129836 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.129573    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-trusted-ca-bundle\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.129836 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.129590    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-oauth-serving-cert\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.129836 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.129605    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpm4c\" (UniqueName: \"kubernetes.io/projected/e846f82b-2faf-460b-9160-360df9136313-kube-api-access-kpm4c\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.129836 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.129630    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-console-config\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.130381 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.130342    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-service-ca\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.130499 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.130440    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-console-config\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.130499 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.130441    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-oauth-serving-cert\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.130660 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.130640    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-trusted-ca-bundle\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.132726 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.132701    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e846f82b-2faf-460b-9160-360df9136313-console-oauth-config\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.132904 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.132883    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e846f82b-2faf-460b-9160-360df9136313-console-serving-cert\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.138482 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.138458    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpm4c\" (UniqueName: \"kubernetes.io/projected/e846f82b-2faf-460b-9160-360df9136313-kube-api-access-kpm4c\") pod \"console-7bf7df85d8-jwv2l\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") " pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.251934 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.251885    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:47.522180 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:47.522154    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bf7df85d8-jwv2l"]
Sep 11 17:58:47.524795 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:58:47.524765    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode846f82b_2faf_460b_9160_360df9136313.slice/crio-ba9f4919371de8de1c5f91e79baeeaa6a8388ee50ad32692b5113da3f1562669 WatchSource:0}: Error finding container ba9f4919371de8de1c5f91e79baeeaa6a8388ee50ad32692b5113da3f1562669: Status 404 returned error can't find the container with id ba9f4919371de8de1c5f91e79baeeaa6a8388ee50ad32692b5113da3f1562669
Sep 11 17:58:48.080257 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:48.080224    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf7df85d8-jwv2l" event={"ID":"e846f82b-2faf-460b-9160-360df9136313","Type":"ContainerStarted","Data":"e92e82c7b995b1fdac4f641b9ce59fbfab26dbdfb9a54cc1ac0d45199966c1ef"}
Sep 11 17:58:48.080634 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:48.080272    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf7df85d8-jwv2l" event={"ID":"e846f82b-2faf-460b-9160-360df9136313","Type":"ContainerStarted","Data":"ba9f4919371de8de1c5f91e79baeeaa6a8388ee50ad32692b5113da3f1562669"}
Sep 11 17:58:48.099806 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:48.099717    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bf7df85d8-jwv2l" podStartSLOduration=2.099700769 podStartE2EDuration="2.099700769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:58:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:58:48.09805153 +0000 UTC m=+285.941286971" watchObservedRunningTime="2026-09-11 17:58:48.099700769 +0000 UTC m=+285.942936209"
Sep 11 17:58:57.252635 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:57.252593    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:57.253096 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:57.252656    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:57.257626 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:57.257603    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:58.126856 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:58.126825    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 17:58:58.178155 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:58:58.178124    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b4fc54849-tpstd"]
Sep 11 17:59:02.568184 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:02.568156    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/2.log"
Sep 11 17:59:02.586113 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:02.586086    2560 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 17:59:02.593426 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:02.593403    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/2.log"
Sep 11 17:59:23.204011 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.203962    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b4fc54849-tpstd" podUID="5badad16-76ee-454d-90f2-9aca295c867c" containerName="console" containerID="cri-o://5cd3f4f4f0064b594f4b09ecaf754ab0e25d32a1d04d4622d852691d71d4cbc5" gracePeriod=15
Sep 11 17:59:23.443819 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.443794    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:59:23.535055 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.535022    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-trusted-ca-bundle\") pod \"5badad16-76ee-454d-90f2-9aca295c867c\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") "
Sep 11 17:59:23.535055 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.535061    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5badad16-76ee-454d-90f2-9aca295c867c-console-serving-cert\") pod \"5badad16-76ee-454d-90f2-9aca295c867c\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") "
Sep 11 17:59:23.535296 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.535101    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5badad16-76ee-454d-90f2-9aca295c867c-console-oauth-config\") pod \"5badad16-76ee-454d-90f2-9aca295c867c\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") "
Sep 11 17:59:23.535296 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.535121    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7hwl\" (UniqueName: \"kubernetes.io/projected/5badad16-76ee-454d-90f2-9aca295c867c-kube-api-access-v7hwl\") pod \"5badad16-76ee-454d-90f2-9aca295c867c\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") "
Sep 11 17:59:23.535296 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.535144    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-console-config\") pod \"5badad16-76ee-454d-90f2-9aca295c867c\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") "
Sep 11 17:59:23.535296 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.535173    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-oauth-serving-cert\") pod \"5badad16-76ee-454d-90f2-9aca295c867c\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") "
Sep 11 17:59:23.535296 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.535188    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-service-ca\") pod \"5badad16-76ee-454d-90f2-9aca295c867c\" (UID: \"5badad16-76ee-454d-90f2-9aca295c867c\") "
Sep 11 17:59:23.535536 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.535441    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5badad16-76ee-454d-90f2-9aca295c867c" (UID: "5badad16-76ee-454d-90f2-9aca295c867c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:59:23.535719 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.535668    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5badad16-76ee-454d-90f2-9aca295c867c" (UID: "5badad16-76ee-454d-90f2-9aca295c867c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:59:23.535719 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.535676    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-console-config" (OuterVolumeSpecName: "console-config") pod "5badad16-76ee-454d-90f2-9aca295c867c" (UID: "5badad16-76ee-454d-90f2-9aca295c867c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:59:23.535719 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.535693    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-service-ca" (OuterVolumeSpecName: "service-ca") pod "5badad16-76ee-454d-90f2-9aca295c867c" (UID: "5badad16-76ee-454d-90f2-9aca295c867c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 17:59:23.537476 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.537448    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5badad16-76ee-454d-90f2-9aca295c867c-kube-api-access-v7hwl" (OuterVolumeSpecName: "kube-api-access-v7hwl") pod "5badad16-76ee-454d-90f2-9aca295c867c" (UID: "5badad16-76ee-454d-90f2-9aca295c867c"). InnerVolumeSpecName "kube-api-access-v7hwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 17:59:23.537476 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.537466    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5badad16-76ee-454d-90f2-9aca295c867c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5badad16-76ee-454d-90f2-9aca295c867c" (UID: "5badad16-76ee-454d-90f2-9aca295c867c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:59:23.537640 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.537488    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5badad16-76ee-454d-90f2-9aca295c867c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5badad16-76ee-454d-90f2-9aca295c867c" (UID: "5badad16-76ee-454d-90f2-9aca295c867c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 17:59:23.636118 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.636081    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5badad16-76ee-454d-90f2-9aca295c867c-console-oauth-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:59:23.636118 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.636112    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7hwl\" (UniqueName: \"kubernetes.io/projected/5badad16-76ee-454d-90f2-9aca295c867c-kube-api-access-v7hwl\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:59:23.636118 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.636123    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-console-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:59:23.636347 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.636132    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-oauth-serving-cert\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:59:23.636347 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.636141    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:59:23.636347 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.636149    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5badad16-76ee-454d-90f2-9aca295c867c-trusted-ca-bundle\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:59:23.636347 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:23.636157    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5badad16-76ee-454d-90f2-9aca295c867c-console-serving-cert\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 17:59:24.233596 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:24.233567    2560 generic.go:356] "Generic (PLEG): container finished" podID="5badad16-76ee-454d-90f2-9aca295c867c" containerID="5cd3f4f4f0064b594f4b09ecaf754ab0e25d32a1d04d4622d852691d71d4cbc5" exitCode=0
Sep 11 17:59:24.234019 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:24.233607    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b4fc54849-tpstd"
Sep 11 17:59:24.234019 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:24.233652    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b4fc54849-tpstd" event={"ID":"5badad16-76ee-454d-90f2-9aca295c867c","Type":"ContainerDied","Data":"5cd3f4f4f0064b594f4b09ecaf754ab0e25d32a1d04d4622d852691d71d4cbc5"}
Sep 11 17:59:24.234019 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:24.233705    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b4fc54849-tpstd" event={"ID":"5badad16-76ee-454d-90f2-9aca295c867c","Type":"ContainerDied","Data":"602cff60d79b4b5447494e43b055939854b366806ebbad2b3d8b83e401cb565d"}
Sep 11 17:59:24.234019 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:24.233726    2560 scope.go:117] "RemoveContainer" containerID="5cd3f4f4f0064b594f4b09ecaf754ab0e25d32a1d04d4622d852691d71d4cbc5"
Sep 11 17:59:24.242560 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:24.242545    2560 scope.go:117] "RemoveContainer" containerID="5cd3f4f4f0064b594f4b09ecaf754ab0e25d32a1d04d4622d852691d71d4cbc5"
Sep 11 17:59:24.242847 ip-10-0-128-29 kubenswrapper[2560]: E0911 17:59:24.242827    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd3f4f4f0064b594f4b09ecaf754ab0e25d32a1d04d4622d852691d71d4cbc5\": container with ID starting with 5cd3f4f4f0064b594f4b09ecaf754ab0e25d32a1d04d4622d852691d71d4cbc5 not found: ID does not exist" containerID="5cd3f4f4f0064b594f4b09ecaf754ab0e25d32a1d04d4622d852691d71d4cbc5"
Sep 11 17:59:24.242924 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:24.242854    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd3f4f4f0064b594f4b09ecaf754ab0e25d32a1d04d4622d852691d71d4cbc5"} err="failed to get container status \"5cd3f4f4f0064b594f4b09ecaf754ab0e25d32a1d04d4622d852691d71d4cbc5\": rpc error: code = NotFound desc = could not find container \"5cd3f4f4f0064b594f4b09ecaf754ab0e25d32a1d04d4622d852691d71d4cbc5\": container with ID starting with 5cd3f4f4f0064b594f4b09ecaf754ab0e25d32a1d04d4622d852691d71d4cbc5 not found: ID does not exist"
Sep 11 17:59:24.265815 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:24.265787    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b4fc54849-tpstd"]
Sep 11 17:59:24.273244 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:24.273224    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b4fc54849-tpstd"]
Sep 11 17:59:24.687019 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:24.686945    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5badad16-76ee-454d-90f2-9aca295c867c" path="/var/lib/kubelet/pods/5badad16-76ee-454d-90f2-9aca295c867c/volumes"
Sep 11 17:59:29.743668 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:29.743632    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-59m6d"]
Sep 11 17:59:29.744080 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:29.743969    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5badad16-76ee-454d-90f2-9aca295c867c" containerName="console"
Sep 11 17:59:29.744080 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:29.743982    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5badad16-76ee-454d-90f2-9aca295c867c" containerName="console"
Sep 11 17:59:29.744080 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:29.744053    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5badad16-76ee-454d-90f2-9aca295c867c" containerName="console"
Sep 11 17:59:29.748296 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:29.748278    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-59m6d"
Sep 11 17:59:29.751172 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:29.751153    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-s8z7n\""
Sep 11 17:59:29.751282 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:29.751157    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 17:59:29.751282 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:29.751189    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 17:59:29.757614 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:29.757589    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-59m6d"]
Sep 11 17:59:29.788140 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:29.788108    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3febdbe-c41e-41bf-ad54-d1e52f9c804d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-59m6d\" (UID: \"e3febdbe-c41e-41bf-ad54-d1e52f9c804d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-59m6d"
Sep 11 17:59:29.788325 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:29.788160    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp7qs\" (UniqueName: \"kubernetes.io/projected/e3febdbe-c41e-41bf-ad54-d1e52f9c804d-kube-api-access-rp7qs\") pod \"cert-manager-webhook-6486ffd997-59m6d\" (UID: \"e3febdbe-c41e-41bf-ad54-d1e52f9c804d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-59m6d"
Sep 11 17:59:29.889276 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:29.889240    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3febdbe-c41e-41bf-ad54-d1e52f9c804d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-59m6d\" (UID: \"e3febdbe-c41e-41bf-ad54-d1e52f9c804d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-59m6d"
Sep 11 17:59:29.889454 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:29.889292    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp7qs\" (UniqueName: \"kubernetes.io/projected/e3febdbe-c41e-41bf-ad54-d1e52f9c804d-kube-api-access-rp7qs\") pod \"cert-manager-webhook-6486ffd997-59m6d\" (UID: \"e3febdbe-c41e-41bf-ad54-d1e52f9c804d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-59m6d"
Sep 11 17:59:29.901806 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:29.901773    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e3febdbe-c41e-41bf-ad54-d1e52f9c804d-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-59m6d\" (UID: \"e3febdbe-c41e-41bf-ad54-d1e52f9c804d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-59m6d"
Sep 11 17:59:29.903191 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:29.903166    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp7qs\" (UniqueName: \"kubernetes.io/projected/e3febdbe-c41e-41bf-ad54-d1e52f9c804d-kube-api-access-rp7qs\") pod \"cert-manager-webhook-6486ffd997-59m6d\" (UID: \"e3febdbe-c41e-41bf-ad54-d1e52f9c804d\") " pod="cert-manager/cert-manager-webhook-6486ffd997-59m6d"
Sep 11 17:59:30.073618 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:30.073522    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-59m6d"
Sep 11 17:59:30.346236 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:30.346213    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-59m6d"]
Sep 11 17:59:30.348695 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:59:30.348663    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3febdbe_c41e_41bf_ad54_d1e52f9c804d.slice/crio-9a0a845071ea22a1e43d66cb2296cce8ede82347180e80919179f880898dd800 WatchSource:0}: Error finding container 9a0a845071ea22a1e43d66cb2296cce8ede82347180e80919179f880898dd800: Status 404 returned error can't find the container with id 9a0a845071ea22a1e43d66cb2296cce8ede82347180e80919179f880898dd800
Sep 11 17:59:30.350702 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:30.350688    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 17:59:31.269493 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:31.269454    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-59m6d" event={"ID":"e3febdbe-c41e-41bf-ad54-d1e52f9c804d","Type":"ContainerStarted","Data":"9a0a845071ea22a1e43d66cb2296cce8ede82347180e80919179f880898dd800"}
Sep 11 17:59:34.286858 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:34.286826    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-59m6d" event={"ID":"e3febdbe-c41e-41bf-ad54-d1e52f9c804d","Type":"ContainerStarted","Data":"3a1323423f5439e3103f0e02780dbc07f69cd43f9c116ae373a74a7e0065134c"}
Sep 11 17:59:34.287304 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:34.286889    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-59m6d"
Sep 11 17:59:34.306725 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:34.306668    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-59m6d" podStartSLOduration=2.166630977 podStartE2EDuration="5.306646007s" totalImagesPullingTime="3.14001503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:59:34.303700494 +0000 UTC m=+332.146935935" watchObservedRunningTime="2026-09-11 17:59:34.306646007 +0000 UTC m=+332.149881520"
Sep 11 17:59:40.293401 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:40.293371    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-59m6d"
Sep 11 17:59:41.304276 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:41.304197    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-s8rzz"]
Sep 11 17:59:41.307405 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:41.307388    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-s8rzz"
Sep 11 17:59:41.310218 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:41.310200    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-xtdgk\""
Sep 11 17:59:41.316653 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:41.316632    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-s8rzz"]
Sep 11 17:59:41.381874 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:41.381846    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c1afafb-5dda-468e-92d4-0ef0fd8fac87-bound-sa-token\") pod \"cert-manager-55b4d954d5-s8rzz\" (UID: \"8c1afafb-5dda-468e-92d4-0ef0fd8fac87\") " pod="cert-manager/cert-manager-55b4d954d5-s8rzz"
Sep 11 17:59:41.381874 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:41.381875    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmzw4\" (UniqueName: \"kubernetes.io/projected/8c1afafb-5dda-468e-92d4-0ef0fd8fac87-kube-api-access-gmzw4\") pod \"cert-manager-55b4d954d5-s8rzz\" (UID: \"8c1afafb-5dda-468e-92d4-0ef0fd8fac87\") " pod="cert-manager/cert-manager-55b4d954d5-s8rzz"
Sep 11 17:59:41.483135 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:41.483103    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c1afafb-5dda-468e-92d4-0ef0fd8fac87-bound-sa-token\") pod \"cert-manager-55b4d954d5-s8rzz\" (UID: \"8c1afafb-5dda-468e-92d4-0ef0fd8fac87\") " pod="cert-manager/cert-manager-55b4d954d5-s8rzz"
Sep 11 17:59:41.483135 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:41.483136    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmzw4\" (UniqueName: \"kubernetes.io/projected/8c1afafb-5dda-468e-92d4-0ef0fd8fac87-kube-api-access-gmzw4\") pod \"cert-manager-55b4d954d5-s8rzz\" (UID: \"8c1afafb-5dda-468e-92d4-0ef0fd8fac87\") " pod="cert-manager/cert-manager-55b4d954d5-s8rzz"
Sep 11 17:59:41.491894 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:41.491865    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c1afafb-5dda-468e-92d4-0ef0fd8fac87-bound-sa-token\") pod \"cert-manager-55b4d954d5-s8rzz\" (UID: \"8c1afafb-5dda-468e-92d4-0ef0fd8fac87\") " pod="cert-manager/cert-manager-55b4d954d5-s8rzz"
Sep 11 17:59:41.492044 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:41.492028    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmzw4\" (UniqueName: \"kubernetes.io/projected/8c1afafb-5dda-468e-92d4-0ef0fd8fac87-kube-api-access-gmzw4\") pod \"cert-manager-55b4d954d5-s8rzz\" (UID: \"8c1afafb-5dda-468e-92d4-0ef0fd8fac87\") " pod="cert-manager/cert-manager-55b4d954d5-s8rzz"
Sep 11 17:59:41.616639 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:41.616555    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-s8rzz"
Sep 11 17:59:41.884276 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:41.884254    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-s8rzz"]
Sep 11 17:59:41.886452 ip-10-0-128-29 kubenswrapper[2560]: W0911 17:59:41.886427    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c1afafb_5dda_468e_92d4_0ef0fd8fac87.slice/crio-87d0b2e45c39a80d1d468af2e9257cc520ef75cf4d44f1923e0a1f4f448745d0 WatchSource:0}: Error finding container 87d0b2e45c39a80d1d468af2e9257cc520ef75cf4d44f1923e0a1f4f448745d0: Status 404 returned error can't find the container with id 87d0b2e45c39a80d1d468af2e9257cc520ef75cf4d44f1923e0a1f4f448745d0
Sep 11 17:59:42.325684 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:42.325640    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-s8rzz" event={"ID":"8c1afafb-5dda-468e-92d4-0ef0fd8fac87","Type":"ContainerStarted","Data":"b9c57935ce8f701e5bb0c0a6b461c3d545f75d0227f257b507683ea4da953081"}
Sep 11 17:59:42.325684 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:42.325687    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-s8rzz" event={"ID":"8c1afafb-5dda-468e-92d4-0ef0fd8fac87","Type":"ContainerStarted","Data":"87d0b2e45c39a80d1d468af2e9257cc520ef75cf4d44f1923e0a1f4f448745d0"}
Sep 11 17:59:42.343392 ip-10-0-128-29 kubenswrapper[2560]: I0911 17:59:42.343347    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-s8rzz" podStartSLOduration=1.343330624 podStartE2EDuration="1.343330624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 17:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 17:59:42.34152195 +0000 UTC m=+340.184757401" watchObservedRunningTime="2026-09-11 17:59:42.343330624 +0000 UTC m=+340.186566066"
Sep 11 18:00:10.388770 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.388710    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln"]
Sep 11 18:00:10.393090 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.393073    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln"
Sep 11 18:00:10.396420 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.396398    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 18:00:10.398227 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.398208    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wjldh\""
Sep 11 18:00:10.398353 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.398263    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 18:00:10.398353 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.398273    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 18:00:10.398353 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.398273    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:00:10.398353 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.398327    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 18:00:10.403600 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.403578    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln"]
Sep 11 18:00:10.525519 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.525484    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6af40138-8dae-46c2-a76c-5163b937a4a8-metrics-cert\") pod \"lws-controller-manager-ffbd766d4-9mjln\" (UID: \"6af40138-8dae-46c2-a76c-5163b937a4a8\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln"
Sep 11 18:00:10.525677 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.525551    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6af40138-8dae-46c2-a76c-5163b937a4a8-manager-config\") pod \"lws-controller-manager-ffbd766d4-9mjln\" (UID: \"6af40138-8dae-46c2-a76c-5163b937a4a8\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln"
Sep 11 18:00:10.525677 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.525573    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kljb\" (UniqueName: \"kubernetes.io/projected/6af40138-8dae-46c2-a76c-5163b937a4a8-kube-api-access-9kljb\") pod \"lws-controller-manager-ffbd766d4-9mjln\" (UID: \"6af40138-8dae-46c2-a76c-5163b937a4a8\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln"
Sep 11 18:00:10.525677 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.525647    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6af40138-8dae-46c2-a76c-5163b937a4a8-cert\") pod \"lws-controller-manager-ffbd766d4-9mjln\" (UID: \"6af40138-8dae-46c2-a76c-5163b937a4a8\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln"
Sep 11 18:00:10.627074 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.627039    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6af40138-8dae-46c2-a76c-5163b937a4a8-cert\") pod \"lws-controller-manager-ffbd766d4-9mjln\" (UID: \"6af40138-8dae-46c2-a76c-5163b937a4a8\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln"
Sep 11 18:00:10.627231 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.627092    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6af40138-8dae-46c2-a76c-5163b937a4a8-metrics-cert\") pod \"lws-controller-manager-ffbd766d4-9mjln\" (UID: \"6af40138-8dae-46c2-a76c-5163b937a4a8\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln"
Sep 11 18:00:10.627231 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.627119    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6af40138-8dae-46c2-a76c-5163b937a4a8-manager-config\") pod \"lws-controller-manager-ffbd766d4-9mjln\" (UID: \"6af40138-8dae-46c2-a76c-5163b937a4a8\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln"
Sep 11 18:00:10.627231 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.627134    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kljb\" (UniqueName: \"kubernetes.io/projected/6af40138-8dae-46c2-a76c-5163b937a4a8-kube-api-access-9kljb\") pod \"lws-controller-manager-ffbd766d4-9mjln\" (UID: \"6af40138-8dae-46c2-a76c-5163b937a4a8\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln"
Sep 11 18:00:10.627865 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.627846    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6af40138-8dae-46c2-a76c-5163b937a4a8-manager-config\") pod \"lws-controller-manager-ffbd766d4-9mjln\" (UID: \"6af40138-8dae-46c2-a76c-5163b937a4a8\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln"
Sep 11 18:00:10.629617 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.629582    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6af40138-8dae-46c2-a76c-5163b937a4a8-metrics-cert\") pod \"lws-controller-manager-ffbd766d4-9mjln\" (UID: \"6af40138-8dae-46c2-a76c-5163b937a4a8\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln"
Sep 11 18:00:10.629728 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.629643    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6af40138-8dae-46c2-a76c-5163b937a4a8-cert\") pod \"lws-controller-manager-ffbd766d4-9mjln\" (UID: \"6af40138-8dae-46c2-a76c-5163b937a4a8\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln"
Sep 11 18:00:10.635526 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.635500    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kljb\" (UniqueName: \"kubernetes.io/projected/6af40138-8dae-46c2-a76c-5163b937a4a8-kube-api-access-9kljb\") pod \"lws-controller-manager-ffbd766d4-9mjln\" (UID: \"6af40138-8dae-46c2-a76c-5163b937a4a8\") " pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln"
Sep 11 18:00:10.702767 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:10.702675    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln"
Sep 11 18:00:11.013435 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:11.013402    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln"]
Sep 11 18:00:11.013665 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:00:11.013640    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af40138_8dae_46c2_a76c_5163b937a4a8.slice/crio-90ca92aaffd571ee520af4be72d0d42bd87604cd64d930fe5134607f1a5d4fea WatchSource:0}: Error finding container 90ca92aaffd571ee520af4be72d0d42bd87604cd64d930fe5134607f1a5d4fea: Status 404 returned error can't find the container with id 90ca92aaffd571ee520af4be72d0d42bd87604cd64d930fe5134607f1a5d4fea
Sep 11 18:00:11.448495 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:11.448410    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln" event={"ID":"6af40138-8dae-46c2-a76c-5163b937a4a8","Type":"ContainerStarted","Data":"90ca92aaffd571ee520af4be72d0d42bd87604cd64d930fe5134607f1a5d4fea"}
Sep 11 18:00:12.670275 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.669738    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"]
Sep 11 18:00:12.672550 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.672527    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"
Sep 11 18:00:12.679651 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.679628    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 18:00:12.680217 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.680123    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 18:00:12.681093 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.681056    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 18:00:12.681214 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.681127    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 18:00:12.681214 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.681199    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-srmfv\""
Sep 11 18:00:12.746246 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.746211    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e01fbe9e-fc87-4548-8223-64276fabad17-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7\" (UID: \"e01fbe9e-fc87-4548-8223-64276fabad17\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"
Sep 11 18:00:12.746438 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.746298    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e01fbe9e-fc87-4548-8223-64276fabad17-tmp\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7\" (UID: \"e01fbe9e-fc87-4548-8223-64276fabad17\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"
Sep 11 18:00:12.746438 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.746383    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e01fbe9e-fc87-4548-8223-64276fabad17-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7\" (UID: \"e01fbe9e-fc87-4548-8223-64276fabad17\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"
Sep 11 18:00:12.746438 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.746409    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfjgk\" (UniqueName: \"kubernetes.io/projected/e01fbe9e-fc87-4548-8223-64276fabad17-kube-api-access-jfjgk\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7\" (UID: \"e01fbe9e-fc87-4548-8223-64276fabad17\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"
Sep 11 18:00:12.746628 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.746452    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e01fbe9e-fc87-4548-8223-64276fabad17-manifests\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7\" (UID: \"e01fbe9e-fc87-4548-8223-64276fabad17\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"
Sep 11 18:00:12.791810 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.791772    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"]
Sep 11 18:00:12.847639 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.847606    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e01fbe9e-fc87-4548-8223-64276fabad17-tmp\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7\" (UID: \"e01fbe9e-fc87-4548-8223-64276fabad17\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"
Sep 11 18:00:12.847828 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.847683    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e01fbe9e-fc87-4548-8223-64276fabad17-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7\" (UID: \"e01fbe9e-fc87-4548-8223-64276fabad17\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"
Sep 11 18:00:12.847828 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.847705    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfjgk\" (UniqueName: \"kubernetes.io/projected/e01fbe9e-fc87-4548-8223-64276fabad17-kube-api-access-jfjgk\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7\" (UID: \"e01fbe9e-fc87-4548-8223-64276fabad17\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"
Sep 11 18:00:12.847828 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.847732    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e01fbe9e-fc87-4548-8223-64276fabad17-manifests\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7\" (UID: \"e01fbe9e-fc87-4548-8223-64276fabad17\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"
Sep 11 18:00:12.847828 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.847785    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e01fbe9e-fc87-4548-8223-64276fabad17-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7\" (UID: \"e01fbe9e-fc87-4548-8223-64276fabad17\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"
Sep 11 18:00:12.848237 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.848212    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e01fbe9e-fc87-4548-8223-64276fabad17-manifests\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7\" (UID: \"e01fbe9e-fc87-4548-8223-64276fabad17\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"
Sep 11 18:00:12.850951 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.850921    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e01fbe9e-fc87-4548-8223-64276fabad17-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7\" (UID: \"e01fbe9e-fc87-4548-8223-64276fabad17\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"
Sep 11 18:00:12.851058 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.850921    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e01fbe9e-fc87-4548-8223-64276fabad17-tmp\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7\" (UID: \"e01fbe9e-fc87-4548-8223-64276fabad17\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"
Sep 11 18:00:12.852305 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.852284    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e01fbe9e-fc87-4548-8223-64276fabad17-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7\" (UID: \"e01fbe9e-fc87-4548-8223-64276fabad17\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"
Sep 11 18:00:12.858822 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.858803    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfjgk\" (UniqueName: \"kubernetes.io/projected/e01fbe9e-fc87-4548-8223-64276fabad17-kube-api-access-jfjgk\") pod \"opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7\" (UID: \"e01fbe9e-fc87-4548-8223-64276fabad17\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"
Sep 11 18:00:12.987120 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:12.987084    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"
Sep 11 18:00:14.154743 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:00:14.154713    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01fbe9e_fc87_4548_8223_64276fabad17.slice/crio-2dbb5cdfd8c827dd50d314ecac5b203727cc146039ffd48cb45679134dd1c3c4 WatchSource:0}: Error finding container 2dbb5cdfd8c827dd50d314ecac5b203727cc146039ffd48cb45679134dd1c3c4: Status 404 returned error can't find the container with id 2dbb5cdfd8c827dd50d314ecac5b203727cc146039ffd48cb45679134dd1c3c4
Sep 11 18:00:14.162709 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:14.162688    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"]
Sep 11 18:00:14.465424 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:14.465387    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7" event={"ID":"e01fbe9e-fc87-4548-8223-64276fabad17","Type":"ContainerStarted","Data":"2dbb5cdfd8c827dd50d314ecac5b203727cc146039ffd48cb45679134dd1c3c4"}
Sep 11 18:00:14.466986 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:14.466962    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln" event={"ID":"6af40138-8dae-46c2-a76c-5163b937a4a8","Type":"ContainerStarted","Data":"e9eebd6c229d914a1159bb7f621dea911ebd55dbc74e2f6b6bc4a9ae4a32f7f4"}
Sep 11 18:00:14.467105 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:14.467092    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln"
Sep 11 18:00:14.487894 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:14.487847    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln" podStartSLOduration=1.575837508 podStartE2EDuration="4.487828894s" totalImagesPullingTime="2.911991386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:00:14.486782405 +0000 UTC m=+372.330017841" watchObservedRunningTime="2026-09-11 18:00:14.487828894 +0000 UTC m=+372.331064335"
Sep 11 18:00:18.488377 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:18.488337    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7" event={"ID":"e01fbe9e-fc87-4548-8223-64276fabad17","Type":"ContainerStarted","Data":"b50de0edf1ed4b8240c5deacedceda1a77a036e51a53c2c8097170238fc6b84b"}
Sep 11 18:00:19.494392 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:19.494359    2560 generic.go:356] "Generic (PLEG): container finished" podID="e01fbe9e-fc87-4548-8223-64276fabad17" containerID="b50de0edf1ed4b8240c5deacedceda1a77a036e51a53c2c8097170238fc6b84b" exitCode=0
Sep 11 18:00:19.494761 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:19.494407    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7" event={"ID":"e01fbe9e-fc87-4548-8223-64276fabad17","Type":"ContainerDied","Data":"b50de0edf1ed4b8240c5deacedceda1a77a036e51a53c2c8097170238fc6b84b"}
Sep 11 18:00:20.501102 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:20.501069    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7" event={"ID":"e01fbe9e-fc87-4548-8223-64276fabad17","Type":"ContainerStarted","Data":"ddb2cfe51cc51154ec0343daed0ace13ab57f4252633520ca529c8c8cdbab214"}
Sep 11 18:00:20.501504 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:20.501176    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"
Sep 11 18:00:20.526213 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:20.526164    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7" podStartSLOduration=3.095095776 podStartE2EDuration="8.526145575s" totalImagesPullingTime="4.335743071s" totalInitContainerRuntime="1.095306728s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:00:20.523936929 +0000 UTC m=+378.367172371" watchObservedRunningTime="2026-09-11 18:00:20.526145575 +0000 UTC m=+378.369381016"
Sep 11 18:00:25.473430 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:25.473396    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-ffbd766d4-9mjln"
Sep 11 18:00:31.508060 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:31.508023    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7"
Sep 11 18:00:33.664371 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:33.664338    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7ff57cc6b9-bnldt"]
Sep 11 18:00:33.673362 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:33.673341    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-bnldt"
Sep 11 18:00:33.677339 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:33.677311    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 18:00:33.677494 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:33.677430    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 18:00:33.677700 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:33.677679    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 18:00:33.678586 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:33.678569    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 18:00:33.678700 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:33.678603    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mjp97\""
Sep 11 18:00:33.693359 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:33.693332    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7ff57cc6b9-bnldt"]
Sep 11 18:00:33.736239 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:33.736203    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snzl9\" (UniqueName: \"kubernetes.io/projected/cd43172e-8f56-47b0-a780-fa9232ff2d41-kube-api-access-snzl9\") pod \"kube-auth-proxy-7ff57cc6b9-bnldt\" (UID: \"cd43172e-8f56-47b0-a780-fa9232ff2d41\") " pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-bnldt"
Sep 11 18:00:33.736398 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:33.736265    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd43172e-8f56-47b0-a780-fa9232ff2d41-tmp\") pod \"kube-auth-proxy-7ff57cc6b9-bnldt\" (UID: \"cd43172e-8f56-47b0-a780-fa9232ff2d41\") " pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-bnldt"
Sep 11 18:00:33.736398 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:33.736296    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd43172e-8f56-47b0-a780-fa9232ff2d41-tls-certs\") pod \"kube-auth-proxy-7ff57cc6b9-bnldt\" (UID: \"cd43172e-8f56-47b0-a780-fa9232ff2d41\") " pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-bnldt"
Sep 11 18:00:33.837363 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:33.837328    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snzl9\" (UniqueName: \"kubernetes.io/projected/cd43172e-8f56-47b0-a780-fa9232ff2d41-kube-api-access-snzl9\") pod \"kube-auth-proxy-7ff57cc6b9-bnldt\" (UID: \"cd43172e-8f56-47b0-a780-fa9232ff2d41\") " pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-bnldt"
Sep 11 18:00:33.837544 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:33.837376    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd43172e-8f56-47b0-a780-fa9232ff2d41-tmp\") pod \"kube-auth-proxy-7ff57cc6b9-bnldt\" (UID: \"cd43172e-8f56-47b0-a780-fa9232ff2d41\") " pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-bnldt"
Sep 11 18:00:33.837544 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:33.837495    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd43172e-8f56-47b0-a780-fa9232ff2d41-tls-certs\") pod \"kube-auth-proxy-7ff57cc6b9-bnldt\" (UID: \"cd43172e-8f56-47b0-a780-fa9232ff2d41\") " pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-bnldt"
Sep 11 18:00:33.839841 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:33.839816    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd43172e-8f56-47b0-a780-fa9232ff2d41-tmp\") pod \"kube-auth-proxy-7ff57cc6b9-bnldt\" (UID: \"cd43172e-8f56-47b0-a780-fa9232ff2d41\") " pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-bnldt"
Sep 11 18:00:33.840079 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:33.840061    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd43172e-8f56-47b0-a780-fa9232ff2d41-tls-certs\") pod \"kube-auth-proxy-7ff57cc6b9-bnldt\" (UID: \"cd43172e-8f56-47b0-a780-fa9232ff2d41\") " pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-bnldt"
Sep 11 18:00:33.854143 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:33.854114    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snzl9\" (UniqueName: \"kubernetes.io/projected/cd43172e-8f56-47b0-a780-fa9232ff2d41-kube-api-access-snzl9\") pod \"kube-auth-proxy-7ff57cc6b9-bnldt\" (UID: \"cd43172e-8f56-47b0-a780-fa9232ff2d41\") " pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-bnldt"
Sep 11 18:00:33.982356 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:33.982326    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-bnldt"
Sep 11 18:00:34.259271 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:34.259246    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7ff57cc6b9-bnldt"]
Sep 11 18:00:34.261317 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:00:34.261291    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd43172e_8f56_47b0_a780_fa9232ff2d41.slice/crio-cff2610b50ff039cf91d63bc638804c427f18958268a73cfcc90c7fd879266df WatchSource:0}: Error finding container cff2610b50ff039cf91d63bc638804c427f18958268a73cfcc90c7fd879266df: Status 404 returned error can't find the container with id cff2610b50ff039cf91d63bc638804c427f18958268a73cfcc90c7fd879266df
Sep 11 18:00:34.575868 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:34.575784    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-bnldt" event={"ID":"cd43172e-8f56-47b0-a780-fa9232ff2d41","Type":"ContainerStarted","Data":"cff2610b50ff039cf91d63bc638804c427f18958268a73cfcc90c7fd879266df"}
Sep 11 18:00:37.592999 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:37.592966    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-bnldt" event={"ID":"cd43172e-8f56-47b0-a780-fa9232ff2d41","Type":"ContainerStarted","Data":"ce78f80892739dc01c098b6b118ba33bf19273c27210fd8982286351a318eaf8"}
Sep 11 18:00:37.611546 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:37.611491    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7ff57cc6b9-bnldt" podStartSLOduration=1.969095966 podStartE2EDuration="4.61147476s" totalImagesPullingTime="2.642378794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:00:37.610057494 +0000 UTC m=+395.453292950" watchObservedRunningTime="2026-09-11 18:00:37.61147476 +0000 UTC m=+395.454710201"
Sep 11 18:00:56.593622 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.593584    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79bff6d9c6-fwbmn"]
Sep 11 18:00:56.596501 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.596476    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.611636 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.611607    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79bff6d9c6-fwbmn"]
Sep 11 18:00:56.736546 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.736516    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fd520e35-84d7-4368-b0c9-96d5c3b10a15-console-oauth-config\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.736546 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.736551    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvgvv\" (UniqueName: \"kubernetes.io/projected/fd520e35-84d7-4368-b0c9-96d5c3b10a15-kube-api-access-jvgvv\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.736736 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.736582    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fd520e35-84d7-4368-b0c9-96d5c3b10a15-oauth-serving-cert\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.736736 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.736697    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fd520e35-84d7-4368-b0c9-96d5c3b10a15-service-ca\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.736828 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.736734    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd520e35-84d7-4368-b0c9-96d5c3b10a15-trusted-ca-bundle\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.736861 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.736821    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fd520e35-84d7-4368-b0c9-96d5c3b10a15-console-config\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.736898 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.736866    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fd520e35-84d7-4368-b0c9-96d5c3b10a15-console-serving-cert\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.838085 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.838038    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fd520e35-84d7-4368-b0c9-96d5c3b10a15-service-ca\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.838085 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.838086    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd520e35-84d7-4368-b0c9-96d5c3b10a15-trusted-ca-bundle\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.838308 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.838200    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fd520e35-84d7-4368-b0c9-96d5c3b10a15-console-config\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.838308 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.838246    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fd520e35-84d7-4368-b0c9-96d5c3b10a15-console-serving-cert\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.838418 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.838390    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fd520e35-84d7-4368-b0c9-96d5c3b10a15-console-oauth-config\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.838470 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.838438    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvgvv\" (UniqueName: \"kubernetes.io/projected/fd520e35-84d7-4368-b0c9-96d5c3b10a15-kube-api-access-jvgvv\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.838526 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.838499    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fd520e35-84d7-4368-b0c9-96d5c3b10a15-oauth-serving-cert\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.838929 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.838900    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fd520e35-84d7-4368-b0c9-96d5c3b10a15-service-ca\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.839225 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.838907    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fd520e35-84d7-4368-b0c9-96d5c3b10a15-console-config\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.839225 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.838998    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd520e35-84d7-4368-b0c9-96d5c3b10a15-trusted-ca-bundle\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.839446 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.839423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fd520e35-84d7-4368-b0c9-96d5c3b10a15-oauth-serving-cert\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.840934 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.840907    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fd520e35-84d7-4368-b0c9-96d5c3b10a15-console-serving-cert\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.841032 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.840946    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fd520e35-84d7-4368-b0c9-96d5c3b10a15-console-oauth-config\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.848653 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.848608    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvgvv\" (UniqueName: \"kubernetes.io/projected/fd520e35-84d7-4368-b0c9-96d5c3b10a15-kube-api-access-jvgvv\") pod \"console-79bff6d9c6-fwbmn\" (UID: \"fd520e35-84d7-4368-b0c9-96d5c3b10a15\") " pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:56.907930 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:56.907901    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:00:57.182282 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:57.182253    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79bff6d9c6-fwbmn"]
Sep 11 18:00:57.184559 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:00:57.184532    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd520e35_84d7_4368_b0c9_96d5c3b10a15.slice/crio-f9d49e3d7f27ec2a77b266926bbf9835802cb153cd0076f8dadce23edd83032b WatchSource:0}: Error finding container f9d49e3d7f27ec2a77b266926bbf9835802cb153cd0076f8dadce23edd83032b: Status 404 returned error can't find the container with id f9d49e3d7f27ec2a77b266926bbf9835802cb153cd0076f8dadce23edd83032b
Sep 11 18:00:57.692701 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:57.692661    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bff6d9c6-fwbmn" event={"ID":"fd520e35-84d7-4368-b0c9-96d5c3b10a15","Type":"ContainerStarted","Data":"c9b7d1fc0ad95e3f9e333390aac2abbaadbb5bbc3ec301b2a1a9027bc483097b"}
Sep 11 18:00:57.693087 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:57.692709    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bff6d9c6-fwbmn" event={"ID":"fd520e35-84d7-4368-b0c9-96d5c3b10a15","Type":"ContainerStarted","Data":"f9d49e3d7f27ec2a77b266926bbf9835802cb153cd0076f8dadce23edd83032b"}
Sep 11 18:00:57.715468 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:00:57.715421    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79bff6d9c6-fwbmn" podStartSLOduration=1.715404368 podStartE2EDuration="1.715404368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:00:57.714149648 +0000 UTC m=+415.557385090" watchObservedRunningTime="2026-09-11 18:00:57.715404368 +0000 UTC m=+415.558639808"
Sep 11 18:01:06.908917 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:06.908879    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:01:06.909421 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:06.908956    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:01:06.913572 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:06.913549    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:01:07.740699 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:07.740671    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79bff6d9c6-fwbmn"
Sep 11 18:01:07.801668 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:07.801636    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bf7df85d8-jwv2l"]
Sep 11 18:01:32.828621 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:32.828527    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bf7df85d8-jwv2l" podUID="e846f82b-2faf-460b-9160-360df9136313" containerName="console" containerID="cri-o://e92e82c7b995b1fdac4f641b9ce59fbfab26dbdfb9a54cc1ac0d45199966c1ef" gracePeriod=15
Sep 11 18:01:33.078882 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.078827    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 18:01:33.144015 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.143979    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-console-config\") pod \"e846f82b-2faf-460b-9160-360df9136313\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") "
Sep 11 18:01:33.144192 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.144024    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpm4c\" (UniqueName: \"kubernetes.io/projected/e846f82b-2faf-460b-9160-360df9136313-kube-api-access-kpm4c\") pod \"e846f82b-2faf-460b-9160-360df9136313\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") "
Sep 11 18:01:33.144192 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.144050    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-service-ca\") pod \"e846f82b-2faf-460b-9160-360df9136313\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") "
Sep 11 18:01:33.144192 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.144175    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e846f82b-2faf-460b-9160-360df9136313-console-serving-cert\") pod \"e846f82b-2faf-460b-9160-360df9136313\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") "
Sep 11 18:01:33.144355 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.144223    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-oauth-serving-cert\") pod \"e846f82b-2faf-460b-9160-360df9136313\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") "
Sep 11 18:01:33.144355 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.144293    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e846f82b-2faf-460b-9160-360df9136313-console-oauth-config\") pod \"e846f82b-2faf-460b-9160-360df9136313\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") "
Sep 11 18:01:33.144355 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.144335    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-trusted-ca-bundle\") pod \"e846f82b-2faf-460b-9160-360df9136313\" (UID: \"e846f82b-2faf-460b-9160-360df9136313\") "
Sep 11 18:01:33.144508 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.144449    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-console-config" (OuterVolumeSpecName: "console-config") pod "e846f82b-2faf-460b-9160-360df9136313" (UID: "e846f82b-2faf-460b-9160-360df9136313"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:01:33.144564 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.144509    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-service-ca" (OuterVolumeSpecName: "service-ca") pod "e846f82b-2faf-460b-9160-360df9136313" (UID: "e846f82b-2faf-460b-9160-360df9136313"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:01:33.144774 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.144633    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e846f82b-2faf-460b-9160-360df9136313" (UID: "e846f82b-2faf-460b-9160-360df9136313"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:01:33.144774 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.144732    2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-console-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:01:33.144901 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.144790    2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:01:33.144901 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.144848    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e846f82b-2faf-460b-9160-360df9136313" (UID: "e846f82b-2faf-460b-9160-360df9136313"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:01:33.146434 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.146411    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e846f82b-2faf-460b-9160-360df9136313-kube-api-access-kpm4c" (OuterVolumeSpecName: "kube-api-access-kpm4c") pod "e846f82b-2faf-460b-9160-360df9136313" (UID: "e846f82b-2faf-460b-9160-360df9136313"). InnerVolumeSpecName "kube-api-access-kpm4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:01:33.146611 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.146584    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e846f82b-2faf-460b-9160-360df9136313-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e846f82b-2faf-460b-9160-360df9136313" (UID: "e846f82b-2faf-460b-9160-360df9136313"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:01:33.146798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.146776    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e846f82b-2faf-460b-9160-360df9136313-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e846f82b-2faf-460b-9160-360df9136313" (UID: "e846f82b-2faf-460b-9160-360df9136313"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:01:33.245378 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.245341    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpm4c\" (UniqueName: \"kubernetes.io/projected/e846f82b-2faf-460b-9160-360df9136313-kube-api-access-kpm4c\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:01:33.245378 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.245370    2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e846f82b-2faf-460b-9160-360df9136313-console-serving-cert\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:01:33.245378 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.245380    2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-oauth-serving-cert\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:01:33.245378 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.245388    2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e846f82b-2faf-460b-9160-360df9136313-console-oauth-config\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:01:33.245636 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.245396    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e846f82b-2faf-460b-9160-360df9136313-trusted-ca-bundle\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:01:33.861430 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.861395    2560 generic.go:356] "Generic (PLEG): container finished" podID="e846f82b-2faf-460b-9160-360df9136313" containerID="e92e82c7b995b1fdac4f641b9ce59fbfab26dbdfb9a54cc1ac0d45199966c1ef" exitCode=0
Sep 11 18:01:33.861887 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.861443    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf7df85d8-jwv2l" event={"ID":"e846f82b-2faf-460b-9160-360df9136313","Type":"ContainerDied","Data":"e92e82c7b995b1fdac4f641b9ce59fbfab26dbdfb9a54cc1ac0d45199966c1ef"}
Sep 11 18:01:33.861887 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.861450    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bf7df85d8-jwv2l"
Sep 11 18:01:33.861887 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.861504    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bf7df85d8-jwv2l" event={"ID":"e846f82b-2faf-460b-9160-360df9136313","Type":"ContainerDied","Data":"ba9f4919371de8de1c5f91e79baeeaa6a8388ee50ad32692b5113da3f1562669"}
Sep 11 18:01:33.861887 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.861523    2560 scope.go:117] "RemoveContainer" containerID="e92e82c7b995b1fdac4f641b9ce59fbfab26dbdfb9a54cc1ac0d45199966c1ef"
Sep 11 18:01:33.870635 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.870615    2560 scope.go:117] "RemoveContainer" containerID="e92e82c7b995b1fdac4f641b9ce59fbfab26dbdfb9a54cc1ac0d45199966c1ef"
Sep 11 18:01:33.870991 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:01:33.870970    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e92e82c7b995b1fdac4f641b9ce59fbfab26dbdfb9a54cc1ac0d45199966c1ef\": container with ID starting with e92e82c7b995b1fdac4f641b9ce59fbfab26dbdfb9a54cc1ac0d45199966c1ef not found: ID does not exist" containerID="e92e82c7b995b1fdac4f641b9ce59fbfab26dbdfb9a54cc1ac0d45199966c1ef"
Sep 11 18:01:33.871066 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.871000    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e92e82c7b995b1fdac4f641b9ce59fbfab26dbdfb9a54cc1ac0d45199966c1ef"} err="failed to get container status \"e92e82c7b995b1fdac4f641b9ce59fbfab26dbdfb9a54cc1ac0d45199966c1ef\": rpc error: code = NotFound desc = could not find container \"e92e82c7b995b1fdac4f641b9ce59fbfab26dbdfb9a54cc1ac0d45199966c1ef\": container with ID starting with e92e82c7b995b1fdac4f641b9ce59fbfab26dbdfb9a54cc1ac0d45199966c1ef not found: ID does not exist"
Sep 11 18:01:33.883336 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.883315    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bf7df85d8-jwv2l"]
Sep 11 18:01:33.887086 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:33.887068    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bf7df85d8-jwv2l"]
Sep 11 18:01:34.687689 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:01:34.687656    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e846f82b-2faf-460b-9160-360df9136313" path="/var/lib/kubelet/pods/e846f82b-2faf-460b-9160-360df9136313/volumes"
Sep 11 18:02:12.998599 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:12.998563    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-68g6n"]
Sep 11 18:02:12.999078 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:12.998909    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e846f82b-2faf-460b-9160-360df9136313" containerName="console"
Sep 11 18:02:12.999078 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:12.998922    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e846f82b-2faf-460b-9160-360df9136313" containerName="console"
Sep 11 18:02:12.999078 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:12.998987    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e846f82b-2faf-460b-9160-360df9136313" containerName="console"
Sep 11 18:02:13.003240 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:13.003222    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-68g6n"
Sep 11 18:02:13.005702 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:13.005683    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 11 18:02:13.005882 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:13.005863    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 11 18:02:13.005988 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:13.005871    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 18:02:13.006947 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:13.006926    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dwv54\""
Sep 11 18:02:13.007095 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:13.006926    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 18:02:13.011461 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:13.011437    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-68g6n"]
Sep 11 18:02:13.084552 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:13.084522    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc5d3dba-0fae-4dcf-a68f-0a0071f07ada-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-68g6n\" (UID: \"bc5d3dba-0fae-4dcf-a68f-0a0071f07ada\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-68g6n"
Sep 11 18:02:13.084731 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:13.084571    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f74xt\" (UniqueName: \"kubernetes.io/projected/bc5d3dba-0fae-4dcf-a68f-0a0071f07ada-kube-api-access-f74xt\") pod \"kuadrant-console-plugin-fbdd4f445-68g6n\" (UID: \"bc5d3dba-0fae-4dcf-a68f-0a0071f07ada\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-68g6n"
Sep 11 18:02:13.084731 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:13.084644    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc5d3dba-0fae-4dcf-a68f-0a0071f07ada-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-68g6n\" (UID: \"bc5d3dba-0fae-4dcf-a68f-0a0071f07ada\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-68g6n"
Sep 11 18:02:13.185381 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:13.185346    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc5d3dba-0fae-4dcf-a68f-0a0071f07ada-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-68g6n\" (UID: \"bc5d3dba-0fae-4dcf-a68f-0a0071f07ada\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-68g6n"
Sep 11 18:02:13.185534 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:13.185403    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f74xt\" (UniqueName: \"kubernetes.io/projected/bc5d3dba-0fae-4dcf-a68f-0a0071f07ada-kube-api-access-f74xt\") pod \"kuadrant-console-plugin-fbdd4f445-68g6n\" (UID: \"bc5d3dba-0fae-4dcf-a68f-0a0071f07ada\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-68g6n"
Sep 11 18:02:13.185534 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:13.185467    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc5d3dba-0fae-4dcf-a68f-0a0071f07ada-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-68g6n\" (UID: \"bc5d3dba-0fae-4dcf-a68f-0a0071f07ada\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-68g6n"
Sep 11 18:02:13.186128 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:13.186109    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bc5d3dba-0fae-4dcf-a68f-0a0071f07ada-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-68g6n\" (UID: \"bc5d3dba-0fae-4dcf-a68f-0a0071f07ada\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-68g6n"
Sep 11 18:02:13.188069 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:13.188050    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/bc5d3dba-0fae-4dcf-a68f-0a0071f07ada-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-68g6n\" (UID: \"bc5d3dba-0fae-4dcf-a68f-0a0071f07ada\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-68g6n"
Sep 11 18:02:13.197036 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:13.197014    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f74xt\" (UniqueName: \"kubernetes.io/projected/bc5d3dba-0fae-4dcf-a68f-0a0071f07ada-kube-api-access-f74xt\") pod \"kuadrant-console-plugin-fbdd4f445-68g6n\" (UID: \"bc5d3dba-0fae-4dcf-a68f-0a0071f07ada\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-68g6n"
Sep 11 18:02:13.312552 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:13.312469    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-68g6n"
Sep 11 18:02:13.585187 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:13.585157    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-68g6n"]
Sep 11 18:02:13.587156 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:02:13.587129    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc5d3dba_0fae_4dcf_a68f_0a0071f07ada.slice/crio-7c1b36bf81b61bc33a1e7805e4cc3a1423bbf1a20efabc316b38dc208d3b11c1 WatchSource:0}: Error finding container 7c1b36bf81b61bc33a1e7805e4cc3a1423bbf1a20efabc316b38dc208d3b11c1: Status 404 returned error can't find the container with id 7c1b36bf81b61bc33a1e7805e4cc3a1423bbf1a20efabc316b38dc208d3b11c1
Sep 11 18:02:14.047525 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:14.047491    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-68g6n" event={"ID":"bc5d3dba-0fae-4dcf-a68f-0a0071f07ada","Type":"ContainerStarted","Data":"7c1b36bf81b61bc33a1e7805e4cc3a1423bbf1a20efabc316b38dc208d3b11c1"}
Sep 11 18:02:19.074005 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:19.073967    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-68g6n" event={"ID":"bc5d3dba-0fae-4dcf-a68f-0a0071f07ada","Type":"ContainerStarted","Data":"6c7816da49693890081eda9558026e0a080393434b561e414f44c885a57d1a76"}
Sep 11 18:02:19.097594 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:19.097539    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-68g6n" podStartSLOduration=2.5399233199999998 podStartE2EDuration="7.097522313s" totalImagesPullingTime="4.557598993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:02:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:02:19.096020368 +0000 UTC m=+496.939255809" watchObservedRunningTime="2026-09-11 18:02:19.097522313 +0000 UTC m=+496.940757801"
Sep 11 18:02:57.449344 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:57.449312    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:02:57.452462 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:57.452447    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"
Sep 11 18:02:57.455099 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:57.455079    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 18:02:57.463590 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:57.463568    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:02:57.499612 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:57.499585    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:02:57.501181 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:57.501160    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vkvf\" (UniqueName: \"kubernetes.io/projected/69c6a8d9-6459-4b7e-b0b6-f2edbb2f6ec7-kube-api-access-2vkvf\") pod \"limitador-limitador-84994f7ffc-t5cm4\" (UID: \"69c6a8d9-6459-4b7e-b0b6-f2edbb2f6ec7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"
Sep 11 18:02:57.501240 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:57.501210    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/69c6a8d9-6459-4b7e-b0b6-f2edbb2f6ec7-config-file\") pod \"limitador-limitador-84994f7ffc-t5cm4\" (UID: \"69c6a8d9-6459-4b7e-b0b6-f2edbb2f6ec7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"
Sep 11 18:02:57.602577 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:57.602551    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/69c6a8d9-6459-4b7e-b0b6-f2edbb2f6ec7-config-file\") pod \"limitador-limitador-84994f7ffc-t5cm4\" (UID: \"69c6a8d9-6459-4b7e-b0b6-f2edbb2f6ec7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"
Sep 11 18:02:57.602722 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:57.602635    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vkvf\" (UniqueName: \"kubernetes.io/projected/69c6a8d9-6459-4b7e-b0b6-f2edbb2f6ec7-kube-api-access-2vkvf\") pod \"limitador-limitador-84994f7ffc-t5cm4\" (UID: \"69c6a8d9-6459-4b7e-b0b6-f2edbb2f6ec7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"
Sep 11 18:02:57.603204 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:57.603185    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/69c6a8d9-6459-4b7e-b0b6-f2edbb2f6ec7-config-file\") pod \"limitador-limitador-84994f7ffc-t5cm4\" (UID: \"69c6a8d9-6459-4b7e-b0b6-f2edbb2f6ec7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"
Sep 11 18:02:57.611287 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:57.611264    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vkvf\" (UniqueName: \"kubernetes.io/projected/69c6a8d9-6459-4b7e-b0b6-f2edbb2f6ec7-kube-api-access-2vkvf\") pod \"limitador-limitador-84994f7ffc-t5cm4\" (UID: \"69c6a8d9-6459-4b7e-b0b6-f2edbb2f6ec7\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"
Sep 11 18:02:57.762935 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:57.762899    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"
Sep 11 18:02:58.031912 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:58.031892    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:02:58.034159 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:02:58.034133    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c6a8d9_6459_4b7e_b0b6_f2edbb2f6ec7.slice/crio-3b0d2b169893f4a395a4b07367be77dc8bcc3f69ec5297ecf8fdfd7334650ebe WatchSource:0}: Error finding container 3b0d2b169893f4a395a4b07367be77dc8bcc3f69ec5297ecf8fdfd7334650ebe: Status 404 returned error can't find the container with id 3b0d2b169893f4a395a4b07367be77dc8bcc3f69ec5297ecf8fdfd7334650ebe
Sep 11 18:02:58.249446 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:02:58.249411    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-t5cm4" event={"ID":"69c6a8d9-6459-4b7e-b0b6-f2edbb2f6ec7","Type":"ContainerStarted","Data":"3b0d2b169893f4a395a4b07367be77dc8bcc3f69ec5297ecf8fdfd7334650ebe"}
Sep 11 18:03:01.268915 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:03:01.268877    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-t5cm4" event={"ID":"69c6a8d9-6459-4b7e-b0b6-f2edbb2f6ec7","Type":"ContainerStarted","Data":"de303c360d1e85ce4e403cc20c41e9e6cf2096e24fb012d72bd82378e1cefcb5"}
Sep 11 18:03:01.269292 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:03:01.268950    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"
Sep 11 18:03:01.293332 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:03:01.293282    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-t5cm4" podStartSLOduration=2.009507166 podStartE2EDuration="4.293265989s" totalImagesPullingTime="2.283758823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:03:01.291954009 +0000 UTC m=+539.135189461" watchObservedRunningTime="2026-09-11 18:03:01.293265989 +0000 UTC m=+539.136501430"
Sep 11 18:03:12.274136 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:03:12.274105    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"
Sep 11 18:04:02.606249 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:04:02.606215    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/2.log"
Sep 11 18:04:02.621780 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:04:02.621736    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/2.log"
Sep 11 18:04:15.258669 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:04:15.258636    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:04:34.115693 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:04:34.115605    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:04:49.313603 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:04:49.313563    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:05:02.225476 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.225445    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"]
Sep 11 18:05:02.228789 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.228744    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:02.233023 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.233001    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 11 18:05:02.233150 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.233011    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-d5n2t\""
Sep 11 18:05:02.233150 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.233011    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 18:05:02.233268 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.233015    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 18:05:02.241704 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.241683    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"]
Sep 11 18:05:02.400199 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.400167    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6fcc7c5-7038-40a0-a002-6240f8be2549-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4\" (UID: \"a6fcc7c5-7038-40a0-a002-6240f8be2549\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:02.400403 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.400210    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6fcc7c5-7038-40a0-a002-6240f8be2549-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4\" (UID: \"a6fcc7c5-7038-40a0-a002-6240f8be2549\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:02.400403 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.400282    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6fcc7c5-7038-40a0-a002-6240f8be2549-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4\" (UID: \"a6fcc7c5-7038-40a0-a002-6240f8be2549\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:02.400403 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.400381    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pm8s\" (UniqueName: \"kubernetes.io/projected/a6fcc7c5-7038-40a0-a002-6240f8be2549-kube-api-access-5pm8s\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4\" (UID: \"a6fcc7c5-7038-40a0-a002-6240f8be2549\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:02.400573 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.400422    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6fcc7c5-7038-40a0-a002-6240f8be2549-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4\" (UID: \"a6fcc7c5-7038-40a0-a002-6240f8be2549\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:02.400573 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.400456    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6fcc7c5-7038-40a0-a002-6240f8be2549-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4\" (UID: \"a6fcc7c5-7038-40a0-a002-6240f8be2549\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:02.501543 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.501461    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6fcc7c5-7038-40a0-a002-6240f8be2549-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4\" (UID: \"a6fcc7c5-7038-40a0-a002-6240f8be2549\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:02.501543 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.501517    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pm8s\" (UniqueName: \"kubernetes.io/projected/a6fcc7c5-7038-40a0-a002-6240f8be2549-kube-api-access-5pm8s\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4\" (UID: \"a6fcc7c5-7038-40a0-a002-6240f8be2549\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:02.501729 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.501550    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6fcc7c5-7038-40a0-a002-6240f8be2549-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4\" (UID: \"a6fcc7c5-7038-40a0-a002-6240f8be2549\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:02.501729 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.501570    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6fcc7c5-7038-40a0-a002-6240f8be2549-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4\" (UID: \"a6fcc7c5-7038-40a0-a002-6240f8be2549\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:02.501729 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.501586    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6fcc7c5-7038-40a0-a002-6240f8be2549-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4\" (UID: \"a6fcc7c5-7038-40a0-a002-6240f8be2549\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:02.501729 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.501621    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6fcc7c5-7038-40a0-a002-6240f8be2549-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4\" (UID: \"a6fcc7c5-7038-40a0-a002-6240f8be2549\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:02.501958 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.501934    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6fcc7c5-7038-40a0-a002-6240f8be2549-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4\" (UID: \"a6fcc7c5-7038-40a0-a002-6240f8be2549\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:02.502019 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.502001    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6fcc7c5-7038-40a0-a002-6240f8be2549-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4\" (UID: \"a6fcc7c5-7038-40a0-a002-6240f8be2549\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:02.502074 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.502058    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6fcc7c5-7038-40a0-a002-6240f8be2549-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4\" (UID: \"a6fcc7c5-7038-40a0-a002-6240f8be2549\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:02.503987 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.503967    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6fcc7c5-7038-40a0-a002-6240f8be2549-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4\" (UID: \"a6fcc7c5-7038-40a0-a002-6240f8be2549\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:02.504253 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.504233    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6fcc7c5-7038-40a0-a002-6240f8be2549-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4\" (UID: \"a6fcc7c5-7038-40a0-a002-6240f8be2549\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:02.511979 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.511958    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pm8s\" (UniqueName: \"kubernetes.io/projected/a6fcc7c5-7038-40a0-a002-6240f8be2549-kube-api-access-5pm8s\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4\" (UID: \"a6fcc7c5-7038-40a0-a002-6240f8be2549\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:02.542528 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.542506    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-d5n2t\""
Sep 11 18:05:02.550275 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.550257    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:02.843013 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.842992    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"]
Sep 11 18:05:02.845134 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:05:02.845104    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6fcc7c5_7038_40a0_a002_6240f8be2549.slice/crio-3c827ccf870cfea558bbfba802ede59912faff9381b065e3a608f88ea04dd038 WatchSource:0}: Error finding container 3c827ccf870cfea558bbfba802ede59912faff9381b065e3a608f88ea04dd038: Status 404 returned error can't find the container with id 3c827ccf870cfea558bbfba802ede59912faff9381b065e3a608f88ea04dd038
Sep 11 18:05:02.846536 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:02.846516    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:05:03.242165 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:03.242126    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:05:03.829223 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:03.829190    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4" event={"ID":"a6fcc7c5-7038-40a0-a002-6240f8be2549","Type":"ContainerStarted","Data":"3c827ccf870cfea558bbfba802ede59912faff9381b065e3a608f88ea04dd038"}
Sep 11 18:05:04.835129 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:04.835097    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4" event={"ID":"a6fcc7c5-7038-40a0-a002-6240f8be2549","Type":"ContainerStarted","Data":"4d3e40daa816afbe82ce9a9db1f6bff6d580da784315c53674783a608a600021"}
Sep 11 18:05:04.858683 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:04.858637    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4" podStartSLOduration=1.68795198 podStartE2EDuration="2.858623499s" totalImagesPullingTime="1.170671519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:05:04.856655204 +0000 UTC m=+662.699890680" watchObservedRunningTime="2026-09-11 18:05:04.858623499 +0000 UTC m=+662.701858939"
Sep 11 18:05:12.550907 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:12.550854    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:12.551394 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:12.550941    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:12.563009 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:12.562987    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:12.882405 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:12.882324    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4"
Sep 11 18:05:16.720021 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.719991    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"]
Sep 11 18:05:16.723583 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.723565    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:16.727000 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.726982    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 11 18:05:16.737777 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.737738    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"]
Sep 11 18:05:16.834911 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.834878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5437bb20-46f3-4188-b578-f1a027c36b91-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsdqh\" (UID: \"5437bb20-46f3-4188-b578-f1a027c36b91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:16.835071 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.834921    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5437bb20-46f3-4188-b578-f1a027c36b91-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsdqh\" (UID: \"5437bb20-46f3-4188-b578-f1a027c36b91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:16.835071 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.834973    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5437bb20-46f3-4188-b578-f1a027c36b91-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsdqh\" (UID: \"5437bb20-46f3-4188-b578-f1a027c36b91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:16.835071 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.835017    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5437bb20-46f3-4188-b578-f1a027c36b91-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsdqh\" (UID: \"5437bb20-46f3-4188-b578-f1a027c36b91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:16.835071 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.835056    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5437bb20-46f3-4188-b578-f1a027c36b91-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsdqh\" (UID: \"5437bb20-46f3-4188-b578-f1a027c36b91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:16.835205 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.835081    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvjrv\" (UniqueName: \"kubernetes.io/projected/5437bb20-46f3-4188-b578-f1a027c36b91-kube-api-access-vvjrv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsdqh\" (UID: \"5437bb20-46f3-4188-b578-f1a027c36b91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:16.936390 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.936361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5437bb20-46f3-4188-b578-f1a027c36b91-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsdqh\" (UID: \"5437bb20-46f3-4188-b578-f1a027c36b91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:16.936562 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.936401    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5437bb20-46f3-4188-b578-f1a027c36b91-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsdqh\" (UID: \"5437bb20-46f3-4188-b578-f1a027c36b91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:16.936562 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.936444    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5437bb20-46f3-4188-b578-f1a027c36b91-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsdqh\" (UID: \"5437bb20-46f3-4188-b578-f1a027c36b91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:16.936562 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.936473    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5437bb20-46f3-4188-b578-f1a027c36b91-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsdqh\" (UID: \"5437bb20-46f3-4188-b578-f1a027c36b91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:16.936562 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.936515    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5437bb20-46f3-4188-b578-f1a027c36b91-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsdqh\" (UID: \"5437bb20-46f3-4188-b578-f1a027c36b91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:16.936562 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.936544    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvjrv\" (UniqueName: \"kubernetes.io/projected/5437bb20-46f3-4188-b578-f1a027c36b91-kube-api-access-vvjrv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsdqh\" (UID: \"5437bb20-46f3-4188-b578-f1a027c36b91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:16.936863 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.936844    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5437bb20-46f3-4188-b578-f1a027c36b91-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsdqh\" (UID: \"5437bb20-46f3-4188-b578-f1a027c36b91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:16.936925 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.936891    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5437bb20-46f3-4188-b578-f1a027c36b91-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsdqh\" (UID: \"5437bb20-46f3-4188-b578-f1a027c36b91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:16.936925 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.936915    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5437bb20-46f3-4188-b578-f1a027c36b91-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsdqh\" (UID: \"5437bb20-46f3-4188-b578-f1a027c36b91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:16.938944 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.938920    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5437bb20-46f3-4188-b578-f1a027c36b91-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsdqh\" (UID: \"5437bb20-46f3-4188-b578-f1a027c36b91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:16.939122 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.939107    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5437bb20-46f3-4188-b578-f1a027c36b91-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsdqh\" (UID: \"5437bb20-46f3-4188-b578-f1a027c36b91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:16.945977 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:16.945956    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvjrv\" (UniqueName: \"kubernetes.io/projected/5437bb20-46f3-4188-b578-f1a027c36b91-kube-api-access-vvjrv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tsdqh\" (UID: \"5437bb20-46f3-4188-b578-f1a027c36b91\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:17.034232 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:17.034168    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:17.320534 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:17.320482    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"]
Sep 11 18:05:17.322817 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:05:17.322786    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5437bb20_46f3_4188_b578_f1a027c36b91.slice/crio-3d258aad2f7dc97089b16d05d81ff4c66f0321305beaaa7381caf245d153aca7 WatchSource:0}: Error finding container 3d258aad2f7dc97089b16d05d81ff4c66f0321305beaaa7381caf245d153aca7: Status 404 returned error can't find the container with id 3d258aad2f7dc97089b16d05d81ff4c66f0321305beaaa7381caf245d153aca7
Sep 11 18:05:17.896571 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:17.896486    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh" event={"ID":"5437bb20-46f3-4188-b578-f1a027c36b91","Type":"ContainerStarted","Data":"fc6d161c9f01a059ea17f835774411c6f5cde7a8b012b94f71dd80d89f7e3b53"}
Sep 11 18:05:17.896571 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:17.896530    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh" event={"ID":"5437bb20-46f3-4188-b578-f1a027c36b91","Type":"ContainerStarted","Data":"3d258aad2f7dc97089b16d05d81ff4c66f0321305beaaa7381caf245d153aca7"}
Sep 11 18:05:17.922405 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:17.922356    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh" podStartSLOduration=1.724176599 podStartE2EDuration="1.922341999s" totalImagesPullingTime="198.1654ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:05:17.922022489 +0000 UTC m=+675.765257930" watchObservedRunningTime="2026-09-11 18:05:17.922341999 +0000 UTC m=+675.765577440"
Sep 11 18:05:18.019999 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:18.019969    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:05:26.230067 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:26.230033    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:05:27.035306 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:27.035276    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:27.035474 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:27.035326    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:27.047262 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:27.047234    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:27.952422 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:27.952398    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tsdqh"
Sep 11 18:05:37.923579 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:37.923545    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"]
Sep 11 18:05:37.927015 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:37.927001    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:37.931595 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:37.931574    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 11 18:05:37.944972 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:37.944949    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"]
Sep 11 18:05:38.026488 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.026457    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a90df0-a07e-4eb4-b174-dc330824af1a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbk42\" (UID: \"f5a90df0-a07e-4eb4-b174-dc330824af1a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:38.026634 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.026501    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a90df0-a07e-4eb4-b174-dc330824af1a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbk42\" (UID: \"f5a90df0-a07e-4eb4-b174-dc330824af1a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:38.026634 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.026518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5a90df0-a07e-4eb4-b174-dc330824af1a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbk42\" (UID: \"f5a90df0-a07e-4eb4-b174-dc330824af1a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:38.026634 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.026572    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s25gh\" (UniqueName: \"kubernetes.io/projected/f5a90df0-a07e-4eb4-b174-dc330824af1a-kube-api-access-s25gh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbk42\" (UID: \"f5a90df0-a07e-4eb4-b174-dc330824af1a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:38.026634 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.026602    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5a90df0-a07e-4eb4-b174-dc330824af1a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbk42\" (UID: \"f5a90df0-a07e-4eb4-b174-dc330824af1a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:38.026818 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.026673    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a90df0-a07e-4eb4-b174-dc330824af1a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbk42\" (UID: \"f5a90df0-a07e-4eb4-b174-dc330824af1a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:38.127536 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.127488    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a90df0-a07e-4eb4-b174-dc330824af1a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbk42\" (UID: \"f5a90df0-a07e-4eb4-b174-dc330824af1a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:38.127688 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.127557    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a90df0-a07e-4eb4-b174-dc330824af1a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbk42\" (UID: \"f5a90df0-a07e-4eb4-b174-dc330824af1a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:38.127688 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.127594    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a90df0-a07e-4eb4-b174-dc330824af1a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbk42\" (UID: \"f5a90df0-a07e-4eb4-b174-dc330824af1a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:38.127688 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.127607    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5a90df0-a07e-4eb4-b174-dc330824af1a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbk42\" (UID: \"f5a90df0-a07e-4eb4-b174-dc330824af1a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:38.127688 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.127631    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s25gh\" (UniqueName: \"kubernetes.io/projected/f5a90df0-a07e-4eb4-b174-dc330824af1a-kube-api-access-s25gh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbk42\" (UID: \"f5a90df0-a07e-4eb4-b174-dc330824af1a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:38.127688 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.127648    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5a90df0-a07e-4eb4-b174-dc330824af1a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbk42\" (UID: \"f5a90df0-a07e-4eb4-b174-dc330824af1a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:38.127971 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.127949    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5a90df0-a07e-4eb4-b174-dc330824af1a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbk42\" (UID: \"f5a90df0-a07e-4eb4-b174-dc330824af1a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:38.128038 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.128012    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5a90df0-a07e-4eb4-b174-dc330824af1a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbk42\" (UID: \"f5a90df0-a07e-4eb4-b174-dc330824af1a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:38.128099 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.128046    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5a90df0-a07e-4eb4-b174-dc330824af1a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbk42\" (UID: \"f5a90df0-a07e-4eb4-b174-dc330824af1a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:38.130131 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.130105    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5a90df0-a07e-4eb4-b174-dc330824af1a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbk42\" (UID: \"f5a90df0-a07e-4eb4-b174-dc330824af1a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:38.130357 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.130335    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5a90df0-a07e-4eb4-b174-dc330824af1a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbk42\" (UID: \"f5a90df0-a07e-4eb4-b174-dc330824af1a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:38.143043 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.143021    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s25gh\" (UniqueName: \"kubernetes.io/projected/f5a90df0-a07e-4eb4-b174-dc330824af1a-kube-api-access-s25gh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vbk42\" (UID: \"f5a90df0-a07e-4eb4-b174-dc330824af1a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:38.236461 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.236422    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:38.523134 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.523059    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"]
Sep 11 18:05:38.527145 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:05:38.527123    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a90df0_a07e_4eb4_b174_dc330824af1a.slice/crio-18f4a1a119d52d588e2070d71cca37992bad93a1fd2e4d3dae1632c2f4b9ad8e WatchSource:0}: Error finding container 18f4a1a119d52d588e2070d71cca37992bad93a1fd2e4d3dae1632c2f4b9ad8e: Status 404 returned error can't find the container with id 18f4a1a119d52d588e2070d71cca37992bad93a1fd2e4d3dae1632c2f4b9ad8e
Sep 11 18:05:38.993876 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.993847    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42" event={"ID":"f5a90df0-a07e-4eb4-b174-dc330824af1a","Type":"ContainerStarted","Data":"6316d0469a336f857581e9a63e65387260bd5095090a65b64b70105aef6cafda"}
Sep 11 18:05:38.994264 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:38.993894    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42" event={"ID":"f5a90df0-a07e-4eb4-b174-dc330824af1a","Type":"ContainerStarted","Data":"18f4a1a119d52d588e2070d71cca37992bad93a1fd2e4d3dae1632c2f4b9ad8e"}
Sep 11 18:05:39.034538 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:39.034487    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42" podStartSLOduration=1.851922845 podStartE2EDuration="2.034472045s" totalImagesPullingTime="182.5492ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:05:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:05:39.033365981 +0000 UTC m=+696.876601420" watchObservedRunningTime="2026-09-11 18:05:39.034472045 +0000 UTC m=+696.877707484"
Sep 11 18:05:39.939307 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:39.939278    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:05:48.237241 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:48.237196    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:48.237668 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:48.237268    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:48.249798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:48.249744    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:05:49.050984 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:05:49.050956    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vbk42"
Sep 11 18:06:00.844016 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:00.843979    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"]
Sep 11 18:06:00.847426 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:00.847404    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:00.850649 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:00.850629    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 18:06:00.859792 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:00.859770    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"]
Sep 11 18:06:00.920697 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:00.920661    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j99fc\" (UniqueName: \"kubernetes.io/projected/f48c605d-4dab-420f-9bf9-1301f822afe2-kube-api-access-j99fc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b46qk\" (UID: \"f48c605d-4dab-420f-9bf9-1301f822afe2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:00.920888 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:00.920726    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f48c605d-4dab-420f-9bf9-1301f822afe2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b46qk\" (UID: \"f48c605d-4dab-420f-9bf9-1301f822afe2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:00.920888 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:00.920791    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f48c605d-4dab-420f-9bf9-1301f822afe2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b46qk\" (UID: \"f48c605d-4dab-420f-9bf9-1301f822afe2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:00.920888 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:00.920809    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f48c605d-4dab-420f-9bf9-1301f822afe2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b46qk\" (UID: \"f48c605d-4dab-420f-9bf9-1301f822afe2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:00.920888 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:00.920841    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f48c605d-4dab-420f-9bf9-1301f822afe2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b46qk\" (UID: \"f48c605d-4dab-420f-9bf9-1301f822afe2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:00.920888 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:00.920888    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f48c605d-4dab-420f-9bf9-1301f822afe2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b46qk\" (UID: \"f48c605d-4dab-420f-9bf9-1301f822afe2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:01.021776 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:01.021710    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j99fc\" (UniqueName: \"kubernetes.io/projected/f48c605d-4dab-420f-9bf9-1301f822afe2-kube-api-access-j99fc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b46qk\" (UID: \"f48c605d-4dab-420f-9bf9-1301f822afe2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:01.021964 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:01.021816    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f48c605d-4dab-420f-9bf9-1301f822afe2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b46qk\" (UID: \"f48c605d-4dab-420f-9bf9-1301f822afe2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:01.021964 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:01.021862    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f48c605d-4dab-420f-9bf9-1301f822afe2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b46qk\" (UID: \"f48c605d-4dab-420f-9bf9-1301f822afe2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:01.021964 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:01.021885    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f48c605d-4dab-420f-9bf9-1301f822afe2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b46qk\" (UID: \"f48c605d-4dab-420f-9bf9-1301f822afe2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:01.021964 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:01.021933    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f48c605d-4dab-420f-9bf9-1301f822afe2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b46qk\" (UID: \"f48c605d-4dab-420f-9bf9-1301f822afe2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:01.022163 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:01.021975    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f48c605d-4dab-420f-9bf9-1301f822afe2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b46qk\" (UID: \"f48c605d-4dab-420f-9bf9-1301f822afe2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:01.022370 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:01.022339    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f48c605d-4dab-420f-9bf9-1301f822afe2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b46qk\" (UID: \"f48c605d-4dab-420f-9bf9-1301f822afe2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:01.022496 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:01.022376    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f48c605d-4dab-420f-9bf9-1301f822afe2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b46qk\" (UID: \"f48c605d-4dab-420f-9bf9-1301f822afe2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:01.022496 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:01.022478    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f48c605d-4dab-420f-9bf9-1301f822afe2-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b46qk\" (UID: \"f48c605d-4dab-420f-9bf9-1301f822afe2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:01.024333 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:01.024308    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f48c605d-4dab-420f-9bf9-1301f822afe2-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b46qk\" (UID: \"f48c605d-4dab-420f-9bf9-1301f822afe2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:01.024576 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:01.024554    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f48c605d-4dab-420f-9bf9-1301f822afe2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b46qk\" (UID: \"f48c605d-4dab-420f-9bf9-1301f822afe2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:01.033668 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:01.033648    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j99fc\" (UniqueName: \"kubernetes.io/projected/f48c605d-4dab-420f-9bf9-1301f822afe2-kube-api-access-j99fc\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-b46qk\" (UID: \"f48c605d-4dab-420f-9bf9-1301f822afe2\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:01.157327 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:01.157243    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:01.436426 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:01.436401    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"]
Sep 11 18:06:01.438420 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:06:01.438387    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf48c605d_4dab_420f_9bf9_1301f822afe2.slice/crio-541890e226a8284fd9b231b63256206685c452812628c549969f338568c9dc1d WatchSource:0}: Error finding container 541890e226a8284fd9b231b63256206685c452812628c549969f338568c9dc1d: Status 404 returned error can't find the container with id 541890e226a8284fd9b231b63256206685c452812628c549969f338568c9dc1d
Sep 11 18:06:02.102067 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:02.102029    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk" event={"ID":"f48c605d-4dab-420f-9bf9-1301f822afe2","Type":"ContainerStarted","Data":"48299f09aab35058f0de38b9241d3163939e4192d8c41ad1428d006039e2a4cd"}
Sep 11 18:06:02.102455 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:02.102081    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk" event={"ID":"f48c605d-4dab-420f-9bf9-1301f822afe2","Type":"ContainerStarted","Data":"541890e226a8284fd9b231b63256206685c452812628c549969f338568c9dc1d"}
Sep 11 18:06:02.137387 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:02.134204    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk" podStartSLOduration=1.919279599 podStartE2EDuration="2.134184327s" totalImagesPullingTime="214.904728ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:06:02.131109702 +0000 UTC m=+719.974345144" watchObservedRunningTime="2026-09-11 18:06:02.134184327 +0000 UTC m=+719.977419772"
Sep 11 18:06:03.100599 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:03.100566    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:06:11.157842 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:11.157800    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:11.158314 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:11.157858    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:11.170893 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:11.170867    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:06:12.162953 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:06:12.162927    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-b46qk"
Sep 11 18:07:19.200255 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:19.200224    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:07:29.502985 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.502903    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"]
Sep 11 18:07:29.507833 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.507803    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"
Sep 11 18:07:29.511901 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.511876    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-736ba3-serving-cert\""
Sep 11 18:07:29.512040 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.511931    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 18:07:29.512945 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.512924    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2sf5b\""
Sep 11 18:07:29.513056 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.512987    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:07:29.513056 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.513002    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 18:07:29.521712 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.521684    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"]
Sep 11 18:07:29.551098 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.551063    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9lt8\" (UniqueName: \"kubernetes.io/projected/c0c98b02-b638-4f04-83ec-7c4589a93efd-kube-api-access-f9lt8\") pod \"maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr\" (UID: \"c0c98b02-b638-4f04-83ec-7c4589a93efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"
Sep 11 18:07:29.551288 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.551131    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c98b02-b638-4f04-83ec-7c4589a93efd-openshift-service-ca\") pod \"maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr\" (UID: \"c0c98b02-b638-4f04-83ec-7c4589a93efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"
Sep 11 18:07:29.551288 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.551180    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c98b02-b638-4f04-83ec-7c4589a93efd-metrics-certs\") pod \"maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr\" (UID: \"c0c98b02-b638-4f04-83ec-7c4589a93efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"
Sep 11 18:07:29.551288 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.551253    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0c98b02-b638-4f04-83ec-7c4589a93efd-maas-api-tls\") pod \"maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr\" (UID: \"c0c98b02-b638-4f04-83ec-7c4589a93efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"
Sep 11 18:07:29.652364 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.652329    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c98b02-b638-4f04-83ec-7c4589a93efd-openshift-service-ca\") pod \"maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr\" (UID: \"c0c98b02-b638-4f04-83ec-7c4589a93efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"
Sep 11 18:07:29.652364 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.652364    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c98b02-b638-4f04-83ec-7c4589a93efd-metrics-certs\") pod \"maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr\" (UID: \"c0c98b02-b638-4f04-83ec-7c4589a93efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"
Sep 11 18:07:29.652571 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.652411    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0c98b02-b638-4f04-83ec-7c4589a93efd-maas-api-tls\") pod \"maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr\" (UID: \"c0c98b02-b638-4f04-83ec-7c4589a93efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"
Sep 11 18:07:29.652571 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.652502    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9lt8\" (UniqueName: \"kubernetes.io/projected/c0c98b02-b638-4f04-83ec-7c4589a93efd-kube-api-access-f9lt8\") pod \"maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr\" (UID: \"c0c98b02-b638-4f04-83ec-7c4589a93efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"
Sep 11 18:07:29.653127 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.653097    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c98b02-b638-4f04-83ec-7c4589a93efd-openshift-service-ca\") pod \"maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr\" (UID: \"c0c98b02-b638-4f04-83ec-7c4589a93efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"
Sep 11 18:07:29.655070 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.655050    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0c98b02-b638-4f04-83ec-7c4589a93efd-maas-api-tls\") pod \"maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr\" (UID: \"c0c98b02-b638-4f04-83ec-7c4589a93efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"
Sep 11 18:07:29.655156 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.655101    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c98b02-b638-4f04-83ec-7c4589a93efd-metrics-certs\") pod \"maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr\" (UID: \"c0c98b02-b638-4f04-83ec-7c4589a93efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"
Sep 11 18:07:29.663312 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.663292    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9lt8\" (UniqueName: \"kubernetes.io/projected/c0c98b02-b638-4f04-83ec-7c4589a93efd-kube-api-access-f9lt8\") pod \"maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr\" (UID: \"c0c98b02-b638-4f04-83ec-7c4589a93efd\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"
Sep 11 18:07:29.821655 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:29.821572    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"
Sep 11 18:07:30.124764 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:30.124713    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"]
Sep 11 18:07:30.124899 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:07:30.124815    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c98b02_b638_4f04_83ec_7c4589a93efd.slice/crio-59638cb663dd39f0d23c30da5526e2c9b94d68d6e3d1ac2ffee27ec9b4e7c627 WatchSource:0}: Error finding container 59638cb663dd39f0d23c30da5526e2c9b94d68d6e3d1ac2ffee27ec9b4e7c627: Status 404 returned error can't find the container with id 59638cb663dd39f0d23c30da5526e2c9b94d68d6e3d1ac2ffee27ec9b4e7c627
Sep 11 18:07:30.528583 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:30.528544    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr" event={"ID":"c0c98b02-b638-4f04-83ec-7c4589a93efd","Type":"ContainerStarted","Data":"59638cb663dd39f0d23c30da5526e2c9b94d68d6e3d1ac2ffee27ec9b4e7c627"}
Sep 11 18:07:32.541983 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:32.541946    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr" event={"ID":"c0c98b02-b638-4f04-83ec-7c4589a93efd","Type":"ContainerStarted","Data":"6089d7d50977ac420281b873f31f15dc2baa9ab1fa8877b8be4db234facbcdb4"}
Sep 11 18:07:32.542406 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:32.542069    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"
Sep 11 18:07:32.568246 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:32.568193    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr" podStartSLOduration=1.258555988 podStartE2EDuration="3.568175815s" totalImagesPullingTime="2.309619827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:32.566508738 +0000 UTC m=+810.409744192" watchObservedRunningTime="2026-09-11 18:07:32.568175815 +0000 UTC m=+810.411411256"
Sep 11 18:07:38.030837 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:38.030791    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:07:38.553413 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:38.553384    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"
Sep 11 18:07:39.442137 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:39.442102    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"]
Sep 11 18:07:39.445857 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:39.445834    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"
Sep 11 18:07:39.448412 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:39.448390    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-33958e97-serving-cert\""
Sep 11 18:07:39.460185 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:39.460158    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"]
Sep 11 18:07:39.547603 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:39.547566    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6632c831-fb4e-4fd2-8206-76adbee7a55e-openshift-service-ca\") pod \"maas-api-e2e-ait-33958e97-5989c8686c-m54h6\" (UID: \"6632c831-fb4e-4fd2-8206-76adbee7a55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"
Sep 11 18:07:39.547603 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:39.547608    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6632c831-fb4e-4fd2-8206-76adbee7a55e-metrics-certs\") pod \"maas-api-e2e-ait-33958e97-5989c8686c-m54h6\" (UID: \"6632c831-fb4e-4fd2-8206-76adbee7a55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"
Sep 11 18:07:39.547886 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:39.547667    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7wh4\" (UniqueName: \"kubernetes.io/projected/6632c831-fb4e-4fd2-8206-76adbee7a55e-kube-api-access-l7wh4\") pod \"maas-api-e2e-ait-33958e97-5989c8686c-m54h6\" (UID: \"6632c831-fb4e-4fd2-8206-76adbee7a55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"
Sep 11 18:07:39.547886 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:39.547695    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6632c831-fb4e-4fd2-8206-76adbee7a55e-maas-api-tls\") pod \"maas-api-e2e-ait-33958e97-5989c8686c-m54h6\" (UID: \"6632c831-fb4e-4fd2-8206-76adbee7a55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"
Sep 11 18:07:39.648450 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:39.648416    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6632c831-fb4e-4fd2-8206-76adbee7a55e-openshift-service-ca\") pod \"maas-api-e2e-ait-33958e97-5989c8686c-m54h6\" (UID: \"6632c831-fb4e-4fd2-8206-76adbee7a55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"
Sep 11 18:07:39.648450 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:39.648451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6632c831-fb4e-4fd2-8206-76adbee7a55e-metrics-certs\") pod \"maas-api-e2e-ait-33958e97-5989c8686c-m54h6\" (UID: \"6632c831-fb4e-4fd2-8206-76adbee7a55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"
Sep 11 18:07:39.648722 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:39.648486    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7wh4\" (UniqueName: \"kubernetes.io/projected/6632c831-fb4e-4fd2-8206-76adbee7a55e-kube-api-access-l7wh4\") pod \"maas-api-e2e-ait-33958e97-5989c8686c-m54h6\" (UID: \"6632c831-fb4e-4fd2-8206-76adbee7a55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"
Sep 11 18:07:39.648722 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:39.648531    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6632c831-fb4e-4fd2-8206-76adbee7a55e-maas-api-tls\") pod \"maas-api-e2e-ait-33958e97-5989c8686c-m54h6\" (UID: \"6632c831-fb4e-4fd2-8206-76adbee7a55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"
Sep 11 18:07:39.648722 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:07:39.648648    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-serving-cert: secret "maas-api-e2e-ait-33958e97-serving-cert" not found
Sep 11 18:07:39.648722 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:07:39.648714    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6632c831-fb4e-4fd2-8206-76adbee7a55e-maas-api-tls podName:6632c831-fb4e-4fd2-8206-76adbee7a55e nodeName:}" failed. No retries permitted until 2026-09-11 18:07:40.148694951 +0000 UTC m=+817.991930373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6632c831-fb4e-4fd2-8206-76adbee7a55e-maas-api-tls") pod "maas-api-e2e-ait-33958e97-5989c8686c-m54h6" (UID: "6632c831-fb4e-4fd2-8206-76adbee7a55e") : secret "maas-api-e2e-ait-33958e97-serving-cert" not found
Sep 11 18:07:39.649305 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:39.649282    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6632c831-fb4e-4fd2-8206-76adbee7a55e-openshift-service-ca\") pod \"maas-api-e2e-ait-33958e97-5989c8686c-m54h6\" (UID: \"6632c831-fb4e-4fd2-8206-76adbee7a55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"
Sep 11 18:07:39.651086 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:39.651065    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6632c831-fb4e-4fd2-8206-76adbee7a55e-metrics-certs\") pod \"maas-api-e2e-ait-33958e97-5989c8686c-m54h6\" (UID: \"6632c831-fb4e-4fd2-8206-76adbee7a55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"
Sep 11 18:07:39.661362 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:39.661341    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7wh4\" (UniqueName: \"kubernetes.io/projected/6632c831-fb4e-4fd2-8206-76adbee7a55e-kube-api-access-l7wh4\") pod \"maas-api-e2e-ait-33958e97-5989c8686c-m54h6\" (UID: \"6632c831-fb4e-4fd2-8206-76adbee7a55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"
Sep 11 18:07:40.154578 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:40.154542    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6632c831-fb4e-4fd2-8206-76adbee7a55e-maas-api-tls\") pod \"maas-api-e2e-ait-33958e97-5989c8686c-m54h6\" (UID: \"6632c831-fb4e-4fd2-8206-76adbee7a55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"
Sep 11 18:07:40.154777 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:07:40.154715    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-serving-cert: secret "maas-api-e2e-ait-33958e97-serving-cert" not found
Sep 11 18:07:40.154836 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:07:40.154807    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6632c831-fb4e-4fd2-8206-76adbee7a55e-maas-api-tls podName:6632c831-fb4e-4fd2-8206-76adbee7a55e nodeName:}" failed. No retries permitted until 2026-09-11 18:07:41.154788742 +0000 UTC m=+818.998024162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6632c831-fb4e-4fd2-8206-76adbee7a55e-maas-api-tls") pod "maas-api-e2e-ait-33958e97-5989c8686c-m54h6" (UID: "6632c831-fb4e-4fd2-8206-76adbee7a55e") : secret "maas-api-e2e-ait-33958e97-serving-cert" not found
Sep 11 18:07:41.166367 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:41.166329    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6632c831-fb4e-4fd2-8206-76adbee7a55e-maas-api-tls\") pod \"maas-api-e2e-ait-33958e97-5989c8686c-m54h6\" (UID: \"6632c831-fb4e-4fd2-8206-76adbee7a55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"
Sep 11 18:07:41.168912 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:41.168886    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6632c831-fb4e-4fd2-8206-76adbee7a55e-maas-api-tls\") pod \"maas-api-e2e-ait-33958e97-5989c8686c-m54h6\" (UID: \"6632c831-fb4e-4fd2-8206-76adbee7a55e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"
Sep 11 18:07:41.257097 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:41.256987    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"
Sep 11 18:07:41.534734 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:41.534710    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"]
Sep 11 18:07:41.536503 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:07:41.536478    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6632c831_fb4e_4fd2_8206_76adbee7a55e.slice/crio-6daa9a39f35cd5cce1101cbfb3bbeaee250983cee2680affa9f2293065bd610e WatchSource:0}: Error finding container 6daa9a39f35cd5cce1101cbfb3bbeaee250983cee2680affa9f2293065bd610e: Status 404 returned error can't find the container with id 6daa9a39f35cd5cce1101cbfb3bbeaee250983cee2680affa9f2293065bd610e
Sep 11 18:07:41.590010 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:41.589977    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6" event={"ID":"6632c831-fb4e-4fd2-8206-76adbee7a55e","Type":"ContainerStarted","Data":"6daa9a39f35cd5cce1101cbfb3bbeaee250983cee2680affa9f2293065bd610e"}
Sep 11 18:07:42.597292 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:42.597251    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6" event={"ID":"6632c831-fb4e-4fd2-8206-76adbee7a55e","Type":"ContainerStarted","Data":"98dde4c791d545d538d889e0f78f415e9785e65398bb67e2bbce11935ab90b32"}
Sep 11 18:07:42.597692 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:42.597316    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"
Sep 11 18:07:42.631194 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:42.631137    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6" podStartSLOduration=3.380537387 podStartE2EDuration="3.631121514s" totalImagesPullingTime="250.584127ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:42.629501316 +0000 UTC m=+820.472736757" watchObservedRunningTime="2026-09-11 18:07:42.631121514 +0000 UTC m=+820.474356954"
Sep 11 18:07:47.389835 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:47.389797    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"]
Sep 11 18:07:47.393549 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:47.393529    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"
Sep 11 18:07:47.396420 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:47.396400    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d59a02-serving-cert\""
Sep 11 18:07:47.407109 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:47.407081    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"]
Sep 11 18:07:47.524810 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:47.524744    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/000c8e16-03b0-406c-b818-355e531789a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7\" (UID: \"000c8e16-03b0-406c-b818-355e531789a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"
Sep 11 18:07:47.524998 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:47.524825    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj5mk\" (UniqueName: \"kubernetes.io/projected/000c8e16-03b0-406c-b818-355e531789a9-kube-api-access-tj5mk\") pod \"maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7\" (UID: \"000c8e16-03b0-406c-b818-355e531789a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"
Sep 11 18:07:47.524998 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:47.524865    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/000c8e16-03b0-406c-b818-355e531789a9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7\" (UID: \"000c8e16-03b0-406c-b818-355e531789a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"
Sep 11 18:07:47.524998 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:47.524879    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/000c8e16-03b0-406c-b818-355e531789a9-metrics-certs\") pod \"maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7\" (UID: \"000c8e16-03b0-406c-b818-355e531789a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"
Sep 11 18:07:47.625820 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:47.625788    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj5mk\" (UniqueName: \"kubernetes.io/projected/000c8e16-03b0-406c-b818-355e531789a9-kube-api-access-tj5mk\") pod \"maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7\" (UID: \"000c8e16-03b0-406c-b818-355e531789a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"
Sep 11 18:07:47.626015 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:47.625836    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/000c8e16-03b0-406c-b818-355e531789a9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7\" (UID: \"000c8e16-03b0-406c-b818-355e531789a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"
Sep 11 18:07:47.626015 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:47.625852    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/000c8e16-03b0-406c-b818-355e531789a9-metrics-certs\") pod \"maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7\" (UID: \"000c8e16-03b0-406c-b818-355e531789a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"
Sep 11 18:07:47.626015 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:47.625943    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/000c8e16-03b0-406c-b818-355e531789a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7\" (UID: \"000c8e16-03b0-406c-b818-355e531789a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"
Sep 11 18:07:47.626205 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:07:47.626025    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-serving-cert: secret "maas-api-e2e-inf-b-d59a02-serving-cert" not found
Sep 11 18:07:47.626205 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:07:47.626078    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/000c8e16-03b0-406c-b818-355e531789a9-maas-api-tls podName:000c8e16-03b0-406c-b818-355e531789a9 nodeName:}" failed. No retries permitted until 2026-09-11 18:07:48.126061942 +0000 UTC m=+825.969297361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/000c8e16-03b0-406c-b818-355e531789a9-maas-api-tls") pod "maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7" (UID: "000c8e16-03b0-406c-b818-355e531789a9") : secret "maas-api-e2e-inf-b-d59a02-serving-cert" not found
Sep 11 18:07:47.626477 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:47.626421    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/000c8e16-03b0-406c-b818-355e531789a9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7\" (UID: \"000c8e16-03b0-406c-b818-355e531789a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"
Sep 11 18:07:47.628861 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:47.628838    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/000c8e16-03b0-406c-b818-355e531789a9-metrics-certs\") pod \"maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7\" (UID: \"000c8e16-03b0-406c-b818-355e531789a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"
Sep 11 18:07:47.635060 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:47.635033    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj5mk\" (UniqueName: \"kubernetes.io/projected/000c8e16-03b0-406c-b818-355e531789a9-kube-api-access-tj5mk\") pod \"maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7\" (UID: \"000c8e16-03b0-406c-b818-355e531789a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"
Sep 11 18:07:48.130263 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:48.130227    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/000c8e16-03b0-406c-b818-355e531789a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7\" (UID: \"000c8e16-03b0-406c-b818-355e531789a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"
Sep 11 18:07:48.132870 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:48.132841    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/000c8e16-03b0-406c-b818-355e531789a9-maas-api-tls\") pod \"maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7\" (UID: \"000c8e16-03b0-406c-b818-355e531789a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"
Sep 11 18:07:48.304213 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:48.304176    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"
Sep 11 18:07:48.584143 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:48.584119    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"]
Sep 11 18:07:48.585243 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:07:48.585220    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod000c8e16_03b0_406c_b818_355e531789a9.slice/crio-5976256fddffd88943d0c8c2a76406f2f19b21689d343cec397f5da9a48296d3 WatchSource:0}: Error finding container 5976256fddffd88943d0c8c2a76406f2f19b21689d343cec397f5da9a48296d3: Status 404 returned error can't find the container with id 5976256fddffd88943d0c8c2a76406f2f19b21689d343cec397f5da9a48296d3
Sep 11 18:07:48.607084 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:48.607059    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"
Sep 11 18:07:48.628972 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:48.628882    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7" event={"ID":"000c8e16-03b0-406c-b818-355e531789a9","Type":"ContainerStarted","Data":"5976256fddffd88943d0c8c2a76406f2f19b21689d343cec397f5da9a48296d3"}
Sep 11 18:07:49.635351 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:49.635313    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7" event={"ID":"000c8e16-03b0-406c-b818-355e531789a9","Type":"ContainerStarted","Data":"a3e693e5537eaca1a5a952cc55985cbee3df27501db767bda18f0839a7e8eb1d"}
Sep 11 18:07:49.635889 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:49.635408    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"
Sep 11 18:07:49.657890 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:49.657837    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7" podStartSLOduration=2.271855053 podStartE2EDuration="2.657820414s" totalImagesPullingTime="385.965361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:07:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:07:49.656537539 +0000 UTC m=+827.499772982" watchObservedRunningTime="2026-09-11 18:07:49.657820414 +0000 UTC m=+827.501055906"
Sep 11 18:07:55.646496 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:07:55.646466    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"
Sep 11 18:08:06.698059 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.698027    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"]
Sep 11 18:08:06.703703 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.703678    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:06.706501 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.706475    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f8a2d4\"/\"openshift-service-ca.crt\""
Sep 11 18:08:06.707592 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.707575    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f8a2d4\"/\"simulator-w0-f8a2d4-kserve-self-signed-certs\""
Sep 11 18:08:06.707681 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.707575    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f8a2d4\"/\"kube-root-ca.crt\""
Sep 11 18:08:06.707681 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.707639    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f8a2d4\"/\"default-dockercfg-l4gw2\""
Sep 11 18:08:06.711648 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.711300    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"]
Sep 11 18:08:06.798604 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.798565    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-home\") pod \"simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") " pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:06.798830 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.798627    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-model-cache\") pod \"simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") " pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:06.798830 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.798672    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35248822-aefa-4202-a5d2-da92d6b5fc51-tls-certs\") pod \"simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") " pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:06.798830 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.798783    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-tmp-dir\") pod \"simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") " pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:06.798830 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.798824    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tbqs\" (UniqueName: \"kubernetes.io/projected/35248822-aefa-4202-a5d2-da92d6b5fc51-kube-api-access-6tbqs\") pod \"simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") " pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:06.799044 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.798881    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-dshm\") pod \"simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") " pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:06.900227 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.900188    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35248822-aefa-4202-a5d2-da92d6b5fc51-tls-certs\") pod \"simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") " pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:06.900448 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.900257    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-tmp-dir\") pod \"simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") " pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:06.900448 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.900290    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tbqs\" (UniqueName: \"kubernetes.io/projected/35248822-aefa-4202-a5d2-da92d6b5fc51-kube-api-access-6tbqs\") pod \"simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") " pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:06.900448 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.900318    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-dshm\") pod \"simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") " pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:06.900448 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.900340    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-home\") pod \"simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") " pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:06.900448 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.900373    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-model-cache\") pod \"simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") " pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:06.900738 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.900721    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-tmp-dir\") pod \"simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") " pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:06.900852 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.900829    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-home\") pod \"simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") " pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:06.900902 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.900856    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-model-cache\") pod \"simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") " pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:06.902801 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.902780    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-dshm\") pod \"simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") " pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:06.902998 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.902981    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35248822-aefa-4202-a5d2-da92d6b5fc51-tls-certs\") pod \"simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") " pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:06.910210 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.910187    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tbqs\" (UniqueName: \"kubernetes.io/projected/35248822-aefa-4202-a5d2-da92d6b5fc51-kube-api-access-6tbqs\") pod \"simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") " pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:06.943546 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:06.943496    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:08:07.014202 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:07.014170    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:07.325427 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:07.325400    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"]
Sep 11 18:08:07.327146 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:08:07.327119    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35248822_aefa_4202_a5d2_da92d6b5fc51.slice/crio-b1e96305c53881b12fb71c0333f809108eb7052cf737246df7b5399d2181da57 WatchSource:0}: Error finding container b1e96305c53881b12fb71c0333f809108eb7052cf737246df7b5399d2181da57: Status 404 returned error can't find the container with id b1e96305c53881b12fb71c0333f809108eb7052cf737246df7b5399d2181da57
Sep 11 18:08:07.725353 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:07.725313    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl" event={"ID":"35248822-aefa-4202-a5d2-da92d6b5fc51","Type":"ContainerStarted","Data":"b6c223beda3975f9845426bb5b062ff2f7dcaeb6c83b99ff66e350d171d0dc35"}
Sep 11 18:08:07.725764 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:07.725364    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl" event={"ID":"35248822-aefa-4202-a5d2-da92d6b5fc51","Type":"ContainerStarted","Data":"b1e96305c53881b12fb71c0333f809108eb7052cf737246df7b5399d2181da57"}
Sep 11 18:08:07.747163 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:07.747103    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl" podStartSLOduration=1.747085449 podStartE2EDuration="1.747085449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:08:07.745636606 +0000 UTC m=+845.588872048" watchObservedRunningTime="2026-09-11 18:08:07.747085449 +0000 UTC m=+845.590320889"
Sep 11 18:08:16.923005 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:16.922974    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"]
Sep 11 18:08:16.923410 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:16.923162    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6" podUID="6632c831-fb4e-4fd2-8206-76adbee7a55e" containerName="maas-api" containerID="cri-o://98dde4c791d545d538d889e0f78f415e9785e65398bb67e2bbce11935ab90b32" gracePeriod=30
Sep 11 18:08:17.014778 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.014712    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:17.014778 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.014784    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:17.029038 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.028996    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:17.175099 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.175034    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"
Sep 11 18:08:17.307076 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.307039    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6632c831-fb4e-4fd2-8206-76adbee7a55e-openshift-service-ca\") pod \"6632c831-fb4e-4fd2-8206-76adbee7a55e\" (UID: \"6632c831-fb4e-4fd2-8206-76adbee7a55e\") "
Sep 11 18:08:17.307283 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.307144    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6632c831-fb4e-4fd2-8206-76adbee7a55e-maas-api-tls\") pod \"6632c831-fb4e-4fd2-8206-76adbee7a55e\" (UID: \"6632c831-fb4e-4fd2-8206-76adbee7a55e\") "
Sep 11 18:08:17.307283 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.307175    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6632c831-fb4e-4fd2-8206-76adbee7a55e-metrics-certs\") pod \"6632c831-fb4e-4fd2-8206-76adbee7a55e\" (UID: \"6632c831-fb4e-4fd2-8206-76adbee7a55e\") "
Sep 11 18:08:17.307283 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.307236    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7wh4\" (UniqueName: \"kubernetes.io/projected/6632c831-fb4e-4fd2-8206-76adbee7a55e-kube-api-access-l7wh4\") pod \"6632c831-fb4e-4fd2-8206-76adbee7a55e\" (UID: \"6632c831-fb4e-4fd2-8206-76adbee7a55e\") "
Sep 11 18:08:17.307443 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.307413    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6632c831-fb4e-4fd2-8206-76adbee7a55e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6632c831-fb4e-4fd2-8206-76adbee7a55e" (UID: "6632c831-fb4e-4fd2-8206-76adbee7a55e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:08:17.307602 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.307584    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6632c831-fb4e-4fd2-8206-76adbee7a55e-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:08:17.309602 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.309575    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6632c831-fb4e-4fd2-8206-76adbee7a55e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6632c831-fb4e-4fd2-8206-76adbee7a55e" (UID: "6632c831-fb4e-4fd2-8206-76adbee7a55e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:08:17.309689 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.309630    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6632c831-fb4e-4fd2-8206-76adbee7a55e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6632c831-fb4e-4fd2-8206-76adbee7a55e" (UID: "6632c831-fb4e-4fd2-8206-76adbee7a55e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:08:17.309689 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.309656    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6632c831-fb4e-4fd2-8206-76adbee7a55e-kube-api-access-l7wh4" (OuterVolumeSpecName: "kube-api-access-l7wh4") pod "6632c831-fb4e-4fd2-8206-76adbee7a55e" (UID: "6632c831-fb4e-4fd2-8206-76adbee7a55e"). InnerVolumeSpecName "kube-api-access-l7wh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:08:17.408614 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.408575    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7wh4\" (UniqueName: \"kubernetes.io/projected/6632c831-fb4e-4fd2-8206-76adbee7a55e-kube-api-access-l7wh4\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:08:17.408614 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.408607    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6632c831-fb4e-4fd2-8206-76adbee7a55e-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:08:17.408614 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.408617    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6632c831-fb4e-4fd2-8206-76adbee7a55e-metrics-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:08:17.779519 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.779481    2560 generic.go:356] "Generic (PLEG): container finished" podID="6632c831-fb4e-4fd2-8206-76adbee7a55e" containerID="98dde4c791d545d538d889e0f78f415e9785e65398bb67e2bbce11935ab90b32" exitCode=0
Sep 11 18:08:17.779695 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.779523    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"
Sep 11 18:08:17.779695 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.779572    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6" event={"ID":"6632c831-fb4e-4fd2-8206-76adbee7a55e","Type":"ContainerDied","Data":"98dde4c791d545d538d889e0f78f415e9785e65398bb67e2bbce11935ab90b32"}
Sep 11 18:08:17.779695 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.779622    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6" event={"ID":"6632c831-fb4e-4fd2-8206-76adbee7a55e","Type":"ContainerDied","Data":"6daa9a39f35cd5cce1101cbfb3bbeaee250983cee2680affa9f2293065bd610e"}
Sep 11 18:08:17.779695 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.779650    2560 scope.go:117] "RemoveContainer" containerID="98dde4c791d545d538d889e0f78f415e9785e65398bb67e2bbce11935ab90b32"
Sep 11 18:08:17.789677 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.789657    2560 scope.go:117] "RemoveContainer" containerID="98dde4c791d545d538d889e0f78f415e9785e65398bb67e2bbce11935ab90b32"
Sep 11 18:08:17.790000 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:08:17.789984    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98dde4c791d545d538d889e0f78f415e9785e65398bb67e2bbce11935ab90b32\": container with ID starting with 98dde4c791d545d538d889e0f78f415e9785e65398bb67e2bbce11935ab90b32 not found: ID does not exist" containerID="98dde4c791d545d538d889e0f78f415e9785e65398bb67e2bbce11935ab90b32"
Sep 11 18:08:17.790081 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.790010    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98dde4c791d545d538d889e0f78f415e9785e65398bb67e2bbce11935ab90b32"} err="failed to get container status \"98dde4c791d545d538d889e0f78f415e9785e65398bb67e2bbce11935ab90b32\": rpc error: code = NotFound desc = could not find container \"98dde4c791d545d538d889e0f78f415e9785e65398bb67e2bbce11935ab90b32\": container with ID starting with 98dde4c791d545d538d889e0f78f415e9785e65398bb67e2bbce11935ab90b32 not found: ID does not exist"
Sep 11 18:08:17.792234 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.792213    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:08:17.813629 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.813601    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"]
Sep 11 18:08:17.823002 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:17.822974    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33958e97-5989c8686c-m54h6"]
Sep 11 18:08:18.687983 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:18.687949    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6632c831-fb4e-4fd2-8206-76adbee7a55e" path="/var/lib/kubelet/pods/6632c831-fb4e-4fd2-8206-76adbee7a55e/volumes"
Sep 11 18:08:24.534890 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:24.534854    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:08:24.542431 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:24.542402    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"]
Sep 11 18:08:24.542889 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:24.542871    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6632c831-fb4e-4fd2-8206-76adbee7a55e" containerName="maas-api"
Sep 11 18:08:24.542989 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:24.542891    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6632c831-fb4e-4fd2-8206-76adbee7a55e" containerName="maas-api"
Sep 11 18:08:24.543046 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:24.543034    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6632c831-fb4e-4fd2-8206-76adbee7a55e" containerName="maas-api"
Sep 11 18:08:24.547468 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:24.547446    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"
Sep 11 18:08:24.551268 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:24.551242    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-44ba3006-serving-cert\""
Sep 11 18:08:24.619733 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:24.619695    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"]
Sep 11 18:08:24.683695 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:24.683662    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6pk8\" (UniqueName: \"kubernetes.io/projected/33314811-b807-46db-8588-782fae07d7e9-kube-api-access-r6pk8\") pod \"maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4\" (UID: \"33314811-b807-46db-8588-782fae07d7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"
Sep 11 18:08:24.683922 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:24.683742    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33314811-b807-46db-8588-782fae07d7e9-openshift-service-ca\") pod \"maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4\" (UID: \"33314811-b807-46db-8588-782fae07d7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"
Sep 11 18:08:24.683922 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:24.683835    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33314811-b807-46db-8588-782fae07d7e9-metrics-certs\") pod \"maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4\" (UID: \"33314811-b807-46db-8588-782fae07d7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"
Sep 11 18:08:24.683922 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:24.683893    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33314811-b807-46db-8588-782fae07d7e9-maas-api-tls\") pod \"maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4\" (UID: \"33314811-b807-46db-8588-782fae07d7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"
Sep 11 18:08:24.785324 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:24.785233    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33314811-b807-46db-8588-782fae07d7e9-maas-api-tls\") pod \"maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4\" (UID: \"33314811-b807-46db-8588-782fae07d7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"
Sep 11 18:08:24.785324 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:24.785285    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6pk8\" (UniqueName: \"kubernetes.io/projected/33314811-b807-46db-8588-782fae07d7e9-kube-api-access-r6pk8\") pod \"maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4\" (UID: \"33314811-b807-46db-8588-782fae07d7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"
Sep 11 18:08:24.785530 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:24.785358    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33314811-b807-46db-8588-782fae07d7e9-openshift-service-ca\") pod \"maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4\" (UID: \"33314811-b807-46db-8588-782fae07d7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"
Sep 11 18:08:24.785530 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:08:24.785382    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-serving-cert: secret "maas-api-e2e-migrate-44ba3006-serving-cert" not found
Sep 11 18:08:24.785530 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:08:24.785461    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33314811-b807-46db-8588-782fae07d7e9-maas-api-tls podName:33314811-b807-46db-8588-782fae07d7e9 nodeName:}" failed. No retries permitted until 2026-09-11 18:08:25.285438202 +0000 UTC m=+863.128673642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/33314811-b807-46db-8588-782fae07d7e9-maas-api-tls") pod "maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4" (UID: "33314811-b807-46db-8588-782fae07d7e9") : secret "maas-api-e2e-migrate-44ba3006-serving-cert" not found
Sep 11 18:08:24.785711 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:24.785548    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33314811-b807-46db-8588-782fae07d7e9-metrics-certs\") pod \"maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4\" (UID: \"33314811-b807-46db-8588-782fae07d7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"
Sep 11 18:08:24.786012 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:24.785995    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33314811-b807-46db-8588-782fae07d7e9-openshift-service-ca\") pod \"maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4\" (UID: \"33314811-b807-46db-8588-782fae07d7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"
Sep 11 18:08:24.788086 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:24.788069    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33314811-b807-46db-8588-782fae07d7e9-metrics-certs\") pod \"maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4\" (UID: \"33314811-b807-46db-8588-782fae07d7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"
Sep 11 18:08:24.799243 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:24.799211    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6pk8\" (UniqueName: \"kubernetes.io/projected/33314811-b807-46db-8588-782fae07d7e9-kube-api-access-r6pk8\") pod \"maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4\" (UID: \"33314811-b807-46db-8588-782fae07d7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"
Sep 11 18:08:25.291049 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:25.291009    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33314811-b807-46db-8588-782fae07d7e9-maas-api-tls\") pod \"maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4\" (UID: \"33314811-b807-46db-8588-782fae07d7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"
Sep 11 18:08:25.293673 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:25.293647    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33314811-b807-46db-8588-782fae07d7e9-maas-api-tls\") pod \"maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4\" (UID: \"33314811-b807-46db-8588-782fae07d7e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"
Sep 11 18:08:25.458672 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:25.458635    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"
Sep 11 18:08:25.739894 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:25.739740    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"]
Sep 11 18:08:25.742774 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:08:25.742729    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33314811_b807_46db_8588_782fae07d7e9.slice/crio-aa15ebdcb93247414de3a4d21ce51cc3342c57890feff3a9243afae359bba5e4 WatchSource:0}: Error finding container aa15ebdcb93247414de3a4d21ce51cc3342c57890feff3a9243afae359bba5e4: Status 404 returned error can't find the container with id aa15ebdcb93247414de3a4d21ce51cc3342c57890feff3a9243afae359bba5e4
Sep 11 18:08:25.821266 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:25.821233    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4" event={"ID":"33314811-b807-46db-8588-782fae07d7e9","Type":"ContainerStarted","Data":"aa15ebdcb93247414de3a4d21ce51cc3342c57890feff3a9243afae359bba5e4"}
Sep 11 18:08:26.827761 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:26.827721    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4" event={"ID":"33314811-b807-46db-8588-782fae07d7e9","Type":"ContainerStarted","Data":"fc1c7ff60f9880c772358cfeeb58bcf9ea2ba5c400affd6481bd971fdb9d6b25"}
Sep 11 18:08:26.828240 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:26.827971    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"
Sep 11 18:08:26.850310 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:26.850216    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4" podStartSLOduration=2.491942098 podStartE2EDuration="2.850201959s" totalImagesPullingTime="358.259861ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:08:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:08:26.850052302 +0000 UTC m=+864.693287742" watchObservedRunningTime="2026-09-11 18:08:26.850201959 +0000 UTC m=+864.693437438"
Sep 11 18:08:32.838172 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:32.838145    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"
Sep 11 18:08:32.912883 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:32.912852    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"]
Sep 11 18:08:32.916406 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:32.916389    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:32.919877 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:32.919846    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4e0b0b\"/\"openshift-service-ca.crt\""
Sep 11 18:08:32.920023 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:32.919905    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4e0b0b\"/\"simulator-w4-4e0b0b-kserve-self-signed-certs\""
Sep 11 18:08:32.920023 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:32.919934    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4e0b0b\"/\"kube-root-ca.crt\""
Sep 11 18:08:32.920023 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:32.919910    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4e0b0b\"/\"default-dockercfg-fzqfp\""
Sep 11 18:08:32.931480 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:32.931455    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"]
Sep 11 18:08:33.066835 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.066784    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-dshm\") pod \"simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") " pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:33.067038 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.066849    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-tmp-dir\") pod \"simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") " pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:33.067038 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.066900    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/221f6e7a-44be-4e66-82db-4b852b2ef7e2-tls-certs\") pod \"simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") " pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:33.067038 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.066965    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdmdt\" (UniqueName: \"kubernetes.io/projected/221f6e7a-44be-4e66-82db-4b852b2ef7e2-kube-api-access-hdmdt\") pod \"simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") " pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:33.067038 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.066983    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-home\") pod \"simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") " pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:33.067181 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.067045    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-model-cache\") pod \"simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") " pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:33.168354 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.168263    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-dshm\") pod \"simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") " pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:33.168354 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.168307    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-tmp-dir\") pod \"simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") " pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:33.168354 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.168348    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/221f6e7a-44be-4e66-82db-4b852b2ef7e2-tls-certs\") pod \"simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") " pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:33.168642 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.168390    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdmdt\" (UniqueName: \"kubernetes.io/projected/221f6e7a-44be-4e66-82db-4b852b2ef7e2-kube-api-access-hdmdt\") pod \"simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") " pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:33.168642 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.168409    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-home\") pod \"simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") " pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:33.168642 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.168436    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-model-cache\") pod \"simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") " pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:33.168828 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.168803    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-tmp-dir\") pod \"simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") " pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:33.168888 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.168866    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-model-cache\") pod \"simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") " pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:33.168925 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.168883    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-home\") pod \"simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") " pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:33.170787 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.170745    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-dshm\") pod \"simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") " pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:33.171040 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.171025    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/221f6e7a-44be-4e66-82db-4b852b2ef7e2-tls-certs\") pod \"simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") " pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:33.179510 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.179485    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdmdt\" (UniqueName: \"kubernetes.io/projected/221f6e7a-44be-4e66-82db-4b852b2ef7e2-kube-api-access-hdmdt\") pod \"simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") " pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:33.226023 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.225981    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:33.502635 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.502609    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"]
Sep 11 18:08:33.504611 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:08:33.504584    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod221f6e7a_44be_4e66_82db_4b852b2ef7e2.slice/crio-a8d60aad90cc30f7dc860202522d47a4675599cc6297a814b63b65db7d5e9bd3 WatchSource:0}: Error finding container a8d60aad90cc30f7dc860202522d47a4675599cc6297a814b63b65db7d5e9bd3: Status 404 returned error can't find the container with id a8d60aad90cc30f7dc860202522d47a4675599cc6297a814b63b65db7d5e9bd3
Sep 11 18:08:33.866484 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.866394    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5" event={"ID":"221f6e7a-44be-4e66-82db-4b852b2ef7e2","Type":"ContainerStarted","Data":"8400d7c80dccdc0c767bf7242d7c4184b27a02f9a363d388dd93709f9327c7e6"}
Sep 11 18:08:33.866484 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.866438    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5" event={"ID":"221f6e7a-44be-4e66-82db-4b852b2ef7e2","Type":"ContainerStarted","Data":"a8d60aad90cc30f7dc860202522d47a4675599cc6297a814b63b65db7d5e9bd3"}
Sep 11 18:08:33.889150 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:33.889091    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5" podStartSLOduration=1.889072782 podStartE2EDuration="1.889072782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:08:33.888198522 +0000 UTC m=+871.731433992" watchObservedRunningTime="2026-09-11 18:08:33.889072782 +0000 UTC m=+871.732308221"
Sep 11 18:08:43.226584 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:43.226541    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:43.227119 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:43.226604    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:43.238856 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:43.238830    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:43.340647 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:43.340601    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:08:43.930094 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:43.930068    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:08:56.632546 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.632511    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"]
Sep 11 18:08:56.637208 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.637190    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:08:56.642259 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.642237    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-736ba3\"/\"autores-model-736ba3-kserve-self-signed-certs\""
Sep 11 18:08:56.643378 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.643355    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-736ba3\"/\"openshift-service-ca.crt\""
Sep 11 18:08:56.643511 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.643380    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-736ba3\"/\"kube-root-ca.crt\""
Sep 11 18:08:56.643511 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.643486    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-736ba3\"/\"default-dockercfg-x5dkk\""
Sep 11 18:08:56.650237 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.650217    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"]
Sep 11 18:08:56.702263 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.702231    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-dshm\") pod \"autores-model-736ba3-kserve-85cd768599-sj8ss\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") " pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:08:56.702449 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.702274    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-home\") pod \"autores-model-736ba3-kserve-85cd768599-sj8ss\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") " pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:08:56.702449 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.702420    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvjp7\" (UniqueName: \"kubernetes.io/projected/4f54dd6c-8a4a-432b-8d18-e78430943353-kube-api-access-nvjp7\") pod \"autores-model-736ba3-kserve-85cd768599-sj8ss\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") " pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:08:56.702535 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.702459    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-tmp-dir\") pod \"autores-model-736ba3-kserve-85cd768599-sj8ss\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") " pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:08:56.702580 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.702545    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-model-cache\") pod \"autores-model-736ba3-kserve-85cd768599-sj8ss\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") " pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:08:56.702634 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.702590    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f54dd6c-8a4a-432b-8d18-e78430943353-tls-certs\") pod \"autores-model-736ba3-kserve-85cd768599-sj8ss\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") " pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:08:56.804144 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.804045    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvjp7\" (UniqueName: \"kubernetes.io/projected/4f54dd6c-8a4a-432b-8d18-e78430943353-kube-api-access-nvjp7\") pod \"autores-model-736ba3-kserve-85cd768599-sj8ss\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") " pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:08:56.804144 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.804132    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-tmp-dir\") pod \"autores-model-736ba3-kserve-85cd768599-sj8ss\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") " pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:08:56.804383 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.804187    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-model-cache\") pod \"autores-model-736ba3-kserve-85cd768599-sj8ss\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") " pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:08:56.804383 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.804220    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f54dd6c-8a4a-432b-8d18-e78430943353-tls-certs\") pod \"autores-model-736ba3-kserve-85cd768599-sj8ss\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") " pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:08:56.804383 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.804270    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-dshm\") pod \"autores-model-736ba3-kserve-85cd768599-sj8ss\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") " pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:08:56.804545 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.804313    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-home\") pod \"autores-model-736ba3-kserve-85cd768599-sj8ss\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") " pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:08:56.804637 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.804614    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-tmp-dir\") pod \"autores-model-736ba3-kserve-85cd768599-sj8ss\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") " pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:08:56.804701 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.804640    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-model-cache\") pod \"autores-model-736ba3-kserve-85cd768599-sj8ss\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") " pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:08:56.804801 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.804782    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-home\") pod \"autores-model-736ba3-kserve-85cd768599-sj8ss\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") " pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:08:56.806844 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.806827    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-dshm\") pod \"autores-model-736ba3-kserve-85cd768599-sj8ss\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") " pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:08:56.807034 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.807017    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f54dd6c-8a4a-432b-8d18-e78430943353-tls-certs\") pod \"autores-model-736ba3-kserve-85cd768599-sj8ss\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") " pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:08:56.825871 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.825843    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvjp7\" (UniqueName: \"kubernetes.io/projected/4f54dd6c-8a4a-432b-8d18-e78430943353-kube-api-access-nvjp7\") pod \"autores-model-736ba3-kserve-85cd768599-sj8ss\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") " pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:08:56.946955 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:56.946916    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:08:57.225451 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:57.225428    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"]
Sep 11 18:08:57.227024 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:08:57.227003    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f54dd6c_8a4a_432b_8d18_e78430943353.slice/crio-f25165270ceb01d97e1a412cd0d7092def53e60d1a26788a7c8e0f580d3b48c8 WatchSource:0}: Error finding container f25165270ceb01d97e1a412cd0d7092def53e60d1a26788a7c8e0f580d3b48c8: Status 404 returned error can't find the container with id f25165270ceb01d97e1a412cd0d7092def53e60d1a26788a7c8e0f580d3b48c8
Sep 11 18:08:57.991203 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:57.991168    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss" event={"ID":"4f54dd6c-8a4a-432b-8d18-e78430943353","Type":"ContainerStarted","Data":"46b595b3257faef475894fb85de171d7e592fcdbed852c54b58aa982c577e48e"}
Sep 11 18:08:57.991203 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:57.991211    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss" event={"ID":"4f54dd6c-8a4a-432b-8d18-e78430943353","Type":"ContainerStarted","Data":"f25165270ceb01d97e1a412cd0d7092def53e60d1a26788a7c8e0f580d3b48c8"}
Sep 11 18:08:58.009281 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:08:58.009226    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss" podStartSLOduration=2.009210241 podStartE2EDuration="2.009210241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:08:58.008843118 +0000 UTC m=+895.852078561" watchObservedRunningTime="2026-09-11 18:08:58.009210241 +0000 UTC m=+895.852445682"
Sep 11 18:09:02.640949 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:02.640913    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/2.log"
Sep 11 18:09:02.659834 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:02.659810    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/2.log"
Sep 11 18:09:03.275284 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:03.275251    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:09:04.394527 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:04.394493    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"]
Sep 11 18:09:04.395167 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:04.394719    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4" podUID="33314811-b807-46db-8588-782fae07d7e9" containerName="maas-api" containerID="cri-o://fc1c7ff60f9880c772358cfeeb58bcf9ea2ba5c400affd6481bd971fdb9d6b25" gracePeriod=30
Sep 11 18:09:04.644083 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:04.644058    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"
Sep 11 18:09:04.785198 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:04.785157    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33314811-b807-46db-8588-782fae07d7e9-openshift-service-ca\") pod \"33314811-b807-46db-8588-782fae07d7e9\" (UID: \"33314811-b807-46db-8588-782fae07d7e9\") "
Sep 11 18:09:04.785407 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:04.785270    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33314811-b807-46db-8588-782fae07d7e9-maas-api-tls\") pod \"33314811-b807-46db-8588-782fae07d7e9\" (UID: \"33314811-b807-46db-8588-782fae07d7e9\") "
Sep 11 18:09:04.785407 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:04.785297    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6pk8\" (UniqueName: \"kubernetes.io/projected/33314811-b807-46db-8588-782fae07d7e9-kube-api-access-r6pk8\") pod \"33314811-b807-46db-8588-782fae07d7e9\" (UID: \"33314811-b807-46db-8588-782fae07d7e9\") "
Sep 11 18:09:04.785407 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:04.785340    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33314811-b807-46db-8588-782fae07d7e9-metrics-certs\") pod \"33314811-b807-46db-8588-782fae07d7e9\" (UID: \"33314811-b807-46db-8588-782fae07d7e9\") "
Sep 11 18:09:04.785607 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:04.785579    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33314811-b807-46db-8588-782fae07d7e9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "33314811-b807-46db-8588-782fae07d7e9" (UID: "33314811-b807-46db-8588-782fae07d7e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:09:04.785777 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:04.785721    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33314811-b807-46db-8588-782fae07d7e9-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:09:04.787677 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:04.787653    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33314811-b807-46db-8588-782fae07d7e9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "33314811-b807-46db-8588-782fae07d7e9" (UID: "33314811-b807-46db-8588-782fae07d7e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:09:04.787677 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:04.787653    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33314811-b807-46db-8588-782fae07d7e9-kube-api-access-r6pk8" (OuterVolumeSpecName: "kube-api-access-r6pk8") pod "33314811-b807-46db-8588-782fae07d7e9" (UID: "33314811-b807-46db-8588-782fae07d7e9"). InnerVolumeSpecName "kube-api-access-r6pk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:09:04.787855 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:04.787690    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33314811-b807-46db-8588-782fae07d7e9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "33314811-b807-46db-8588-782fae07d7e9" (UID: "33314811-b807-46db-8588-782fae07d7e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:09:04.886964 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:04.886929    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33314811-b807-46db-8588-782fae07d7e9-metrics-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:09:04.886964 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:04.886956    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33314811-b807-46db-8588-782fae07d7e9-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:09:04.886964 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:04.886966    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6pk8\" (UniqueName: \"kubernetes.io/projected/33314811-b807-46db-8588-782fae07d7e9-kube-api-access-r6pk8\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:09:05.028939 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:05.028905    2560 generic.go:356] "Generic (PLEG): container finished" podID="33314811-b807-46db-8588-782fae07d7e9" containerID="fc1c7ff60f9880c772358cfeeb58bcf9ea2ba5c400affd6481bd971fdb9d6b25" exitCode=0
Sep 11 18:09:05.029114 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:05.028949    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"
Sep 11 18:09:05.029114 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:05.029007    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4" event={"ID":"33314811-b807-46db-8588-782fae07d7e9","Type":"ContainerDied","Data":"fc1c7ff60f9880c772358cfeeb58bcf9ea2ba5c400affd6481bd971fdb9d6b25"}
Sep 11 18:09:05.029114 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:05.029067    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4" event={"ID":"33314811-b807-46db-8588-782fae07d7e9","Type":"ContainerDied","Data":"aa15ebdcb93247414de3a4d21ce51cc3342c57890feff3a9243afae359bba5e4"}
Sep 11 18:09:05.029114 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:05.029098    2560 scope.go:117] "RemoveContainer" containerID="fc1c7ff60f9880c772358cfeeb58bcf9ea2ba5c400affd6481bd971fdb9d6b25"
Sep 11 18:09:05.039015 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:05.038995    2560 scope.go:117] "RemoveContainer" containerID="fc1c7ff60f9880c772358cfeeb58bcf9ea2ba5c400affd6481bd971fdb9d6b25"
Sep 11 18:09:05.039301 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:09:05.039285    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc1c7ff60f9880c772358cfeeb58bcf9ea2ba5c400affd6481bd971fdb9d6b25\": container with ID starting with fc1c7ff60f9880c772358cfeeb58bcf9ea2ba5c400affd6481bd971fdb9d6b25 not found: ID does not exist" containerID="fc1c7ff60f9880c772358cfeeb58bcf9ea2ba5c400affd6481bd971fdb9d6b25"
Sep 11 18:09:05.039349 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:05.039311    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc1c7ff60f9880c772358cfeeb58bcf9ea2ba5c400affd6481bd971fdb9d6b25"} err="failed to get container status \"fc1c7ff60f9880c772358cfeeb58bcf9ea2ba5c400affd6481bd971fdb9d6b25\": rpc error: code = NotFound desc = could not find container \"fc1c7ff60f9880c772358cfeeb58bcf9ea2ba5c400affd6481bd971fdb9d6b25\": container with ID starting with fc1c7ff60f9880c772358cfeeb58bcf9ea2ba5c400affd6481bd971fdb9d6b25 not found: ID does not exist"
Sep 11 18:09:05.068688 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:05.068660    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"]
Sep 11 18:09:05.075736 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:05.075709    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-44ba3006-54847d4dcf-ljgv4"]
Sep 11 18:09:06.693660 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:06.693625    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33314811-b807-46db-8588-782fae07d7e9" path="/var/lib/kubelet/pods/33314811-b807-46db-8588-782fae07d7e9/volumes"
Sep 11 18:09:06.947936 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:06.947854    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:09:06.947936 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:06.947896    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:09:06.960559 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:06.960524    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:09:07.052943 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:07.052911    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:09:12.838726 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:12.838700    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"]
Sep 11 18:09:12.839126 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:12.839113    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33314811-b807-46db-8588-782fae07d7e9" containerName="maas-api"
Sep 11 18:09:12.839170 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:12.839128    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="33314811-b807-46db-8588-782fae07d7e9" containerName="maas-api"
Sep 11 18:09:12.839206 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:12.839197    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="33314811-b807-46db-8588-782fae07d7e9" containerName="maas-api"
Sep 11 18:09:12.850280 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:12.850249    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"
Sep 11 18:09:12.854360 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:12.854334    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-820e15b2-serving-cert\""
Sep 11 18:09:12.862689 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:12.862660    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"]
Sep 11 18:09:12.864570 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:12.864532    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/278c8dac-4a47-4c32-814f-9f252ee64b0c-openshift-service-ca\") pod \"maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf\" (UID: \"278c8dac-4a47-4c32-814f-9f252ee64b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"
Sep 11 18:09:12.864698 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:12.864570    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278c8dac-4a47-4c32-814f-9f252ee64b0c-metrics-certs\") pod \"maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf\" (UID: \"278c8dac-4a47-4c32-814f-9f252ee64b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"
Sep 11 18:09:12.864698 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:12.864669    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j9fq\" (UniqueName: \"kubernetes.io/projected/278c8dac-4a47-4c32-814f-9f252ee64b0c-kube-api-access-7j9fq\") pod \"maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf\" (UID: \"278c8dac-4a47-4c32-814f-9f252ee64b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"
Sep 11 18:09:12.864896 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:12.864706    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/278c8dac-4a47-4c32-814f-9f252ee64b0c-maas-api-tls\") pod \"maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf\" (UID: \"278c8dac-4a47-4c32-814f-9f252ee64b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"
Sep 11 18:09:12.965324 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:12.965286    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/278c8dac-4a47-4c32-814f-9f252ee64b0c-maas-api-tls\") pod \"maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf\" (UID: \"278c8dac-4a47-4c32-814f-9f252ee64b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"
Sep 11 18:09:12.965533 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:12.965362    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/278c8dac-4a47-4c32-814f-9f252ee64b0c-openshift-service-ca\") pod \"maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf\" (UID: \"278c8dac-4a47-4c32-814f-9f252ee64b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"
Sep 11 18:09:12.965533 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:12.965384    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278c8dac-4a47-4c32-814f-9f252ee64b0c-metrics-certs\") pod \"maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf\" (UID: \"278c8dac-4a47-4c32-814f-9f252ee64b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"
Sep 11 18:09:12.965533 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:09:12.965432    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-serving-cert: secret "maas-api-e2e-ait-820e15b2-serving-cert" not found
Sep 11 18:09:12.965533 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:12.965458    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j9fq\" (UniqueName: \"kubernetes.io/projected/278c8dac-4a47-4c32-814f-9f252ee64b0c-kube-api-access-7j9fq\") pod \"maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf\" (UID: \"278c8dac-4a47-4c32-814f-9f252ee64b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"
Sep 11 18:09:12.965533 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:09:12.965500    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/278c8dac-4a47-4c32-814f-9f252ee64b0c-maas-api-tls podName:278c8dac-4a47-4c32-814f-9f252ee64b0c nodeName:}" failed. No retries permitted until 2026-09-11 18:09:13.465480308 +0000 UTC m=+911.308715726 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/278c8dac-4a47-4c32-814f-9f252ee64b0c-maas-api-tls") pod "maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf" (UID: "278c8dac-4a47-4c32-814f-9f252ee64b0c") : secret "maas-api-e2e-ait-820e15b2-serving-cert" not found
Sep 11 18:09:12.966170 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:12.966147    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/278c8dac-4a47-4c32-814f-9f252ee64b0c-openshift-service-ca\") pod \"maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf\" (UID: \"278c8dac-4a47-4c32-814f-9f252ee64b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"
Sep 11 18:09:12.968011 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:12.967993    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278c8dac-4a47-4c32-814f-9f252ee64b0c-metrics-certs\") pod \"maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf\" (UID: \"278c8dac-4a47-4c32-814f-9f252ee64b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"
Sep 11 18:09:12.978306 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:12.978285    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j9fq\" (UniqueName: \"kubernetes.io/projected/278c8dac-4a47-4c32-814f-9f252ee64b0c-kube-api-access-7j9fq\") pod \"maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf\" (UID: \"278c8dac-4a47-4c32-814f-9f252ee64b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"
Sep 11 18:09:13.470876 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:13.470832    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/278c8dac-4a47-4c32-814f-9f252ee64b0c-maas-api-tls\") pod \"maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf\" (UID: \"278c8dac-4a47-4c32-814f-9f252ee64b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"
Sep 11 18:09:13.473367 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:13.473345    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/278c8dac-4a47-4c32-814f-9f252ee64b0c-maas-api-tls\") pod \"maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf\" (UID: \"278c8dac-4a47-4c32-814f-9f252ee64b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"
Sep 11 18:09:13.764594 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:13.764500    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"
Sep 11 18:09:14.045860 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:14.045834    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"]
Sep 11 18:09:14.047861 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:09:14.047834    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278c8dac_4a47_4c32_814f_9f252ee64b0c.slice/crio-bb36ebffc6dcc07212e37a50d375a6672dfe12f6db5baf937c80ba81f2009b5b WatchSource:0}: Error finding container bb36ebffc6dcc07212e37a50d375a6672dfe12f6db5baf937c80ba81f2009b5b: Status 404 returned error can't find the container with id bb36ebffc6dcc07212e37a50d375a6672dfe12f6db5baf937c80ba81f2009b5b
Sep 11 18:09:14.079113 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:14.079077    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf" event={"ID":"278c8dac-4a47-4c32-814f-9f252ee64b0c","Type":"ContainerStarted","Data":"bb36ebffc6dcc07212e37a50d375a6672dfe12f6db5baf937c80ba81f2009b5b"}
Sep 11 18:09:15.087134 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:15.087088    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf" event={"ID":"278c8dac-4a47-4c32-814f-9f252ee64b0c","Type":"ContainerStarted","Data":"a8255cb1fe1d0115aeeb86d3d269adceefef922a4e1db55bb2ebc9034b90e965"}
Sep 11 18:09:15.087593 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:15.087206    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"
Sep 11 18:09:15.138922 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:15.138874    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf" podStartSLOduration=2.7396377210000002 podStartE2EDuration="3.138855521s" totalImagesPullingTime="399.2178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:09:15.137705799 +0000 UTC m=+912.980941241" watchObservedRunningTime="2026-09-11 18:09:15.138855521 +0000 UTC m=+912.982090961"
Sep 11 18:09:21.098852 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:21.098823    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"
Sep 11 18:09:24.319263 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:24.319229    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:09:41.422078 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:41.421992    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"]
Sep 11 18:09:41.422574 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:41.422251    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr" podUID="c0c98b02-b638-4f04-83ec-7c4589a93efd" containerName="maas-api" containerID="cri-o://6089d7d50977ac420281b873f31f15dc2baa9ab1fa8877b8be4db234facbcdb4" gracePeriod=30
Sep 11 18:09:41.678022 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:41.677992    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"]
Sep 11 18:09:41.678215 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:41.678193    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf" podUID="278c8dac-4a47-4c32-814f-9f252ee64b0c" containerName="maas-api" containerID="cri-o://a8255cb1fe1d0115aeeb86d3d269adceefef922a4e1db55bb2ebc9034b90e965" gracePeriod=30
Sep 11 18:09:41.745469 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:41.745438    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"
Sep 11 18:09:41.861356 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:41.861317    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c98b02-b638-4f04-83ec-7c4589a93efd-openshift-service-ca\") pod \"c0c98b02-b638-4f04-83ec-7c4589a93efd\" (UID: \"c0c98b02-b638-4f04-83ec-7c4589a93efd\") "
Sep 11 18:09:41.861523 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:41.861399    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9lt8\" (UniqueName: \"kubernetes.io/projected/c0c98b02-b638-4f04-83ec-7c4589a93efd-kube-api-access-f9lt8\") pod \"c0c98b02-b638-4f04-83ec-7c4589a93efd\" (UID: \"c0c98b02-b638-4f04-83ec-7c4589a93efd\") "
Sep 11 18:09:41.861523 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:41.861430    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c98b02-b638-4f04-83ec-7c4589a93efd-metrics-certs\") pod \"c0c98b02-b638-4f04-83ec-7c4589a93efd\" (UID: \"c0c98b02-b638-4f04-83ec-7c4589a93efd\") "
Sep 11 18:09:41.861523 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:41.861469    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0c98b02-b638-4f04-83ec-7c4589a93efd-maas-api-tls\") pod \"c0c98b02-b638-4f04-83ec-7c4589a93efd\" (UID: \"c0c98b02-b638-4f04-83ec-7c4589a93efd\") "
Sep 11 18:09:41.861787 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:41.861743    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c98b02-b638-4f04-83ec-7c4589a93efd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c0c98b02-b638-4f04-83ec-7c4589a93efd" (UID: "c0c98b02-b638-4f04-83ec-7c4589a93efd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:09:41.861883 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:41.861865    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0c98b02-b638-4f04-83ec-7c4589a93efd-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:09:41.863918 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:41.863893    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c98b02-b638-4f04-83ec-7c4589a93efd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c0c98b02-b638-4f04-83ec-7c4589a93efd" (UID: "c0c98b02-b638-4f04-83ec-7c4589a93efd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:09:41.864035 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:41.863944    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c98b02-b638-4f04-83ec-7c4589a93efd-kube-api-access-f9lt8" (OuterVolumeSpecName: "kube-api-access-f9lt8") pod "c0c98b02-b638-4f04-83ec-7c4589a93efd" (UID: "c0c98b02-b638-4f04-83ec-7c4589a93efd"). InnerVolumeSpecName "kube-api-access-f9lt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:09:41.865237 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:41.865211    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c98b02-b638-4f04-83ec-7c4589a93efd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c0c98b02-b638-4f04-83ec-7c4589a93efd" (UID: "c0c98b02-b638-4f04-83ec-7c4589a93efd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:09:41.917913 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:41.917888    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"
Sep 11 18:09:41.963277 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:41.963253    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9lt8\" (UniqueName: \"kubernetes.io/projected/c0c98b02-b638-4f04-83ec-7c4589a93efd-kube-api-access-f9lt8\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:09:41.963419 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:41.963279    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0c98b02-b638-4f04-83ec-7c4589a93efd-metrics-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:09:41.963419 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:41.963288    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0c98b02-b638-4f04-83ec-7c4589a93efd-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:09:42.064284 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.064243    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/278c8dac-4a47-4c32-814f-9f252ee64b0c-openshift-service-ca\") pod \"278c8dac-4a47-4c32-814f-9f252ee64b0c\" (UID: \"278c8dac-4a47-4c32-814f-9f252ee64b0c\") "
Sep 11 18:09:42.064461 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.064337    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/278c8dac-4a47-4c32-814f-9f252ee64b0c-maas-api-tls\") pod \"278c8dac-4a47-4c32-814f-9f252ee64b0c\" (UID: \"278c8dac-4a47-4c32-814f-9f252ee64b0c\") "
Sep 11 18:09:42.064461 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.064379    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7j9fq\" (UniqueName: \"kubernetes.io/projected/278c8dac-4a47-4c32-814f-9f252ee64b0c-kube-api-access-7j9fq\") pod \"278c8dac-4a47-4c32-814f-9f252ee64b0c\" (UID: \"278c8dac-4a47-4c32-814f-9f252ee64b0c\") "
Sep 11 18:09:42.064461 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.064437    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278c8dac-4a47-4c32-814f-9f252ee64b0c-metrics-certs\") pod \"278c8dac-4a47-4c32-814f-9f252ee64b0c\" (UID: \"278c8dac-4a47-4c32-814f-9f252ee64b0c\") "
Sep 11 18:09:42.064672 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.064643    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/278c8dac-4a47-4c32-814f-9f252ee64b0c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "278c8dac-4a47-4c32-814f-9f252ee64b0c" (UID: "278c8dac-4a47-4c32-814f-9f252ee64b0c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:09:42.064844 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.064816    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/278c8dac-4a47-4c32-814f-9f252ee64b0c-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:09:42.066583 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.066560    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/278c8dac-4a47-4c32-814f-9f252ee64b0c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "278c8dac-4a47-4c32-814f-9f252ee64b0c" (UID: "278c8dac-4a47-4c32-814f-9f252ee64b0c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:09:42.066665 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.066628    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278c8dac-4a47-4c32-814f-9f252ee64b0c-kube-api-access-7j9fq" (OuterVolumeSpecName: "kube-api-access-7j9fq") pod "278c8dac-4a47-4c32-814f-9f252ee64b0c" (UID: "278c8dac-4a47-4c32-814f-9f252ee64b0c"). InnerVolumeSpecName "kube-api-access-7j9fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:09:42.066714 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.066657    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/278c8dac-4a47-4c32-814f-9f252ee64b0c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "278c8dac-4a47-4c32-814f-9f252ee64b0c" (UID: "278c8dac-4a47-4c32-814f-9f252ee64b0c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:09:42.165391 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.165357    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/278c8dac-4a47-4c32-814f-9f252ee64b0c-metrics-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:09:42.165391 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.165384    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/278c8dac-4a47-4c32-814f-9f252ee64b0c-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:09:42.165391 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.165393    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7j9fq\" (UniqueName: \"kubernetes.io/projected/278c8dac-4a47-4c32-814f-9f252ee64b0c-kube-api-access-7j9fq\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:09:42.225372 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.225284    2560 generic.go:356] "Generic (PLEG): container finished" podID="c0c98b02-b638-4f04-83ec-7c4589a93efd" containerID="6089d7d50977ac420281b873f31f15dc2baa9ab1fa8877b8be4db234facbcdb4" exitCode=0
Sep 11 18:09:42.225372 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.225320    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr" event={"ID":"c0c98b02-b638-4f04-83ec-7c4589a93efd","Type":"ContainerDied","Data":"6089d7d50977ac420281b873f31f15dc2baa9ab1fa8877b8be4db234facbcdb4"}
Sep 11 18:09:42.225622 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.225384    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr" event={"ID":"c0c98b02-b638-4f04-83ec-7c4589a93efd","Type":"ContainerDied","Data":"59638cb663dd39f0d23c30da5526e2c9b94d68d6e3d1ac2ffee27ec9b4e7c627"}
Sep 11 18:09:42.225622 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.225328    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"
Sep 11 18:09:42.225622 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.225411    2560 scope.go:117] "RemoveContainer" containerID="6089d7d50977ac420281b873f31f15dc2baa9ab1fa8877b8be4db234facbcdb4"
Sep 11 18:09:42.226952 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.226932    2560 generic.go:356] "Generic (PLEG): container finished" podID="278c8dac-4a47-4c32-814f-9f252ee64b0c" containerID="a8255cb1fe1d0115aeeb86d3d269adceefef922a4e1db55bb2ebc9034b90e965" exitCode=0
Sep 11 18:09:42.227049 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.226964    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf" event={"ID":"278c8dac-4a47-4c32-814f-9f252ee64b0c","Type":"ContainerDied","Data":"a8255cb1fe1d0115aeeb86d3d269adceefef922a4e1db55bb2ebc9034b90e965"}
Sep 11 18:09:42.227049 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.226988    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"
Sep 11 18:09:42.227049 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.227006    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf" event={"ID":"278c8dac-4a47-4c32-814f-9f252ee64b0c","Type":"ContainerDied","Data":"bb36ebffc6dcc07212e37a50d375a6672dfe12f6db5baf937c80ba81f2009b5b"}
Sep 11 18:09:42.240121 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.240097    2560 scope.go:117] "RemoveContainer" containerID="6089d7d50977ac420281b873f31f15dc2baa9ab1fa8877b8be4db234facbcdb4"
Sep 11 18:09:42.240709 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:09:42.240683    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6089d7d50977ac420281b873f31f15dc2baa9ab1fa8877b8be4db234facbcdb4\": container with ID starting with 6089d7d50977ac420281b873f31f15dc2baa9ab1fa8877b8be4db234facbcdb4 not found: ID does not exist" containerID="6089d7d50977ac420281b873f31f15dc2baa9ab1fa8877b8be4db234facbcdb4"
Sep 11 18:09:42.240830 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.240719    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6089d7d50977ac420281b873f31f15dc2baa9ab1fa8877b8be4db234facbcdb4"} err="failed to get container status \"6089d7d50977ac420281b873f31f15dc2baa9ab1fa8877b8be4db234facbcdb4\": rpc error: code = NotFound desc = could not find container \"6089d7d50977ac420281b873f31f15dc2baa9ab1fa8877b8be4db234facbcdb4\": container with ID starting with 6089d7d50977ac420281b873f31f15dc2baa9ab1fa8877b8be4db234facbcdb4 not found: ID does not exist"
Sep 11 18:09:42.240830 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.240743    2560 scope.go:117] "RemoveContainer" containerID="a8255cb1fe1d0115aeeb86d3d269adceefef922a4e1db55bb2ebc9034b90e965"
Sep 11 18:09:42.249725 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.249709    2560 scope.go:117] "RemoveContainer" containerID="a8255cb1fe1d0115aeeb86d3d269adceefef922a4e1db55bb2ebc9034b90e965"
Sep 11 18:09:42.250173 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:09:42.250109    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8255cb1fe1d0115aeeb86d3d269adceefef922a4e1db55bb2ebc9034b90e965\": container with ID starting with a8255cb1fe1d0115aeeb86d3d269adceefef922a4e1db55bb2ebc9034b90e965 not found: ID does not exist" containerID="a8255cb1fe1d0115aeeb86d3d269adceefef922a4e1db55bb2ebc9034b90e965"
Sep 11 18:09:42.250226 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.250189    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8255cb1fe1d0115aeeb86d3d269adceefef922a4e1db55bb2ebc9034b90e965"} err="failed to get container status \"a8255cb1fe1d0115aeeb86d3d269adceefef922a4e1db55bb2ebc9034b90e965\": rpc error: code = NotFound desc = could not find container \"a8255cb1fe1d0115aeeb86d3d269adceefef922a4e1db55bb2ebc9034b90e965\": container with ID starting with a8255cb1fe1d0115aeeb86d3d269adceefef922a4e1db55bb2ebc9034b90e965 not found: ID does not exist"
Sep 11 18:09:42.273895 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.273860    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"]
Sep 11 18:09:42.291080 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.291043    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-736ba3-65f97f6fd5-w25dr"]
Sep 11 18:09:42.340536 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.340502    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"]
Sep 11 18:09:42.366985 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.366946    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-820e15b2-5fd64b6f6c-jgtdf"]
Sep 11 18:09:42.689187 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.689111    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="278c8dac-4a47-4c32-814f-9f252ee64b0c" path="/var/lib/kubelet/pods/278c8dac-4a47-4c32-814f-9f252ee64b0c/volumes"
Sep 11 18:09:42.689546 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:42.689510    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0c98b02-b638-4f04-83ec-7c4589a93efd" path="/var/lib/kubelet/pods/c0c98b02-b638-4f04-83ec-7c4589a93efd/volumes"
Sep 11 18:09:47.131820 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:47.131770    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"]
Sep 11 18:09:48.028465 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.028432    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"]
Sep 11 18:09:48.029081 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.029063    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0c98b02-b638-4f04-83ec-7c4589a93efd" containerName="maas-api"
Sep 11 18:09:48.029137 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.029087    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0c98b02-b638-4f04-83ec-7c4589a93efd" containerName="maas-api"
Sep 11 18:09:48.029137 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.029112    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="278c8dac-4a47-4c32-814f-9f252ee64b0c" containerName="maas-api"
Sep 11 18:09:48.029137 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.029121    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="278c8dac-4a47-4c32-814f-9f252ee64b0c" containerName="maas-api"
Sep 11 18:09:48.029250 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.029239    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0c98b02-b638-4f04-83ec-7c4589a93efd" containerName="maas-api"
Sep 11 18:09:48.029290 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.029258    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="278c8dac-4a47-4c32-814f-9f252ee64b0c" containerName="maas-api"
Sep 11 18:09:48.034274 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.034251    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"
Sep 11 18:09:48.038950 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.038925    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-85c5a226-serving-cert\""
Sep 11 18:09:48.052297 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.052268    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"]
Sep 11 18:09:48.123539 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.123506    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrv6h\" (UniqueName: \"kubernetes.io/projected/7f42b8b9-8017-4130-b8b0-56e38ea7831e-kube-api-access-nrv6h\") pod \"maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57\" (UID: \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"
Sep 11 18:09:48.123731 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.123557    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f42b8b9-8017-4130-b8b0-56e38ea7831e-maas-api-tls\") pod \"maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57\" (UID: \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"
Sep 11 18:09:48.123731 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.123600    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f42b8b9-8017-4130-b8b0-56e38ea7831e-metrics-certs\") pod \"maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57\" (UID: \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"
Sep 11 18:09:48.123731 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.123633    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f42b8b9-8017-4130-b8b0-56e38ea7831e-openshift-service-ca\") pod \"maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57\" (UID: \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"
Sep 11 18:09:48.224653 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.224617    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrv6h\" (UniqueName: \"kubernetes.io/projected/7f42b8b9-8017-4130-b8b0-56e38ea7831e-kube-api-access-nrv6h\") pod \"maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57\" (UID: \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"
Sep 11 18:09:48.225104 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.224678    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f42b8b9-8017-4130-b8b0-56e38ea7831e-maas-api-tls\") pod \"maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57\" (UID: \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"
Sep 11 18:09:48.225104 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.224703    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f42b8b9-8017-4130-b8b0-56e38ea7831e-metrics-certs\") pod \"maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57\" (UID: \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"
Sep 11 18:09:48.225104 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.224725    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f42b8b9-8017-4130-b8b0-56e38ea7831e-openshift-service-ca\") pod \"maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57\" (UID: \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"
Sep 11 18:09:48.225104 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:09:48.224838    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-serving-cert: secret "maas-api-e2e-derive-85c5a226-serving-cert" not found
Sep 11 18:09:48.225104 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:09:48.224906    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f42b8b9-8017-4130-b8b0-56e38ea7831e-maas-api-tls podName:7f42b8b9-8017-4130-b8b0-56e38ea7831e nodeName:}" failed. No retries permitted until 2026-09-11 18:09:48.724886182 +0000 UTC m=+946.568121601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7f42b8b9-8017-4130-b8b0-56e38ea7831e-maas-api-tls") pod "maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57" (UID: "7f42b8b9-8017-4130-b8b0-56e38ea7831e") : secret "maas-api-e2e-derive-85c5a226-serving-cert" not found
Sep 11 18:09:48.225333 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.225315    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f42b8b9-8017-4130-b8b0-56e38ea7831e-openshift-service-ca\") pod \"maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57\" (UID: \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"
Sep 11 18:09:48.227497 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.227476    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f42b8b9-8017-4130-b8b0-56e38ea7831e-metrics-certs\") pod \"maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57\" (UID: \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"
Sep 11 18:09:48.239149 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.239122    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrv6h\" (UniqueName: \"kubernetes.io/projected/7f42b8b9-8017-4130-b8b0-56e38ea7831e-kube-api-access-nrv6h\") pod \"maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57\" (UID: \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"
Sep 11 18:09:48.728619 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.728579    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f42b8b9-8017-4130-b8b0-56e38ea7831e-maas-api-tls\") pod \"maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57\" (UID: \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"
Sep 11 18:09:48.731181 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.731155    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f42b8b9-8017-4130-b8b0-56e38ea7831e-maas-api-tls\") pod \"maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57\" (UID: \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"
Sep 11 18:09:48.944490 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:48.944446    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"
Sep 11 18:09:49.223273 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:49.223249    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"]
Sep 11 18:09:49.225717 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:09:49.225693    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f42b8b9_8017_4130_b8b0_56e38ea7831e.slice/crio-201ff7c5c994333756702a7729d220ecf7ff916c6d235e53e76c0f31f31ac958 WatchSource:0}: Error finding container 201ff7c5c994333756702a7729d220ecf7ff916c6d235e53e76c0f31f31ac958: Status 404 returned error can't find the container with id 201ff7c5c994333756702a7729d220ecf7ff916c6d235e53e76c0f31f31ac958
Sep 11 18:09:49.267535 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:49.267456    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57" event={"ID":"7f42b8b9-8017-4130-b8b0-56e38ea7831e","Type":"ContainerStarted","Data":"201ff7c5c994333756702a7729d220ecf7ff916c6d235e53e76c0f31f31ac958"}
Sep 11 18:09:50.120274 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:50.120241    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:09:50.274377 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:50.274344    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57" event={"ID":"7f42b8b9-8017-4130-b8b0-56e38ea7831e","Type":"ContainerStarted","Data":"965149f458aef8d8e896cd440d90016899f0db7f4a5c83c326040247228c5f1a"}
Sep 11 18:09:50.274857 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:50.274485    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"
Sep 11 18:09:50.311886 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:50.311830    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57" podStartSLOduration=1.8968503 podStartE2EDuration="2.311812807s" totalImagesPullingTime="414.962507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:09:50.310456841 +0000 UTC m=+948.153692281" watchObservedRunningTime="2026-09-11 18:09:50.311812807 +0000 UTC m=+948.155048247"
Sep 11 18:09:56.295344 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:09:56.295315    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"
Sep 11 18:10:02.151918 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.151873    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss" podUID="4f54dd6c-8a4a-432b-8d18-e78430943353" containerName="main" containerID="cri-o://46b595b3257faef475894fb85de171d7e592fcdbed852c54b58aa982c577e48e" gracePeriod=45
Sep 11 18:10:02.355011 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.354981    2560 generic.go:356] "Generic (PLEG): container finished" podID="4f54dd6c-8a4a-432b-8d18-e78430943353" containerID="46b595b3257faef475894fb85de171d7e592fcdbed852c54b58aa982c577e48e" exitCode=0
Sep 11 18:10:02.355211 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.355057    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss" event={"ID":"4f54dd6c-8a4a-432b-8d18-e78430943353","Type":"ContainerDied","Data":"46b595b3257faef475894fb85de171d7e592fcdbed852c54b58aa982c577e48e"}
Sep 11 18:10:02.397736 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.397703    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:10:02.459694 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.459610    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-tmp-dir\") pod \"4f54dd6c-8a4a-432b-8d18-e78430943353\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") "
Sep 11 18:10:02.459873 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.459705    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-dshm\") pod \"4f54dd6c-8a4a-432b-8d18-e78430943353\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") "
Sep 11 18:10:02.459873 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.459787    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-model-cache\") pod \"4f54dd6c-8a4a-432b-8d18-e78430943353\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") "
Sep 11 18:10:02.459873 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.459834    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-home\") pod \"4f54dd6c-8a4a-432b-8d18-e78430943353\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") "
Sep 11 18:10:02.459873 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.459868    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvjp7\" (UniqueName: \"kubernetes.io/projected/4f54dd6c-8a4a-432b-8d18-e78430943353-kube-api-access-nvjp7\") pod \"4f54dd6c-8a4a-432b-8d18-e78430943353\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") "
Sep 11 18:10:02.460041 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.459897    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f54dd6c-8a4a-432b-8d18-e78430943353-tls-certs\") pod \"4f54dd6c-8a4a-432b-8d18-e78430943353\" (UID: \"4f54dd6c-8a4a-432b-8d18-e78430943353\") "
Sep 11 18:10:02.460041 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.459908    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f54dd6c-8a4a-432b-8d18-e78430943353" (UID: "4f54dd6c-8a4a-432b-8d18-e78430943353"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:10:02.460143 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.460094    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f54dd6c-8a4a-432b-8d18-e78430943353" (UID: "4f54dd6c-8a4a-432b-8d18-e78430943353"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:10:02.460201 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.460162    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-home" (OuterVolumeSpecName: "home") pod "4f54dd6c-8a4a-432b-8d18-e78430943353" (UID: "4f54dd6c-8a4a-432b-8d18-e78430943353"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:10:02.460283 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.460267    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-model-cache\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:10:02.460336 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.460288    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-home\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:10:02.460336 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.460300    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-tmp-dir\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:10:02.462224 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.462201    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f54dd6c-8a4a-432b-8d18-e78430943353-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f54dd6c-8a4a-432b-8d18-e78430943353" (UID: "4f54dd6c-8a4a-432b-8d18-e78430943353"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:10:02.462378 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.462358    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-dshm" (OuterVolumeSpecName: "dshm") pod "4f54dd6c-8a4a-432b-8d18-e78430943353" (UID: "4f54dd6c-8a4a-432b-8d18-e78430943353"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:10:02.462378 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.462363    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f54dd6c-8a4a-432b-8d18-e78430943353-kube-api-access-nvjp7" (OuterVolumeSpecName: "kube-api-access-nvjp7") pod "4f54dd6c-8a4a-432b-8d18-e78430943353" (UID: "4f54dd6c-8a4a-432b-8d18-e78430943353"). InnerVolumeSpecName "kube-api-access-nvjp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:10:02.562068 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.562034    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f54dd6c-8a4a-432b-8d18-e78430943353-dshm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:10:02.562068 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.562060    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvjp7\" (UniqueName: \"kubernetes.io/projected/4f54dd6c-8a4a-432b-8d18-e78430943353-kube-api-access-nvjp7\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:10:02.562068 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:02.562071    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f54dd6c-8a4a-432b-8d18-e78430943353-tls-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:10:03.362199 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:03.362157    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss" event={"ID":"4f54dd6c-8a4a-432b-8d18-e78430943353","Type":"ContainerDied","Data":"f25165270ceb01d97e1a412cd0d7092def53e60d1a26788a7c8e0f580d3b48c8"}
Sep 11 18:10:03.362602 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:03.362220    2560 scope.go:117] "RemoveContainer" containerID="46b595b3257faef475894fb85de171d7e592fcdbed852c54b58aa982c577e48e"
Sep 11 18:10:03.362602 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:03.362168    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"
Sep 11 18:10:03.381790 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:03.381763    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"]
Sep 11 18:10:03.385260 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:03.385230    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-736ba3/autores-model-736ba3-kserve-85cd768599-sj8ss"]
Sep 11 18:10:04.687914 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:04.687882    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f54dd6c-8a4a-432b-8d18-e78430943353" path="/var/lib/kubelet/pods/4f54dd6c-8a4a-432b-8d18-e78430943353/volumes"
Sep 11 18:10:08.030371 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:08.030336    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:10:17.958674 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:17.958644    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"]
Sep 11 18:10:17.959159 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:17.958925    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57" podUID="7f42b8b9-8017-4130-b8b0-56e38ea7831e" containerName="maas-api" containerID="cri-o://965149f458aef8d8e896cd440d90016899f0db7f4a5c83c326040247228c5f1a" gracePeriod=30
Sep 11 18:10:18.215373 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.215346    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"
Sep 11 18:10:18.320511 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.320476    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrv6h\" (UniqueName: \"kubernetes.io/projected/7f42b8b9-8017-4130-b8b0-56e38ea7831e-kube-api-access-nrv6h\") pod \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\" (UID: \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\") "
Sep 11 18:10:18.320715 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.320538    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f42b8b9-8017-4130-b8b0-56e38ea7831e-metrics-certs\") pod \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\" (UID: \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\") "
Sep 11 18:10:18.320715 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.320667    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f42b8b9-8017-4130-b8b0-56e38ea7831e-openshift-service-ca\") pod \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\" (UID: \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\") "
Sep 11 18:10:18.320715 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.320699    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f42b8b9-8017-4130-b8b0-56e38ea7831e-maas-api-tls\") pod \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\" (UID: \"7f42b8b9-8017-4130-b8b0-56e38ea7831e\") "
Sep 11 18:10:18.321089 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.321059    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f42b8b9-8017-4130-b8b0-56e38ea7831e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7f42b8b9-8017-4130-b8b0-56e38ea7831e" (UID: "7f42b8b9-8017-4130-b8b0-56e38ea7831e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:10:18.322868 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.322838    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f42b8b9-8017-4130-b8b0-56e38ea7831e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7f42b8b9-8017-4130-b8b0-56e38ea7831e" (UID: "7f42b8b9-8017-4130-b8b0-56e38ea7831e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:10:18.322868 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.322860    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f42b8b9-8017-4130-b8b0-56e38ea7831e-kube-api-access-nrv6h" (OuterVolumeSpecName: "kube-api-access-nrv6h") pod "7f42b8b9-8017-4130-b8b0-56e38ea7831e" (UID: "7f42b8b9-8017-4130-b8b0-56e38ea7831e"). InnerVolumeSpecName "kube-api-access-nrv6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:10:18.323004 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.322862    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f42b8b9-8017-4130-b8b0-56e38ea7831e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7f42b8b9-8017-4130-b8b0-56e38ea7831e" (UID: "7f42b8b9-8017-4130-b8b0-56e38ea7831e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:10:18.422321 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.422285    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f42b8b9-8017-4130-b8b0-56e38ea7831e-metrics-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:10:18.422321 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.422315    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f42b8b9-8017-4130-b8b0-56e38ea7831e-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:10:18.422321 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.422327    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f42b8b9-8017-4130-b8b0-56e38ea7831e-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:10:18.422557 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.422335    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrv6h\" (UniqueName: \"kubernetes.io/projected/7f42b8b9-8017-4130-b8b0-56e38ea7831e-kube-api-access-nrv6h\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:10:18.448501 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.448463    2560 generic.go:356] "Generic (PLEG): container finished" podID="7f42b8b9-8017-4130-b8b0-56e38ea7831e" containerID="965149f458aef8d8e896cd440d90016899f0db7f4a5c83c326040247228c5f1a" exitCode=0
Sep 11 18:10:18.448655 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.448537    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"
Sep 11 18:10:18.448655 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.448545    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57" event={"ID":"7f42b8b9-8017-4130-b8b0-56e38ea7831e","Type":"ContainerDied","Data":"965149f458aef8d8e896cd440d90016899f0db7f4a5c83c326040247228c5f1a"}
Sep 11 18:10:18.448655 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.448647    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57" event={"ID":"7f42b8b9-8017-4130-b8b0-56e38ea7831e","Type":"ContainerDied","Data":"201ff7c5c994333756702a7729d220ecf7ff916c6d235e53e76c0f31f31ac958"}
Sep 11 18:10:18.448807 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.448679    2560 scope.go:117] "RemoveContainer" containerID="965149f458aef8d8e896cd440d90016899f0db7f4a5c83c326040247228c5f1a"
Sep 11 18:10:18.459651 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.459633    2560 scope.go:117] "RemoveContainer" containerID="965149f458aef8d8e896cd440d90016899f0db7f4a5c83c326040247228c5f1a"
Sep 11 18:10:18.460029 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:10:18.460005    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"965149f458aef8d8e896cd440d90016899f0db7f4a5c83c326040247228c5f1a\": container with ID starting with 965149f458aef8d8e896cd440d90016899f0db7f4a5c83c326040247228c5f1a not found: ID does not exist" containerID="965149f458aef8d8e896cd440d90016899f0db7f4a5c83c326040247228c5f1a"
Sep 11 18:10:18.460115 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.460041    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"965149f458aef8d8e896cd440d90016899f0db7f4a5c83c326040247228c5f1a"} err="failed to get container status \"965149f458aef8d8e896cd440d90016899f0db7f4a5c83c326040247228c5f1a\": rpc error: code = NotFound desc = could not find container \"965149f458aef8d8e896cd440d90016899f0db7f4a5c83c326040247228c5f1a\": container with ID starting with 965149f458aef8d8e896cd440d90016899f0db7f4a5c83c326040247228c5f1a not found: ID does not exist"
Sep 11 18:10:18.501781 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.501671    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"]
Sep 11 18:10:18.504998 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.504974    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-85c5a226-5d7cc46bbf-xdn57"]
Sep 11 18:10:18.688251 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:18.688213    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f42b8b9-8017-4130-b8b0-56e38ea7831e" path="/var/lib/kubelet/pods/7f42b8b9-8017-4130-b8b0-56e38ea7831e/volumes"
Sep 11 18:10:24.822452 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:24.822371    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:10:42.012172 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.012127    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"]
Sep 11 18:10:42.012717 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.012575    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f54dd6c-8a4a-432b-8d18-e78430943353" containerName="main"
Sep 11 18:10:42.012717 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.012588    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f54dd6c-8a4a-432b-8d18-e78430943353" containerName="main"
Sep 11 18:10:42.012717 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.012599    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f42b8b9-8017-4130-b8b0-56e38ea7831e" containerName="maas-api"
Sep 11 18:10:42.012717 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.012605    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f42b8b9-8017-4130-b8b0-56e38ea7831e" containerName="maas-api"
Sep 11 18:10:42.012717 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.012700    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f42b8b9-8017-4130-b8b0-56e38ea7831e" containerName="maas-api"
Sep 11 18:10:42.012717 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.012714    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f54dd6c-8a4a-432b-8d18-e78430943353" containerName="main"
Sep 11 18:10:42.015951 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.015930    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"
Sep 11 18:10:42.020911 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.020889    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-aedf1a-serving-cert\""
Sep 11 18:10:42.031030 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.031002    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"]
Sep 11 18:10:42.056202 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.056165    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-metrics-certs\") pod \"maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9\" (UID: \"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"
Sep 11 18:10:42.056384 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.056287    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-maas-api-tls\") pod \"maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9\" (UID: \"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"
Sep 11 18:10:42.056384 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.056376    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-openshift-service-ca\") pod \"maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9\" (UID: \"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"
Sep 11 18:10:42.056470 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.056405    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pth55\" (UniqueName: \"kubernetes.io/projected/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-kube-api-access-pth55\") pod \"maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9\" (UID: \"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"
Sep 11 18:10:42.157803 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.157741    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-maas-api-tls\") pod \"maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9\" (UID: \"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"
Sep 11 18:10:42.157995 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.157874    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-openshift-service-ca\") pod \"maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9\" (UID: \"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"
Sep 11 18:10:42.157995 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.157915    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pth55\" (UniqueName: \"kubernetes.io/projected/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-kube-api-access-pth55\") pod \"maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9\" (UID: \"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"
Sep 11 18:10:42.157995 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.157973    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-metrics-certs\") pod \"maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9\" (UID: \"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"
Sep 11 18:10:42.158636 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.158598    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-openshift-service-ca\") pod \"maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9\" (UID: \"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"
Sep 11 18:10:42.160535 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.160512    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-metrics-certs\") pod \"maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9\" (UID: \"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"
Sep 11 18:10:42.160649 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.160608    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-maas-api-tls\") pod \"maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9\" (UID: \"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"
Sep 11 18:10:42.166858 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.166838    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pth55\" (UniqueName: \"kubernetes.io/projected/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-kube-api-access-pth55\") pod \"maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9\" (UID: \"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"
Sep 11 18:10:42.327864 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.327729    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"
Sep 11 18:10:42.605839 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.605814    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"]
Sep 11 18:10:42.607868 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:10:42.607827    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc590a7b_2eaa_4ffe_976d_71fd2195dbf6.slice/crio-63153d422685cf06a435017b227521f690d988a2173c6c037e9ad3a6202dea25 WatchSource:0}: Error finding container 63153d422685cf06a435017b227521f690d988a2173c6c037e9ad3a6202dea25: Status 404 returned error can't find the container with id 63153d422685cf06a435017b227521f690d988a2173c6c037e9ad3a6202dea25
Sep 11 18:10:42.609339 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:42.609322    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:10:43.583996 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:43.583951    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9" event={"ID":"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6","Type":"ContainerStarted","Data":"eeacb08d654207a2c39089de96faa58aa4a3a29b1be11cdf90faaca85d2d3c7d"}
Sep 11 18:10:43.584388 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:43.584024    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9" event={"ID":"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6","Type":"ContainerStarted","Data":"63153d422685cf06a435017b227521f690d988a2173c6c037e9ad3a6202dea25"}
Sep 11 18:10:43.584388 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:43.584055    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"
Sep 11 18:10:43.605187 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:43.605120    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9" podStartSLOduration=2.254172817 podStartE2EDuration="2.605102173s" totalImagesPullingTime="350.929356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:10:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:43.604964608 +0000 UTC m=+1001.448200062" watchObservedRunningTime="2026-09-11 18:10:43.605102173 +0000 UTC m=+1001.448337615"
Sep 11 18:10:48.815646 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:48.815608    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:10:49.595417 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:49.595385    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"
Sep 11 18:10:50.218734 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.218698    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"]
Sep 11 18:10:50.222525 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.222508    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:10:50.225403 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.225384    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f8a2d4\"/\"premium-w0-f8a2d4-kserve-self-signed-certs\""
Sep 11 18:10:50.231436 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.231410    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"]
Sep 11 18:10:50.340858 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.340821    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-dshm\") pod \"premium-w0-f8a2d4-kserve-66f694d96c-cvmmd\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") " pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:10:50.341031 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.340864    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxzk4\" (UniqueName: \"kubernetes.io/projected/05acfa53-02f8-4afc-8159-a495849827da-kube-api-access-bxzk4\") pod \"premium-w0-f8a2d4-kserve-66f694d96c-cvmmd\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") " pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:10:50.341031 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.340913    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-model-cache\") pod \"premium-w0-f8a2d4-kserve-66f694d96c-cvmmd\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") " pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:10:50.341111 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.341031    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-tmp-dir\") pod \"premium-w0-f8a2d4-kserve-66f694d96c-cvmmd\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") " pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:10:50.341111 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.341062    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-home\") pod \"premium-w0-f8a2d4-kserve-66f694d96c-cvmmd\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") " pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:10:50.341182 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.341144    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05acfa53-02f8-4afc-8159-a495849827da-tls-certs\") pod \"premium-w0-f8a2d4-kserve-66f694d96c-cvmmd\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") " pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:10:50.442150 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.442101    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05acfa53-02f8-4afc-8159-a495849827da-tls-certs\") pod \"premium-w0-f8a2d4-kserve-66f694d96c-cvmmd\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") " pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:10:50.442150 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.442160    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-dshm\") pod \"premium-w0-f8a2d4-kserve-66f694d96c-cvmmd\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") " pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:10:50.442377 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.442280    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxzk4\" (UniqueName: \"kubernetes.io/projected/05acfa53-02f8-4afc-8159-a495849827da-kube-api-access-bxzk4\") pod \"premium-w0-f8a2d4-kserve-66f694d96c-cvmmd\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") " pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:10:50.442377 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.442354    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-model-cache\") pod \"premium-w0-f8a2d4-kserve-66f694d96c-cvmmd\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") " pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:10:50.442476 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.442451    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-tmp-dir\") pod \"premium-w0-f8a2d4-kserve-66f694d96c-cvmmd\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") " pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:10:50.442533 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.442480    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-home\") pod \"premium-w0-f8a2d4-kserve-66f694d96c-cvmmd\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") " pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:10:50.442743 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.442723    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-model-cache\") pod \"premium-w0-f8a2d4-kserve-66f694d96c-cvmmd\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") " pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:10:50.442853 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.442791    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-tmp-dir\") pod \"premium-w0-f8a2d4-kserve-66f694d96c-cvmmd\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") " pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:10:50.443061 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.443039    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-home\") pod \"premium-w0-f8a2d4-kserve-66f694d96c-cvmmd\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") " pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:10:50.444519 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.444503    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-dshm\") pod \"premium-w0-f8a2d4-kserve-66f694d96c-cvmmd\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") " pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:10:50.444819 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.444802    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05acfa53-02f8-4afc-8159-a495849827da-tls-certs\") pod \"premium-w0-f8a2d4-kserve-66f694d96c-cvmmd\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") " pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:10:50.452559 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.452528    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxzk4\" (UniqueName: \"kubernetes.io/projected/05acfa53-02f8-4afc-8159-a495849827da-kube-api-access-bxzk4\") pod \"premium-w0-f8a2d4-kserve-66f694d96c-cvmmd\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") " pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:10:50.533806 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.533685    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:10:50.811643 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:50.811600    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"]
Sep 11 18:10:50.813940 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:10:50.813911    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05acfa53_02f8_4afc_8159_a495849827da.slice/crio-1d1635426586b5cb4aafd7e2d96ef58ac623c28d6ff42fd44ba88979536809dd WatchSource:0}: Error finding container 1d1635426586b5cb4aafd7e2d96ef58ac623c28d6ff42fd44ba88979536809dd: Status 404 returned error can't find the container with id 1d1635426586b5cb4aafd7e2d96ef58ac623c28d6ff42fd44ba88979536809dd
Sep 11 18:10:51.625343 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:51.625306    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd" event={"ID":"05acfa53-02f8-4afc-8159-a495849827da","Type":"ContainerStarted","Data":"e081b62c8db910a305fdb776f5650dbf69396a1ded0217b0e802e43f53895c9f"}
Sep 11 18:10:51.625343 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:51.625349    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd" event={"ID":"05acfa53-02f8-4afc-8159-a495849827da","Type":"ContainerStarted","Data":"1d1635426586b5cb4aafd7e2d96ef58ac623c28d6ff42fd44ba88979536809dd"}
Sep 11 18:10:51.649458 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:10:51.649399    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd" podStartSLOduration=1.649383792 podStartE2EDuration="1.649383792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:10:51.645683523 +0000 UTC m=+1009.488918976" watchObservedRunningTime="2026-09-11 18:10:51.649383792 +0000 UTC m=+1009.492619276"
Sep 11 18:11:00.534819 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:00.534782    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:11:00.535286 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:00.534834    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:11:00.547378 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:00.547350    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:11:00.693266 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:00.693239    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:11:08.326810 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:08.326769    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:11:29.528516 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.528480    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"]
Sep 11 18:11:29.531943 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.531924    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:29.534796 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.534768    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4e0b0b\"/\"premium-w4-4e0b0b-kserve-self-signed-certs\""
Sep 11 18:11:29.545234 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.545210    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"]
Sep 11 18:11:29.644200 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.644169    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-model-cache\") pod \"premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") " pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:29.644422 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.644207    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc4cj\" (UniqueName: \"kubernetes.io/projected/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-kube-api-access-dc4cj\") pod \"premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") " pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:29.644422 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.644271    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-tls-certs\") pod \"premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") " pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:29.644422 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.644394    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-dshm\") pod \"premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") " pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:29.644603 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.644427    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-tmp-dir\") pod \"premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") " pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:29.644603 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.644471    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-home\") pod \"premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") " pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:29.745908 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.745875    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-model-cache\") pod \"premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") " pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:29.745908 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.745907    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc4cj\" (UniqueName: \"kubernetes.io/projected/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-kube-api-access-dc4cj\") pod \"premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") " pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:29.746165 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.745937    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-tls-certs\") pod \"premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") " pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:29.746165 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.746014    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-dshm\") pod \"premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") " pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:29.746165 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.746050    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-tmp-dir\") pod \"premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") " pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:29.746165 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.746101    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-home\") pod \"premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") " pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:29.746370 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.746310    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-model-cache\") pod \"premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") " pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:29.746454 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.746433    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-tmp-dir\") pod \"premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") " pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:29.746513 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.746443    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-home\") pod \"premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") " pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:29.748398 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.748378    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-dshm\") pod \"premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") " pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:29.748548 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.748529    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-tls-certs\") pod \"premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") " pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:29.755611 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.755579    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc4cj\" (UniqueName: \"kubernetes.io/projected/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-kube-api-access-dc4cj\") pod \"premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") " pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:29.841556 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:29.841472    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:30.115703 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:30.115677    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"]
Sep 11 18:11:30.117091 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:11:30.117060    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35d12f6a_c0bf_4cdf_ac77_c78e773253c9.slice/crio-26ebbbd32b8eb82aa50f0f5ab384d25479a80ab7fb55afef2934a1526f024a26 WatchSource:0}: Error finding container 26ebbbd32b8eb82aa50f0f5ab384d25479a80ab7fb55afef2934a1526f024a26: Status 404 returned error can't find the container with id 26ebbbd32b8eb82aa50f0f5ab384d25479a80ab7fb55afef2934a1526f024a26
Sep 11 18:11:30.843088 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:30.843047    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr" event={"ID":"35d12f6a-c0bf-4cdf-ac77-c78e773253c9","Type":"ContainerStarted","Data":"3b051e92c0b1fb29c43b53808a7a8ffc7c87a22a83e09e4f44b55d369c5dfa05"}
Sep 11 18:11:30.843088 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:30.843102    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr" event={"ID":"35d12f6a-c0bf-4cdf-ac77-c78e773253c9","Type":"ContainerStarted","Data":"26ebbbd32b8eb82aa50f0f5ab384d25479a80ab7fb55afef2934a1526f024a26"}
Sep 11 18:11:30.860270 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:30.860225    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr" podStartSLOduration=1.8602069 podStartE2EDuration="1.8602069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:11:30.85969336 +0000 UTC m=+1048.702928828" watchObservedRunningTime="2026-09-11 18:11:30.8602069 +0000 UTC m=+1048.703442341"
Sep 11 18:11:34.326062 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:34.326027    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:11:39.842237 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:39.842190    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:39.842724 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:39.842255    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:39.854900 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:39.854872    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:39.900874 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:39.900841    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:11:51.117623 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:11:51.117587    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:12:10.587060 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:10.587022    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"]
Sep 11 18:12:10.590688 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:10.590672    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"
Sep 11 18:12:10.593671 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:10.593648    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7c2684c9-serving-cert\""
Sep 11 18:12:10.606535 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:10.606508    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"]
Sep 11 18:12:10.659164 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:10.659134    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e3dc46-1701-4c06-94ec-6b54d6ae1812-metrics-certs\") pod \"maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc\" (UID: \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"
Sep 11 18:12:10.659353 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:10.659189    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e3dc46-1701-4c06-94ec-6b54d6ae1812-openshift-service-ca\") pod \"maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc\" (UID: \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"
Sep 11 18:12:10.659353 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:10.659258    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc42f\" (UniqueName: \"kubernetes.io/projected/12e3dc46-1701-4c06-94ec-6b54d6ae1812-kube-api-access-zc42f\") pod \"maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc\" (UID: \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"
Sep 11 18:12:10.659353 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:10.659281    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e3dc46-1701-4c06-94ec-6b54d6ae1812-maas-api-tls\") pod \"maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc\" (UID: \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"
Sep 11 18:12:10.760227 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:10.760193    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e3dc46-1701-4c06-94ec-6b54d6ae1812-metrics-certs\") pod \"maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc\" (UID: \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"
Sep 11 18:12:10.760413 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:10.760269    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e3dc46-1701-4c06-94ec-6b54d6ae1812-openshift-service-ca\") pod \"maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc\" (UID: \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"
Sep 11 18:12:10.760413 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:10.760331    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc42f\" (UniqueName: \"kubernetes.io/projected/12e3dc46-1701-4c06-94ec-6b54d6ae1812-kube-api-access-zc42f\") pod \"maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc\" (UID: \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"
Sep 11 18:12:10.760413 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:10.760359    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e3dc46-1701-4c06-94ec-6b54d6ae1812-maas-api-tls\") pod \"maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc\" (UID: \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"
Sep 11 18:12:10.760579 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:12:10.760451    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-serving-cert: secret "maas-api-e2e-mt-7c2684c9-serving-cert" not found
Sep 11 18:12:10.760579 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:12:10.760505    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e3dc46-1701-4c06-94ec-6b54d6ae1812-maas-api-tls podName:12e3dc46-1701-4c06-94ec-6b54d6ae1812 nodeName:}" failed. No retries permitted until 2026-09-11 18:12:11.260488837 +0000 UTC m=+1089.103724256 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12e3dc46-1701-4c06-94ec-6b54d6ae1812-maas-api-tls") pod "maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc" (UID: "12e3dc46-1701-4c06-94ec-6b54d6ae1812") : secret "maas-api-e2e-mt-7c2684c9-serving-cert" not found
Sep 11 18:12:10.761009 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:10.760991    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e3dc46-1701-4c06-94ec-6b54d6ae1812-openshift-service-ca\") pod \"maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc\" (UID: \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"
Sep 11 18:12:10.762899 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:10.762877    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e3dc46-1701-4c06-94ec-6b54d6ae1812-metrics-certs\") pod \"maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc\" (UID: \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"
Sep 11 18:12:10.770349 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:10.770326    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc42f\" (UniqueName: \"kubernetes.io/projected/12e3dc46-1701-4c06-94ec-6b54d6ae1812-kube-api-access-zc42f\") pod \"maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc\" (UID: \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"
Sep 11 18:12:11.264991 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:11.264954    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e3dc46-1701-4c06-94ec-6b54d6ae1812-maas-api-tls\") pod \"maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc\" (UID: \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"
Sep 11 18:12:11.267882 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:11.267854    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e3dc46-1701-4c06-94ec-6b54d6ae1812-maas-api-tls\") pod \"maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc\" (UID: \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"
Sep 11 18:12:11.501386 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:11.501355    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"
Sep 11 18:12:11.776290 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:11.776263    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"]
Sep 11 18:12:11.778293 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:12:11.778267    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e3dc46_1701_4c06_94ec_6b54d6ae1812.slice/crio-f0a3fff5ca6d8abe08219dc576c7c0e230d5dec0f02ba56aa10d352d043ed2b9 WatchSource:0}: Error finding container f0a3fff5ca6d8abe08219dc576c7c0e230d5dec0f02ba56aa10d352d043ed2b9: Status 404 returned error can't find the container with id f0a3fff5ca6d8abe08219dc576c7c0e230d5dec0f02ba56aa10d352d043ed2b9
Sep 11 18:12:12.053167 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:12.052808    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc" event={"ID":"12e3dc46-1701-4c06-94ec-6b54d6ae1812","Type":"ContainerStarted","Data":"f0a3fff5ca6d8abe08219dc576c7c0e230d5dec0f02ba56aa10d352d043ed2b9"}
Sep 11 18:12:13.059599 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:13.059561    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc" event={"ID":"12e3dc46-1701-4c06-94ec-6b54d6ae1812","Type":"ContainerStarted","Data":"ca4244ef6cf7ac2e56c51e5b3a47a99bded68e2d90a0aa7789d5606fd31301cd"}
Sep 11 18:12:13.060046 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:13.059654    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"
Sep 11 18:12:13.079309 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:13.079264    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc" podStartSLOduration=2.5927389229999998 podStartE2EDuration="3.079249227s" totalImagesPullingTime="486.510304ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:12:13.077769929 +0000 UTC m=+1090.921005372" watchObservedRunningTime="2026-09-11 18:12:13.079249227 +0000 UTC m=+1090.922484668"
Sep 11 18:12:18.826534 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:18.826498    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:12:19.070296 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.070265    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"
Sep 11 18:12:19.344625 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.344594    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"]
Sep 11 18:12:19.348200 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.348185    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:19.352633 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.352615    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d59a02\"/\"kube-root-ca.crt\""
Sep 11 18:12:19.352765 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.352639    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d59a02\"/\"test-model-d59a02-kserve-self-signed-certs\""
Sep 11 18:12:19.354444 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.354425    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d59a02\"/\"default-dockercfg-874rj\""
Sep 11 18:12:19.354514 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.354425    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d59a02\"/\"openshift-service-ca.crt\""
Sep 11 18:12:19.363499 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.363477    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"]
Sep 11 18:12:19.446169 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.446137    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-home\") pod \"test-model-d59a02-kserve-86c87db47f-8nljq\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") " pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:19.446348 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.446183    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-tmp-dir\") pod \"test-model-d59a02-kserve-86c87db47f-8nljq\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") " pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:19.446348 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.446217    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-model-cache\") pod \"test-model-d59a02-kserve-86c87db47f-8nljq\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") " pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:19.446348 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.446235    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v97jm\" (UniqueName: \"kubernetes.io/projected/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-kube-api-access-v97jm\") pod \"test-model-d59a02-kserve-86c87db47f-8nljq\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") " pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:19.446348 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.446266    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-dshm\") pod \"test-model-d59a02-kserve-86c87db47f-8nljq\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") " pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:19.446348 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.446283    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-tls-certs\") pod \"test-model-d59a02-kserve-86c87db47f-8nljq\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") " pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:19.547125 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.547089    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-model-cache\") pod \"test-model-d59a02-kserve-86c87db47f-8nljq\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") " pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:19.547125 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.547130    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v97jm\" (UniqueName: \"kubernetes.io/projected/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-kube-api-access-v97jm\") pod \"test-model-d59a02-kserve-86c87db47f-8nljq\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") " pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:19.547373 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.547182    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-dshm\") pod \"test-model-d59a02-kserve-86c87db47f-8nljq\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") " pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:19.547373 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.547210    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-tls-certs\") pod \"test-model-d59a02-kserve-86c87db47f-8nljq\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") " pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:19.547373 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.547324    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-home\") pod \"test-model-d59a02-kserve-86c87db47f-8nljq\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") " pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:19.547526 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.547371    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-tmp-dir\") pod \"test-model-d59a02-kserve-86c87db47f-8nljq\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") " pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:19.547589 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.547557    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-model-cache\") pod \"test-model-d59a02-kserve-86c87db47f-8nljq\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") " pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:19.547725 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.547709    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-tmp-dir\") pod \"test-model-d59a02-kserve-86c87db47f-8nljq\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") " pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:19.547821 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.547706    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-home\") pod \"test-model-d59a02-kserve-86c87db47f-8nljq\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") " pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:19.549738 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.549717    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-dshm\") pod \"test-model-d59a02-kserve-86c87db47f-8nljq\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") " pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:19.549895 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.549880    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-tls-certs\") pod \"test-model-d59a02-kserve-86c87db47f-8nljq\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") " pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:19.556456 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.556438    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v97jm\" (UniqueName: \"kubernetes.io/projected/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-kube-api-access-v97jm\") pod \"test-model-d59a02-kserve-86c87db47f-8nljq\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") " pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:19.658210 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.658114    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:19.936378 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:19.936355    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"]
Sep 11 18:12:19.938599 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:12:19.938571    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod495b2b07_9b01_4091_bb0e_0e0015e0b3cd.slice/crio-9569beaff024a2b0eb36b801b8acdd3e63e67346ff7398f3f8bb204a7a89ee4d WatchSource:0}: Error finding container 9569beaff024a2b0eb36b801b8acdd3e63e67346ff7398f3f8bb204a7a89ee4d: Status 404 returned error can't find the container with id 9569beaff024a2b0eb36b801b8acdd3e63e67346ff7398f3f8bb204a7a89ee4d
Sep 11 18:12:20.101428 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:20.101387    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq" event={"ID":"495b2b07-9b01-4091-bb0e-0e0015e0b3cd","Type":"ContainerStarted","Data":"db94b53b79f2637d5a0f94152b3997d75d1f52023625f3a86c0a01ec0331c78c"}
Sep 11 18:12:20.101428 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:20.101432    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq" event={"ID":"495b2b07-9b01-4091-bb0e-0e0015e0b3cd","Type":"ContainerStarted","Data":"9569beaff024a2b0eb36b801b8acdd3e63e67346ff7398f3f8bb204a7a89ee4d"}
Sep 11 18:12:20.119600 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:20.119553    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq" podStartSLOduration=1.119536237 podStartE2EDuration="1.119536237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:12:20.119460382 +0000 UTC m=+1097.962695822" watchObservedRunningTime="2026-09-11 18:12:20.119536237 +0000 UTC m=+1097.962771679"
Sep 11 18:12:29.658357 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:29.658322    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:29.658778 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:29.658376    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:29.671170 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:29.671140    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:30.162462 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:30.162432    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:12:47.733547 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:47.733505    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:12:55.515459 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:55.515423    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"]
Sep 11 18:12:55.515992 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:55.515685    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc" podUID="12e3dc46-1701-4c06-94ec-6b54d6ae1812" containerName="maas-api" containerID="cri-o://ca4244ef6cf7ac2e56c51e5b3a47a99bded68e2d90a0aa7789d5606fd31301cd" gracePeriod=30
Sep 11 18:12:55.767543 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:55.767474    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"
Sep 11 18:12:55.820668 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:55.820634    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e3dc46-1701-4c06-94ec-6b54d6ae1812-openshift-service-ca\") pod \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\" (UID: \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\") "
Sep 11 18:12:55.820885 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:55.820802    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zc42f\" (UniqueName: \"kubernetes.io/projected/12e3dc46-1701-4c06-94ec-6b54d6ae1812-kube-api-access-zc42f\") pod \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\" (UID: \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\") "
Sep 11 18:12:55.820885 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:55.820831    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e3dc46-1701-4c06-94ec-6b54d6ae1812-metrics-certs\") pod \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\" (UID: \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\") "
Sep 11 18:12:55.820885 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:55.820856    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e3dc46-1701-4c06-94ec-6b54d6ae1812-maas-api-tls\") pod \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\" (UID: \"12e3dc46-1701-4c06-94ec-6b54d6ae1812\") "
Sep 11 18:12:55.821062 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:55.821041    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12e3dc46-1701-4c06-94ec-6b54d6ae1812-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "12e3dc46-1701-4c06-94ec-6b54d6ae1812" (UID: "12e3dc46-1701-4c06-94ec-6b54d6ae1812"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:12:55.821322 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:55.821301    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12e3dc46-1701-4c06-94ec-6b54d6ae1812-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:12:55.823179 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:55.823147    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12e3dc46-1701-4c06-94ec-6b54d6ae1812-kube-api-access-zc42f" (OuterVolumeSpecName: "kube-api-access-zc42f") pod "12e3dc46-1701-4c06-94ec-6b54d6ae1812" (UID: "12e3dc46-1701-4c06-94ec-6b54d6ae1812"). InnerVolumeSpecName "kube-api-access-zc42f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:12:55.823297 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:55.823183    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e3dc46-1701-4c06-94ec-6b54d6ae1812-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "12e3dc46-1701-4c06-94ec-6b54d6ae1812" (UID: "12e3dc46-1701-4c06-94ec-6b54d6ae1812"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:12:55.823297 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:55.823239    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12e3dc46-1701-4c06-94ec-6b54d6ae1812-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "12e3dc46-1701-4c06-94ec-6b54d6ae1812" (UID: "12e3dc46-1701-4c06-94ec-6b54d6ae1812"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:12:55.922844 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:55.922811    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc42f\" (UniqueName: \"kubernetes.io/projected/12e3dc46-1701-4c06-94ec-6b54d6ae1812-kube-api-access-zc42f\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:12:55.922844 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:55.922836    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12e3dc46-1701-4c06-94ec-6b54d6ae1812-metrics-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:12:55.922844 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:55.922847    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12e3dc46-1701-4c06-94ec-6b54d6ae1812-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:12:56.288656 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:56.288622    2560 generic.go:356] "Generic (PLEG): container finished" podID="12e3dc46-1701-4c06-94ec-6b54d6ae1812" containerID="ca4244ef6cf7ac2e56c51e5b3a47a99bded68e2d90a0aa7789d5606fd31301cd" exitCode=0
Sep 11 18:12:56.288877 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:56.288675    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"
Sep 11 18:12:56.288877 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:56.288716    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc" event={"ID":"12e3dc46-1701-4c06-94ec-6b54d6ae1812","Type":"ContainerDied","Data":"ca4244ef6cf7ac2e56c51e5b3a47a99bded68e2d90a0aa7789d5606fd31301cd"}
Sep 11 18:12:56.288877 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:56.288789    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc" event={"ID":"12e3dc46-1701-4c06-94ec-6b54d6ae1812","Type":"ContainerDied","Data":"f0a3fff5ca6d8abe08219dc576c7c0e230d5dec0f02ba56aa10d352d043ed2b9"}
Sep 11 18:12:56.288877 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:56.288812    2560 scope.go:117] "RemoveContainer" containerID="ca4244ef6cf7ac2e56c51e5b3a47a99bded68e2d90a0aa7789d5606fd31301cd"
Sep 11 18:12:56.298832 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:56.298815    2560 scope.go:117] "RemoveContainer" containerID="ca4244ef6cf7ac2e56c51e5b3a47a99bded68e2d90a0aa7789d5606fd31301cd"
Sep 11 18:12:56.299119 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:12:56.299101    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca4244ef6cf7ac2e56c51e5b3a47a99bded68e2d90a0aa7789d5606fd31301cd\": container with ID starting with ca4244ef6cf7ac2e56c51e5b3a47a99bded68e2d90a0aa7789d5606fd31301cd not found: ID does not exist" containerID="ca4244ef6cf7ac2e56c51e5b3a47a99bded68e2d90a0aa7789d5606fd31301cd"
Sep 11 18:12:56.299169 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:56.299128    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca4244ef6cf7ac2e56c51e5b3a47a99bded68e2d90a0aa7789d5606fd31301cd"} err="failed to get container status \"ca4244ef6cf7ac2e56c51e5b3a47a99bded68e2d90a0aa7789d5606fd31301cd\": rpc error: code = NotFound desc = could not find container \"ca4244ef6cf7ac2e56c51e5b3a47a99bded68e2d90a0aa7789d5606fd31301cd\": container with ID starting with ca4244ef6cf7ac2e56c51e5b3a47a99bded68e2d90a0aa7789d5606fd31301cd not found: ID does not exist"
Sep 11 18:12:56.314533 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:56.314506    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"]
Sep 11 18:12:56.319336 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:56.319308    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7c2684c9-779cf77d77-c69tc"]
Sep 11 18:12:56.688181 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:12:56.688101    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12e3dc46-1701-4c06-94ec-6b54d6ae1812" path="/var/lib/kubelet/pods/12e3dc46-1701-4c06-94ec-6b54d6ae1812/volumes"
Sep 11 18:13:17.318728 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.318694    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"]
Sep 11 18:13:17.319160 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.319145    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12e3dc46-1701-4c06-94ec-6b54d6ae1812" containerName="maas-api"
Sep 11 18:13:17.319205 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.319161    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="12e3dc46-1701-4c06-94ec-6b54d6ae1812" containerName="maas-api"
Sep 11 18:13:17.319240 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.319233    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="12e3dc46-1701-4c06-94ec-6b54d6ae1812" containerName="maas-api"
Sep 11 18:13:17.323515 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.323496    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:17.326538 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.326515    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f8a2d4\"/\"distinct-w0-f8a2d4-kserve-self-signed-certs\""
Sep 11 18:13:17.334107 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.334082    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"]
Sep 11 18:13:17.352630 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.352602    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjxmt\" (UniqueName: \"kubernetes.io/projected/82d24234-b1d5-4ec4-abba-aac23e2176f2-kube-api-access-gjxmt\") pod \"distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:17.352811 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.352656    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-model-cache\") pod \"distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:17.352811 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.352701    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-dshm\") pod \"distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:17.352811 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.352725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-tmp-dir\") pod \"distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:17.352811 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.352769    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82d24234-b1d5-4ec4-abba-aac23e2176f2-tls-certs\") pod \"distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:17.352811 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.352786    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-home\") pod \"distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:17.454117 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.454078    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-home\") pod \"distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:17.454323 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.454168    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjxmt\" (UniqueName: \"kubernetes.io/projected/82d24234-b1d5-4ec4-abba-aac23e2176f2-kube-api-access-gjxmt\") pod \"distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:17.454323 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.454228    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-model-cache\") pod \"distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:17.454323 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.454260    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-dshm\") pod \"distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:17.454323 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.454285    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-tmp-dir\") pod \"distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:17.454323 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.454298    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82d24234-b1d5-4ec4-abba-aac23e2176f2-tls-certs\") pod \"distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:17.454581 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.454524    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-home\") pod \"distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:17.454659 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.454633    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-model-cache\") pod \"distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:17.454743 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.454698    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-tmp-dir\") pod \"distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:17.456573 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.456556    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-dshm\") pod \"distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:17.456996 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.456976    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82d24234-b1d5-4ec4-abba-aac23e2176f2-tls-certs\") pod \"distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:17.464068 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.464047    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjxmt\" (UniqueName: \"kubernetes.io/projected/82d24234-b1d5-4ec4-abba-aac23e2176f2-kube-api-access-gjxmt\") pod \"distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") " pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:17.633801 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.633689    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:17.923847 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:17.923821    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"]
Sep 11 18:13:17.924710 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:13:17.924688    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82d24234_b1d5_4ec4_abba_aac23e2176f2.slice/crio-297151e6de60b63c3f99254115c9910ba79979741e9d9b8bf3d6ae8c2586c38f WatchSource:0}: Error finding container 297151e6de60b63c3f99254115c9910ba79979741e9d9b8bf3d6ae8c2586c38f: Status 404 returned error can't find the container with id 297151e6de60b63c3f99254115c9910ba79979741e9d9b8bf3d6ae8c2586c38f
Sep 11 18:13:18.407398 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:18.407358    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9" event={"ID":"82d24234-b1d5-4ec4-abba-aac23e2176f2","Type":"ContainerStarted","Data":"dbfa6f3ab227e214d6d34b32f863552489431b1f47b62a35cf2997c1b2590da1"}
Sep 11 18:13:18.407398 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:18.407403    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9" event={"ID":"82d24234-b1d5-4ec4-abba-aac23e2176f2","Type":"ContainerStarted","Data":"297151e6de60b63c3f99254115c9910ba79979741e9d9b8bf3d6ae8c2586c38f"}
Sep 11 18:13:18.466862 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:18.466812    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9" podStartSLOduration=1.466794743 podStartE2EDuration="1.466794743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:13:18.464661691 +0000 UTC m=+1156.307897133" watchObservedRunningTime="2026-09-11 18:13:18.466794743 +0000 UTC m=+1156.310030189"
Sep 11 18:13:20.420262 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:20.420226    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:13:25.180909 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.180821    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"]
Sep 11 18:13:25.181311 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.181027    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9" podUID="fc590a7b-2eaa-4ffe-976d-71fd2195dbf6" containerName="maas-api" containerID="cri-o://eeacb08d654207a2c39089de96faa58aa4a3a29b1be11cdf90faaca85d2d3c7d" gracePeriod=30
Sep 11 18:13:25.432270 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.432199    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"
Sep 11 18:13:25.454179 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.454146    2560 generic.go:356] "Generic (PLEG): container finished" podID="fc590a7b-2eaa-4ffe-976d-71fd2195dbf6" containerID="eeacb08d654207a2c39089de96faa58aa4a3a29b1be11cdf90faaca85d2d3c7d" exitCode=0
Sep 11 18:13:25.454362 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.454190    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"
Sep 11 18:13:25.454362 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.454228    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9" event={"ID":"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6","Type":"ContainerDied","Data":"eeacb08d654207a2c39089de96faa58aa4a3a29b1be11cdf90faaca85d2d3c7d"}
Sep 11 18:13:25.454362 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.454290    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9" event={"ID":"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6","Type":"ContainerDied","Data":"63153d422685cf06a435017b227521f690d988a2173c6c037e9ad3a6202dea25"}
Sep 11 18:13:25.454362 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.454319    2560 scope.go:117] "RemoveContainer" containerID="eeacb08d654207a2c39089de96faa58aa4a3a29b1be11cdf90faaca85d2d3c7d"
Sep 11 18:13:25.466452 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.466433    2560 scope.go:117] "RemoveContainer" containerID="eeacb08d654207a2c39089de96faa58aa4a3a29b1be11cdf90faaca85d2d3c7d"
Sep 11 18:13:25.466849 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:13:25.466811    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeacb08d654207a2c39089de96faa58aa4a3a29b1be11cdf90faaca85d2d3c7d\": container with ID starting with eeacb08d654207a2c39089de96faa58aa4a3a29b1be11cdf90faaca85d2d3c7d not found: ID does not exist" containerID="eeacb08d654207a2c39089de96faa58aa4a3a29b1be11cdf90faaca85d2d3c7d"
Sep 11 18:13:25.466849 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.466844    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeacb08d654207a2c39089de96faa58aa4a3a29b1be11cdf90faaca85d2d3c7d"} err="failed to get container status \"eeacb08d654207a2c39089de96faa58aa4a3a29b1be11cdf90faaca85d2d3c7d\": rpc error: code = NotFound desc = could not find container \"eeacb08d654207a2c39089de96faa58aa4a3a29b1be11cdf90faaca85d2d3c7d\": container with ID starting with eeacb08d654207a2c39089de96faa58aa4a3a29b1be11cdf90faaca85d2d3c7d not found: ID does not exist"
Sep 11 18:13:25.532403 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.532370    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pth55\" (UniqueName: \"kubernetes.io/projected/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-kube-api-access-pth55\") pod \"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6\" (UID: \"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6\") "
Sep 11 18:13:25.532577 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.532452    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-metrics-certs\") pod \"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6\" (UID: \"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6\") "
Sep 11 18:13:25.532577 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.532509    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-openshift-service-ca\") pod \"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6\" (UID: \"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6\") "
Sep 11 18:13:25.532695 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.532640    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-maas-api-tls\") pod \"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6\" (UID: \"fc590a7b-2eaa-4ffe-976d-71fd2195dbf6\") "
Sep 11 18:13:25.532897 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.532871    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fc590a7b-2eaa-4ffe-976d-71fd2195dbf6" (UID: "fc590a7b-2eaa-4ffe-976d-71fd2195dbf6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:13:25.533108 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.533089    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:13:25.534772 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.534721    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fc590a7b-2eaa-4ffe-976d-71fd2195dbf6" (UID: "fc590a7b-2eaa-4ffe-976d-71fd2195dbf6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:13:25.534772 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.534731    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-kube-api-access-pth55" (OuterVolumeSpecName: "kube-api-access-pth55") pod "fc590a7b-2eaa-4ffe-976d-71fd2195dbf6" (UID: "fc590a7b-2eaa-4ffe-976d-71fd2195dbf6"). InnerVolumeSpecName "kube-api-access-pth55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:13:25.534893 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.534862    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fc590a7b-2eaa-4ffe-976d-71fd2195dbf6" (UID: "fc590a7b-2eaa-4ffe-976d-71fd2195dbf6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:13:25.633707 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.633669    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pth55\" (UniqueName: \"kubernetes.io/projected/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-kube-api-access-pth55\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:13:25.633707 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.633701    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-metrics-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:13:25.633707 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.633710    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:13:25.778122 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.778092    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"]
Sep 11 18:13:25.781947 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:25.781923    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-aedf1a-7676df6dd-tb9b9"]
Sep 11 18:13:26.688109 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:26.688073    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc590a7b-2eaa-4ffe-976d-71fd2195dbf6" path="/var/lib/kubelet/pods/fc590a7b-2eaa-4ffe-976d-71fd2195dbf6/volumes"
Sep 11 18:13:27.634299 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:27.634264    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:27.634502 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:27.634345    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:27.647033 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:27.646997    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:28.482006 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:28.481981    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:13:54.087223 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.087187    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"]
Sep 11 18:13:54.087697 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.087611    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc590a7b-2eaa-4ffe-976d-71fd2195dbf6" containerName="maas-api"
Sep 11 18:13:54.087697 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.087623    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc590a7b-2eaa-4ffe-976d-71fd2195dbf6" containerName="maas-api"
Sep 11 18:13:54.087816 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.087701    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc590a7b-2eaa-4ffe-976d-71fd2195dbf6" containerName="maas-api"
Sep 11 18:13:54.091079 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.091060    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:13:54.094584 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.094564    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4e0b0b\"/\"e2e-conflict-a-74f751c5-kserve-self-signed-certs\""
Sep 11 18:13:54.094712 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.094640    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:13:54.113978 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.113953    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"]
Sep 11 18:13:54.132127 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.132099    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-model-cache\") pod \"e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:13:54.132283 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.132174    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-tmp-dir\") pod \"e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:13:54.132283 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.132218    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-home\") pod \"e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:13:54.132283 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.132240    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-dshm\") pod \"e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:13:54.132432 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.132380    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k22g9\" (UniqueName: \"kubernetes.io/projected/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-kube-api-access-k22g9\") pod \"e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:13:54.132483 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.132473    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-tls-certs\") pod \"e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:13:54.233732 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.233694    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-home\") pod \"e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:13:54.233732 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.233730    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-dshm\") pod \"e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:13:54.233979 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.233786    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k22g9\" (UniqueName: \"kubernetes.io/projected/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-kube-api-access-k22g9\") pod \"e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:13:54.233979 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.233844    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-tls-certs\") pod \"e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:13:54.233979 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.233916    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-model-cache\") pod \"e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:13:54.234120 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.233984    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-tmp-dir\") pod \"e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:13:54.234183 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.234121    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-home\") pod \"e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:13:54.234297 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.234278    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-tmp-dir\") pod \"e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:13:54.234358 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.234318    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-model-cache\") pod \"e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:13:54.236202 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.236184    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-dshm\") pod \"e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:13:54.236454 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.236436    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-tls-certs\") pod \"e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:13:54.243809 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.243788    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k22g9\" (UniqueName: \"kubernetes.io/projected/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-kube-api-access-k22g9\") pod \"e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") " pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:13:54.401431 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.401341    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:13:54.677256 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:54.677233    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"]
Sep 11 18:13:54.678996 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:13:54.678966    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c65b43a_736f_4ca6_8bc5_7e0f14eecd8f.slice/crio-a3c5f68c4d06c4bb5311f1989c7f25ee31ce786ee83f43f1b95753fe948afff4 WatchSource:0}: Error finding container a3c5f68c4d06c4bb5311f1989c7f25ee31ce786ee83f43f1b95753fe948afff4: Status 404 returned error can't find the container with id a3c5f68c4d06c4bb5311f1989c7f25ee31ce786ee83f43f1b95753fe948afff4
Sep 11 18:13:55.616596 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:55.616544    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh" event={"ID":"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f","Type":"ContainerStarted","Data":"0373fcf61969aa2ad810799c2f50f1e5bc8473f72a885094e42b320e83991552"}
Sep 11 18:13:55.616596 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:55.616604    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh" event={"ID":"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f","Type":"ContainerStarted","Data":"a3c5f68c4d06c4bb5311f1989c7f25ee31ce786ee83f43f1b95753fe948afff4"}
Sep 11 18:13:55.637675 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:13:55.637618    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh" podStartSLOduration=1.637603463 podStartE2EDuration="1.637603463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:13:55.637513161 +0000 UTC m=+1193.480748603" watchObservedRunningTime="2026-09-11 18:13:55.637603463 +0000 UTC m=+1193.480838904"
Sep 11 18:14:00.567636 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:00.567600    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"]
Sep 11 18:14:00.571267 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:00.571237    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"
Sep 11 18:14:00.574949 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:00.574929    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-90057a-serving-cert\""
Sep 11 18:14:00.586156 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:00.586128    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"]
Sep 11 18:14:00.698216 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:00.698182    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd879040-5d7d-4c11-9a8f-9c60ec980169-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt\" (UID: \"dd879040-5d7d-4c11-9a8f-9c60ec980169\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"
Sep 11 18:14:00.698358 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:00.698289    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l8m4\" (UniqueName: \"kubernetes.io/projected/dd879040-5d7d-4c11-9a8f-9c60ec980169-kube-api-access-6l8m4\") pod \"maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt\" (UID: \"dd879040-5d7d-4c11-9a8f-9c60ec980169\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"
Sep 11 18:14:00.698358 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:00.698312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd879040-5d7d-4c11-9a8f-9c60ec980169-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt\" (UID: \"dd879040-5d7d-4c11-9a8f-9c60ec980169\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"
Sep 11 18:14:00.698358 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:00.698339    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd879040-5d7d-4c11-9a8f-9c60ec980169-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt\" (UID: \"dd879040-5d7d-4c11-9a8f-9c60ec980169\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"
Sep 11 18:14:00.799306 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:00.799274    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l8m4\" (UniqueName: \"kubernetes.io/projected/dd879040-5d7d-4c11-9a8f-9c60ec980169-kube-api-access-6l8m4\") pod \"maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt\" (UID: \"dd879040-5d7d-4c11-9a8f-9c60ec980169\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"
Sep 11 18:14:00.799306 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:00.799313    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd879040-5d7d-4c11-9a8f-9c60ec980169-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt\" (UID: \"dd879040-5d7d-4c11-9a8f-9c60ec980169\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"
Sep 11 18:14:00.799554 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:00.799337    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd879040-5d7d-4c11-9a8f-9c60ec980169-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt\" (UID: \"dd879040-5d7d-4c11-9a8f-9c60ec980169\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"
Sep 11 18:14:00.799554 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:00.799450    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd879040-5d7d-4c11-9a8f-9c60ec980169-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt\" (UID: \"dd879040-5d7d-4c11-9a8f-9c60ec980169\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"
Sep 11 18:14:00.800148 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:00.800125    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd879040-5d7d-4c11-9a8f-9c60ec980169-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt\" (UID: \"dd879040-5d7d-4c11-9a8f-9c60ec980169\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"
Sep 11 18:14:00.802288 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:00.802263    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd879040-5d7d-4c11-9a8f-9c60ec980169-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt\" (UID: \"dd879040-5d7d-4c11-9a8f-9c60ec980169\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"
Sep 11 18:14:00.802382 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:00.802297    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd879040-5d7d-4c11-9a8f-9c60ec980169-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt\" (UID: \"dd879040-5d7d-4c11-9a8f-9c60ec980169\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"
Sep 11 18:14:00.809304 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:00.809279    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l8m4\" (UniqueName: \"kubernetes.io/projected/dd879040-5d7d-4c11-9a8f-9c60ec980169-kube-api-access-6l8m4\") pod \"maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt\" (UID: \"dd879040-5d7d-4c11-9a8f-9c60ec980169\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"
Sep 11 18:14:00.882469 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:00.882385    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"
Sep 11 18:14:01.157786 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:01.157738    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"]
Sep 11 18:14:01.159603 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:14:01.159570    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd879040_5d7d_4c11_9a8f_9c60ec980169.slice/crio-72e3af50598b4be1fde934dbc484cb065f55f4f853df1fb7956afaf5635a3438 WatchSource:0}: Error finding container 72e3af50598b4be1fde934dbc484cb065f55f4f853df1fb7956afaf5635a3438: Status 404 returned error can't find the container with id 72e3af50598b4be1fde934dbc484cb065f55f4f853df1fb7956afaf5635a3438
Sep 11 18:14:01.648097 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:01.648064    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt" event={"ID":"dd879040-5d7d-4c11-9a8f-9c60ec980169","Type":"ContainerStarted","Data":"a38dd7af42f53005307476bd2ebd3f344834e168568ab92335623173a3ba1041"}
Sep 11 18:14:01.648535 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:01.648155    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt" event={"ID":"dd879040-5d7d-4c11-9a8f-9c60ec980169","Type":"ContainerStarted","Data":"72e3af50598b4be1fde934dbc484cb065f55f4f853df1fb7956afaf5635a3438"}
Sep 11 18:14:01.648535 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:01.648275    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"
Sep 11 18:14:01.670325 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:01.670276    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt" podStartSLOduration=1.299081059 podStartE2EDuration="1.670258624s" totalImagesPullingTime="371.177565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:14:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:14:01.668883045 +0000 UTC m=+1199.512118496" watchObservedRunningTime="2026-09-11 18:14:01.670258624 +0000 UTC m=+1199.513494067"
Sep 11 18:14:02.675723 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:02.675692    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/2.log"
Sep 11 18:14:02.693631 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:02.693607    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/2.log"
Sep 11 18:14:04.402016 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:04.401978    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:14:04.402016 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:04.402027    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:14:04.414393 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:04.414368    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:14:04.674094 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:04.674016    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:14:07.659259 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:07.659226    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"
Sep 11 18:14:15.745296 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:15.745248    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:14:42.625361 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:42.625324    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:14:44.537154 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:44.537124    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"]
Sep 11 18:14:44.540717 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:44.540699    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"
Sep 11 18:14:44.543447 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:44.543424    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3a652320-serving-cert\""
Sep 11 18:14:44.552550 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:44.552468    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"]
Sep 11 18:14:44.651857 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:44.651821    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1ddce6d-1864-4060-a78e-9660801cd906-metrics-certs\") pod \"maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw\" (UID: \"b1ddce6d-1864-4060-a78e-9660801cd906\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"
Sep 11 18:14:44.651857 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:44.651856    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1ddce6d-1864-4060-a78e-9660801cd906-maas-api-tls\") pod \"maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw\" (UID: \"b1ddce6d-1864-4060-a78e-9660801cd906\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"
Sep 11 18:14:44.652066 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:44.651954    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmhv4\" (UniqueName: \"kubernetes.io/projected/b1ddce6d-1864-4060-a78e-9660801cd906-kube-api-access-gmhv4\") pod \"maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw\" (UID: \"b1ddce6d-1864-4060-a78e-9660801cd906\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"
Sep 11 18:14:44.652066 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:44.652011    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ddce6d-1864-4060-a78e-9660801cd906-openshift-service-ca\") pod \"maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw\" (UID: \"b1ddce6d-1864-4060-a78e-9660801cd906\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"
Sep 11 18:14:44.753478 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:44.753433    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmhv4\" (UniqueName: \"kubernetes.io/projected/b1ddce6d-1864-4060-a78e-9660801cd906-kube-api-access-gmhv4\") pod \"maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw\" (UID: \"b1ddce6d-1864-4060-a78e-9660801cd906\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"
Sep 11 18:14:44.753689 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:44.753546    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ddce6d-1864-4060-a78e-9660801cd906-openshift-service-ca\") pod \"maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw\" (UID: \"b1ddce6d-1864-4060-a78e-9660801cd906\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"
Sep 11 18:14:44.753689 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:44.753640    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1ddce6d-1864-4060-a78e-9660801cd906-metrics-certs\") pod \"maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw\" (UID: \"b1ddce6d-1864-4060-a78e-9660801cd906\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"
Sep 11 18:14:44.753689 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:44.753664    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1ddce6d-1864-4060-a78e-9660801cd906-maas-api-tls\") pod \"maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw\" (UID: \"b1ddce6d-1864-4060-a78e-9660801cd906\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"
Sep 11 18:14:44.753987 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:14:44.753966    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-serving-cert: secret "maas-api-e2e-mt-3a652320-serving-cert" not found
Sep 11 18:14:44.754084 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:14:44.754047    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ddce6d-1864-4060-a78e-9660801cd906-maas-api-tls podName:b1ddce6d-1864-4060-a78e-9660801cd906 nodeName:}" failed. No retries permitted until 2026-09-11 18:14:45.254027457 +0000 UTC m=+1243.097262876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b1ddce6d-1864-4060-a78e-9660801cd906-maas-api-tls") pod "maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw" (UID: "b1ddce6d-1864-4060-a78e-9660801cd906") : secret "maas-api-e2e-mt-3a652320-serving-cert" not found
Sep 11 18:14:44.754490 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:44.754471    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ddce6d-1864-4060-a78e-9660801cd906-openshift-service-ca\") pod \"maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw\" (UID: \"b1ddce6d-1864-4060-a78e-9660801cd906\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"
Sep 11 18:14:44.756336 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:44.756317    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1ddce6d-1864-4060-a78e-9660801cd906-metrics-certs\") pod \"maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw\" (UID: \"b1ddce6d-1864-4060-a78e-9660801cd906\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"
Sep 11 18:14:44.763522 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:44.763503    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmhv4\" (UniqueName: \"kubernetes.io/projected/b1ddce6d-1864-4060-a78e-9660801cd906-kube-api-access-gmhv4\") pod \"maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw\" (UID: \"b1ddce6d-1864-4060-a78e-9660801cd906\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"
Sep 11 18:14:45.258576 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:45.258539    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1ddce6d-1864-4060-a78e-9660801cd906-maas-api-tls\") pod \"maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw\" (UID: \"b1ddce6d-1864-4060-a78e-9660801cd906\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"
Sep 11 18:14:45.261263 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:45.261231    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1ddce6d-1864-4060-a78e-9660801cd906-maas-api-tls\") pod \"maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw\" (UID: \"b1ddce6d-1864-4060-a78e-9660801cd906\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"
Sep 11 18:14:45.452102 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:45.452065    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"
Sep 11 18:14:45.725972 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:45.725947    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"]
Sep 11 18:14:45.727128 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:14:45.727102    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ddce6d_1864_4060_a78e_9660801cd906.slice/crio-56ea31ba98d2c2d78ab3d432d977e2a0bef353c995e1aa68005231ccd3394192 WatchSource:0}: Error finding container 56ea31ba98d2c2d78ab3d432d977e2a0bef353c995e1aa68005231ccd3394192: Status 404 returned error can't find the container with id 56ea31ba98d2c2d78ab3d432d977e2a0bef353c995e1aa68005231ccd3394192
Sep 11 18:14:45.877473 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:45.877438    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw" event={"ID":"b1ddce6d-1864-4060-a78e-9660801cd906","Type":"ContainerStarted","Data":"56ea31ba98d2c2d78ab3d432d977e2a0bef353c995e1aa68005231ccd3394192"}
Sep 11 18:14:46.884580 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:46.884541    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw" event={"ID":"b1ddce6d-1864-4060-a78e-9660801cd906","Type":"ContainerStarted","Data":"8ee1b038e80ba8552c448cf71137260b3c1a500babc0b58a29208ae67b168153"}
Sep 11 18:14:46.884994 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:46.884608    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"
Sep 11 18:14:46.909228 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:46.909183    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw" podStartSLOduration=2.5102397549999997 podStartE2EDuration="2.90916707s" totalImagesPullingTime="398.927315ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:14:46.907785942 +0000 UTC m=+1244.751021379" watchObservedRunningTime="2026-09-11 18:14:46.90916707 +0000 UTC m=+1244.752402893"
Sep 11 18:14:52.896179 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:14:52.896153    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"
Sep 11 18:15:00.181578 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:00.181546    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-93598a-29819175-2rjw6"]
Sep 11 18:15:00.185097 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:00.185081    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-93598a-29819175-2rjw6"
Sep 11 18:15:00.190435 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:00.190412    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nxm8z\""
Sep 11 18:15:00.195449 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:00.195426    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-93598a-29819175-2rjw6"]
Sep 11 18:15:00.197009 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:00.196983    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-93598a-29819175-2rjw6"
Sep 11 18:15:00.201123 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:00.201092    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d59a02-29819175-bzj8n"]
Sep 11 18:15:00.206498 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:00.206478    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b05610-29819175-gsvl7"]
Sep 11 18:15:00.206608 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:00.206591    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d59a02-29819175-bzj8n"
Sep 11 18:15:00.214198 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:00.213885    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b05610-29819175-gsvl7"
Sep 11 18:15:00.220319 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:00.220225    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d59a02-29819175-bzj8n"]
Sep 11 18:15:00.223093 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:00.223073    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b05610-29819175-gsvl7"]
Sep 11 18:15:00.246987 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:00.246964    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d59a02-29819175-bzj8n"
Sep 11 18:15:00.255062 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:00.255038    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b05610-29819175-gsvl7"
Sep 11 18:15:00.488619 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:00.488552    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-93598a-29819175-2rjw6"]
Sep 11 18:15:00.491217 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:15:00.491192    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23551345_c9aa_4b52_a6e3_cb0500a99f1b.slice/crio-5a4c07fc39a0d892ca5aa8d3705e6aeccff6eb989b64e419f26c1b4bec4723bd WatchSource:0}: Error finding container 5a4c07fc39a0d892ca5aa8d3705e6aeccff6eb989b64e419f26c1b4bec4723bd: Status 404 returned error can't find the container with id 5a4c07fc39a0d892ca5aa8d3705e6aeccff6eb989b64e419f26c1b4bec4723bd
Sep 11 18:15:00.538804 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:00.538781    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b05610-29819175-gsvl7"]
Sep 11 18:15:00.540944 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:15:00.540915    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc968d5a2_8342_4163_a73b_d31a9d818c70.slice/crio-246ed66f95519556e1df6955927de2d337b211e439261232b202454995f0999e WatchSource:0}: Error finding container 246ed66f95519556e1df6955927de2d337b211e439261232b202454995f0999e: Status 404 returned error can't find the container with id 246ed66f95519556e1df6955927de2d337b211e439261232b202454995f0999e
Sep 11 18:15:00.542163 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:00.542141    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d59a02-29819175-bzj8n"]
Sep 11 18:15:00.543834 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:15:00.543703    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e215f20_a56e_4fb5_a9bd_f037b8819b24.slice/crio-790e5cf71edbb81e5f128589ad6a09798da99d1e64ef3f7daed34648733fd7d2 WatchSource:0}: Error finding container 790e5cf71edbb81e5f128589ad6a09798da99d1e64ef3f7daed34648733fd7d2: Status 404 returned error can't find the container with id 790e5cf71edbb81e5f128589ad6a09798da99d1e64ef3f7daed34648733fd7d2
Sep 11 18:15:00.958889 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:00.958793    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b05610-29819175-gsvl7" event={"ID":"c968d5a2-8342-4163-a73b-d31a9d818c70","Type":"ContainerStarted","Data":"246ed66f95519556e1df6955927de2d337b211e439261232b202454995f0999e"}
Sep 11 18:15:00.960070 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:00.960038    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-93598a-29819175-2rjw6" event={"ID":"23551345-c9aa-4b52-a6e3-cb0500a99f1b","Type":"ContainerStarted","Data":"5a4c07fc39a0d892ca5aa8d3705e6aeccff6eb989b64e419f26c1b4bec4723bd"}
Sep 11 18:15:00.961113 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:00.961096    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d59a02-29819175-bzj8n" event={"ID":"7e215f20-a56e-4fb5-a9bd-f037b8819b24","Type":"ContainerStarted","Data":"790e5cf71edbb81e5f128589ad6a09798da99d1e64ef3f7daed34648733fd7d2"}
Sep 11 18:15:03.982142 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:03.982100    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-93598a-29819175-2rjw6" event={"ID":"23551345-c9aa-4b52-a6e3-cb0500a99f1b","Type":"ContainerStarted","Data":"480e886044e43d24489c7ebcfbb6d7e32dca0f7498b0925d3e0e6dd1ea24e8b2"}
Sep 11 18:15:03.983800 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:03.983773    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d59a02-29819175-bzj8n" event={"ID":"7e215f20-a56e-4fb5-a9bd-f037b8819b24","Type":"ContainerStarted","Data":"66ac85702f445a22d422145a28d052e02a9f8d1e1007ddbf915b259139599161"}
Sep 11 18:15:03.985219 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:03.985189    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b05610-29819175-gsvl7" event={"ID":"c968d5a2-8342-4163-a73b-d31a9d818c70","Type":"ContainerStarted","Data":"fe5bfe49ea0316d817479d5e49a1441ea7771db3309f5d16e07c2eca6bf83a7b"}
Sep 11 18:15:04.001473 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:04.001425    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-93598a-29819175-2rjw6" podStartSLOduration=1.259347176 podStartE2EDuration="4.001409861s" totalImagesPullingTime="2.742062685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:04.000276331 +0000 UTC m=+1261.843511791" watchObservedRunningTime="2026-09-11 18:15:04.001409861 +0000 UTC m=+1261.844645302"
Sep 11 18:15:04.020558 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:04.020513    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d59a02-29819175-bzj8n" podStartSLOduration=1.335455341 podStartE2EDuration="4.02049753s" totalImagesPullingTime="2.685042189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:04.01889558 +0000 UTC m=+1261.862131030" watchObservedRunningTime="2026-09-11 18:15:04.02049753 +0000 UTC m=+1261.863732971"
Sep 11 18:15:04.038222 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:04.038182    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b05610-29819175-gsvl7" podStartSLOduration=1.350247283 podStartE2EDuration="4.038171754s" totalImagesPullingTime="2.687924471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:04.036871995 +0000 UTC m=+1261.880107436" watchObservedRunningTime="2026-09-11 18:15:04.038171754 +0000 UTC m=+1261.881407195"
Sep 11 18:15:04.992940 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:04.992909    2560 generic.go:356] "Generic (PLEG): container finished" podID="23551345-c9aa-4b52-a6e3-cb0500a99f1b" containerID="480e886044e43d24489c7ebcfbb6d7e32dca0f7498b0925d3e0e6dd1ea24e8b2" exitCode=0
Sep 11 18:15:04.993398 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:04.992998    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-93598a-29819175-2rjw6" event={"ID":"23551345-c9aa-4b52-a6e3-cb0500a99f1b","Type":"ContainerDied","Data":"480e886044e43d24489c7ebcfbb6d7e32dca0f7498b0925d3e0e6dd1ea24e8b2"}
Sep 11 18:15:04.994527 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:04.994501    2560 generic.go:356] "Generic (PLEG): container finished" podID="7e215f20-a56e-4fb5-a9bd-f037b8819b24" containerID="66ac85702f445a22d422145a28d052e02a9f8d1e1007ddbf915b259139599161" exitCode=0
Sep 11 18:15:04.994653 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:04.994540    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d59a02-29819175-bzj8n" event={"ID":"7e215f20-a56e-4fb5-a9bd-f037b8819b24","Type":"ContainerDied","Data":"66ac85702f445a22d422145a28d052e02a9f8d1e1007ddbf915b259139599161"}
Sep 11 18:15:04.996114 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:04.996089    2560 generic.go:356] "Generic (PLEG): container finished" podID="c968d5a2-8342-4163-a73b-d31a9d818c70" containerID="fe5bfe49ea0316d817479d5e49a1441ea7771db3309f5d16e07c2eca6bf83a7b" exitCode=0
Sep 11 18:15:04.996219 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:04.996152    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b05610-29819175-gsvl7" event={"ID":"c968d5a2-8342-4163-a73b-d31a9d818c70","Type":"ContainerDied","Data":"fe5bfe49ea0316d817479d5e49a1441ea7771db3309f5d16e07c2eca6bf83a7b"}
Sep 11 18:15:06.168312 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:06.168285    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b05610-29819175-gsvl7"
Sep 11 18:15:06.171672 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:06.171652    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d59a02-29819175-bzj8n"
Sep 11 18:15:06.211205 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:06.211177    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-93598a-29819175-2rjw6"
Sep 11 18:15:07.007765 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:07.007718    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b05610-29819175-gsvl7"
Sep 11 18:15:07.007955 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:07.007769    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b05610-29819175-gsvl7" event={"ID":"c968d5a2-8342-4163-a73b-d31a9d818c70","Type":"ContainerDied","Data":"246ed66f95519556e1df6955927de2d337b211e439261232b202454995f0999e"}
Sep 11 18:15:07.007955 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:07.007822    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="246ed66f95519556e1df6955927de2d337b211e439261232b202454995f0999e"
Sep 11 18:15:07.009160 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:07.009142    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-93598a-29819175-2rjw6"
Sep 11 18:15:07.009305 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:07.009187    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-93598a-29819175-2rjw6" event={"ID":"23551345-c9aa-4b52-a6e3-cb0500a99f1b","Type":"ContainerDied","Data":"5a4c07fc39a0d892ca5aa8d3705e6aeccff6eb989b64e419f26c1b4bec4723bd"}
Sep 11 18:15:07.009305 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:07.009227    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a4c07fc39a0d892ca5aa8d3705e6aeccff6eb989b64e419f26c1b4bec4723bd"
Sep 11 18:15:07.010679 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:07.010662    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d59a02-29819175-bzj8n"
Sep 11 18:15:07.010679 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:07.010672    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d59a02-29819175-bzj8n" event={"ID":"7e215f20-a56e-4fb5-a9bd-f037b8819b24","Type":"ContainerDied","Data":"790e5cf71edbb81e5f128589ad6a09798da99d1e64ef3f7daed34648733fd7d2"}
Sep 11 18:15:07.010855 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:07.010706    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="790e5cf71edbb81e5f128589ad6a09798da99d1e64ef3f7daed34648733fd7d2"
Sep 11 18:15:14.830970 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:14.830931    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:15:19.568408 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:19.568370    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-93598a-29819175-2rjw6"]
Sep 11 18:15:19.572241 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:19.572214    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-93598a-29819175-2rjw6"]
Sep 11 18:15:20.689224 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:20.689194    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23551345-c9aa-4b52-a6e3-cb0500a99f1b" path="/var/lib/kubelet/pods/23551345-c9aa-4b52-a6e3-cb0500a99f1b/volumes"
Sep 11 18:15:31.794020 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:31.793989    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"]
Sep 11 18:15:44.438589 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:44.438550    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:15:46.814049 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:46.813983    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq" podUID="495b2b07-9b01-4091-bb0e-0e0015e0b3cd" containerName="main" containerID="cri-o://db94b53b79f2637d5a0f94152b3997d75d1f52023625f3a86c0a01ec0331c78c" gracePeriod=45
Sep 11 18:15:47.067322 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.067261    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:15:47.203027 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.202982    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-dshm\") pod \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") "
Sep 11 18:15:47.203208 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.203037    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v97jm\" (UniqueName: \"kubernetes.io/projected/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-kube-api-access-v97jm\") pod \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") "
Sep 11 18:15:47.203208 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.203063    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-tmp-dir\") pod \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") "
Sep 11 18:15:47.203208 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.203165    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-model-cache\") pod \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") "
Sep 11 18:15:47.203208 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.203197    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-home\") pod \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") "
Sep 11 18:15:47.203469 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.203219    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-tls-certs\") pod \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\" (UID: \"495b2b07-9b01-4091-bb0e-0e0015e0b3cd\") "
Sep 11 18:15:47.203469 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.203421    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "495b2b07-9b01-4091-bb0e-0e0015e0b3cd" (UID: "495b2b07-9b01-4091-bb0e-0e0015e0b3cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:47.203589 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.203550    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "495b2b07-9b01-4091-bb0e-0e0015e0b3cd" (UID: "495b2b07-9b01-4091-bb0e-0e0015e0b3cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:47.203648 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.203581    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-home" (OuterVolumeSpecName: "home") pod "495b2b07-9b01-4091-bb0e-0e0015e0b3cd" (UID: "495b2b07-9b01-4091-bb0e-0e0015e0b3cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:47.203707 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.203660    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-tmp-dir\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:15:47.203707 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.203672    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-model-cache\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:15:47.205580 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.205554    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-dshm" (OuterVolumeSpecName: "dshm") pod "495b2b07-9b01-4091-bb0e-0e0015e0b3cd" (UID: "495b2b07-9b01-4091-bb0e-0e0015e0b3cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:15:47.205702 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.205554    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "495b2b07-9b01-4091-bb0e-0e0015e0b3cd" (UID: "495b2b07-9b01-4091-bb0e-0e0015e0b3cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:15:47.205702 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.205581    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-kube-api-access-v97jm" (OuterVolumeSpecName: "kube-api-access-v97jm") pod "495b2b07-9b01-4091-bb0e-0e0015e0b3cd" (UID: "495b2b07-9b01-4091-bb0e-0e0015e0b3cd"). InnerVolumeSpecName "kube-api-access-v97jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:15:47.252129 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.252098    2560 generic.go:356] "Generic (PLEG): container finished" podID="495b2b07-9b01-4091-bb0e-0e0015e0b3cd" containerID="db94b53b79f2637d5a0f94152b3997d75d1f52023625f3a86c0a01ec0331c78c" exitCode=0
Sep 11 18:15:47.252293 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.252143    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"
Sep 11 18:15:47.252293 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.252141    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq" event={"ID":"495b2b07-9b01-4091-bb0e-0e0015e0b3cd","Type":"ContainerDied","Data":"db94b53b79f2637d5a0f94152b3997d75d1f52023625f3a86c0a01ec0331c78c"}
Sep 11 18:15:47.252293 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.252187    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq" event={"ID":"495b2b07-9b01-4091-bb0e-0e0015e0b3cd","Type":"ContainerDied","Data":"9569beaff024a2b0eb36b801b8acdd3e63e67346ff7398f3f8bb204a7a89ee4d"}
Sep 11 18:15:47.252293 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.252209    2560 scope.go:117] "RemoveContainer" containerID="db94b53b79f2637d5a0f94152b3997d75d1f52023625f3a86c0a01ec0331c78c"
Sep 11 18:15:47.262493 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.262472    2560 scope.go:117] "RemoveContainer" containerID="db94b53b79f2637d5a0f94152b3997d75d1f52023625f3a86c0a01ec0331c78c"
Sep 11 18:15:47.262742 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:15:47.262722    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db94b53b79f2637d5a0f94152b3997d75d1f52023625f3a86c0a01ec0331c78c\": container with ID starting with db94b53b79f2637d5a0f94152b3997d75d1f52023625f3a86c0a01ec0331c78c not found: ID does not exist" containerID="db94b53b79f2637d5a0f94152b3997d75d1f52023625f3a86c0a01ec0331c78c"
Sep 11 18:15:47.262836 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.262792    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db94b53b79f2637d5a0f94152b3997d75d1f52023625f3a86c0a01ec0331c78c"} err="failed to get container status \"db94b53b79f2637d5a0f94152b3997d75d1f52023625f3a86c0a01ec0331c78c\": rpc error: code = NotFound desc = could not find container \"db94b53b79f2637d5a0f94152b3997d75d1f52023625f3a86c0a01ec0331c78c\": container with ID starting with db94b53b79f2637d5a0f94152b3997d75d1f52023625f3a86c0a01ec0331c78c not found: ID does not exist"
Sep 11 18:15:47.292579 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.292543    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"]
Sep 11 18:15:47.301493 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.301464    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d59a02/test-model-d59a02-kserve-86c87db47f-8nljq"]
Sep 11 18:15:47.304794 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.304774    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-dshm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:15:47.304896 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.304797    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v97jm\" (UniqueName: \"kubernetes.io/projected/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-kube-api-access-v97jm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:15:47.304896 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.304810    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-home\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:15:47.304896 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:47.304825    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/495b2b07-9b01-4091-bb0e-0e0015e0b3cd-tls-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:15:48.688605 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:48.688572    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="495b2b07-9b01-4091-bb0e-0e0015e0b3cd" path="/var/lib/kubelet/pods/495b2b07-9b01-4091-bb0e-0e0015e0b3cd/volumes"
Sep 11 18:15:52.550172 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:52.550137    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"]
Sep 11 18:15:52.550624 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:52.550414    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7" podUID="000c8e16-03b0-406c-b818-355e531789a9" containerName="maas-api" containerID="cri-o://a3e693e5537eaca1a5a952cc55985cbee3df27501db767bda18f0839a7e8eb1d" gracePeriod=30
Sep 11 18:15:52.579008 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:52.578950    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d59a02-29819175-bzj8n"]
Sep 11 18:15:52.582204 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:52.582182    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-d59a02-29819175-bzj8n"]
Sep 11 18:15:52.692684 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:52.692649    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e215f20-a56e-4fb5-a9bd-f037b8819b24" path="/var/lib/kubelet/pods/7e215f20-a56e-4fb5-a9bd-f037b8819b24/volumes"
Sep 11 18:15:52.814144 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:52.814082    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"
Sep 11 18:15:52.969050 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:52.969006    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj5mk\" (UniqueName: \"kubernetes.io/projected/000c8e16-03b0-406c-b818-355e531789a9-kube-api-access-tj5mk\") pod \"000c8e16-03b0-406c-b818-355e531789a9\" (UID: \"000c8e16-03b0-406c-b818-355e531789a9\") "
Sep 11 18:15:52.969233 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:52.969099    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/000c8e16-03b0-406c-b818-355e531789a9-openshift-service-ca\") pod \"000c8e16-03b0-406c-b818-355e531789a9\" (UID: \"000c8e16-03b0-406c-b818-355e531789a9\") "
Sep 11 18:15:52.969277 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:52.969243    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/000c8e16-03b0-406c-b818-355e531789a9-metrics-certs\") pod \"000c8e16-03b0-406c-b818-355e531789a9\" (UID: \"000c8e16-03b0-406c-b818-355e531789a9\") "
Sep 11 18:15:52.969316 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:52.969286    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/000c8e16-03b0-406c-b818-355e531789a9-maas-api-tls\") pod \"000c8e16-03b0-406c-b818-355e531789a9\" (UID: \"000c8e16-03b0-406c-b818-355e531789a9\") "
Sep 11 18:15:52.969525 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:52.969483    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/000c8e16-03b0-406c-b818-355e531789a9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "000c8e16-03b0-406c-b818-355e531789a9" (UID: "000c8e16-03b0-406c-b818-355e531789a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:15:52.969641 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:52.969626    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/000c8e16-03b0-406c-b818-355e531789a9-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:15:52.971464 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:52.971432    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/000c8e16-03b0-406c-b818-355e531789a9-kube-api-access-tj5mk" (OuterVolumeSpecName: "kube-api-access-tj5mk") pod "000c8e16-03b0-406c-b818-355e531789a9" (UID: "000c8e16-03b0-406c-b818-355e531789a9"). InnerVolumeSpecName "kube-api-access-tj5mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:15:52.971648 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:52.971628    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/000c8e16-03b0-406c-b818-355e531789a9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "000c8e16-03b0-406c-b818-355e531789a9" (UID: "000c8e16-03b0-406c-b818-355e531789a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:15:52.971728 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:52.971655    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/000c8e16-03b0-406c-b818-355e531789a9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "000c8e16-03b0-406c-b818-355e531789a9" (UID: "000c8e16-03b0-406c-b818-355e531789a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:15:53.071320 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:53.071240    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/000c8e16-03b0-406c-b818-355e531789a9-metrics-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:15:53.071320 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:53.071267    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/000c8e16-03b0-406c-b818-355e531789a9-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:15:53.071320 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:53.071276    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj5mk\" (UniqueName: \"kubernetes.io/projected/000c8e16-03b0-406c-b818-355e531789a9-kube-api-access-tj5mk\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:15:53.287200 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:53.287160    2560 generic.go:356] "Generic (PLEG): container finished" podID="000c8e16-03b0-406c-b818-355e531789a9" containerID="a3e693e5537eaca1a5a952cc55985cbee3df27501db767bda18f0839a7e8eb1d" exitCode=0
Sep 11 18:15:53.287367 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:53.287209    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"
Sep 11 18:15:53.287367 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:53.287249    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7" event={"ID":"000c8e16-03b0-406c-b818-355e531789a9","Type":"ContainerDied","Data":"a3e693e5537eaca1a5a952cc55985cbee3df27501db767bda18f0839a7e8eb1d"}
Sep 11 18:15:53.287367 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:53.287308    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7" event={"ID":"000c8e16-03b0-406c-b818-355e531789a9","Type":"ContainerDied","Data":"5976256fddffd88943d0c8c2a76406f2f19b21689d343cec397f5da9a48296d3"}
Sep 11 18:15:53.287367 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:53.287338    2560 scope.go:117] "RemoveContainer" containerID="a3e693e5537eaca1a5a952cc55985cbee3df27501db767bda18f0839a7e8eb1d"
Sep 11 18:15:53.297543 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:53.297525    2560 scope.go:117] "RemoveContainer" containerID="a3e693e5537eaca1a5a952cc55985cbee3df27501db767bda18f0839a7e8eb1d"
Sep 11 18:15:53.297831 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:15:53.297812    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e693e5537eaca1a5a952cc55985cbee3df27501db767bda18f0839a7e8eb1d\": container with ID starting with a3e693e5537eaca1a5a952cc55985cbee3df27501db767bda18f0839a7e8eb1d not found: ID does not exist" containerID="a3e693e5537eaca1a5a952cc55985cbee3df27501db767bda18f0839a7e8eb1d"
Sep 11 18:15:53.297918 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:53.297838    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e693e5537eaca1a5a952cc55985cbee3df27501db767bda18f0839a7e8eb1d"} err="failed to get container status \"a3e693e5537eaca1a5a952cc55985cbee3df27501db767bda18f0839a7e8eb1d\": rpc error: code = NotFound desc = could not find container \"a3e693e5537eaca1a5a952cc55985cbee3df27501db767bda18f0839a7e8eb1d\": container with ID starting with a3e693e5537eaca1a5a952cc55985cbee3df27501db767bda18f0839a7e8eb1d not found: ID does not exist"
Sep 11 18:15:53.311494 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:53.311469    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"]
Sep 11 18:15:53.313584 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:53.313562    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d59a02-7855856d69-vbdp7"]
Sep 11 18:15:54.687995 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:15:54.687958    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="000c8e16-03b0-406c-b818-355e531789a9" path="/var/lib/kubelet/pods/000c8e16-03b0-406c-b818-355e531789a9/volumes"
Sep 11 18:16:08.424478 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:08.424436    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:16:09.275856 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.275822    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"]
Sep 11 18:16:09.276476 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.276451    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="000c8e16-03b0-406c-b818-355e531789a9" containerName="maas-api"
Sep 11 18:16:09.276476 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.276474    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="000c8e16-03b0-406c-b818-355e531789a9" containerName="maas-api"
Sep 11 18:16:09.276476 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.276482    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23551345-c9aa-4b52-a6e3-cb0500a99f1b" containerName="cleanup"
Sep 11 18:16:09.276704 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.276489    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="23551345-c9aa-4b52-a6e3-cb0500a99f1b" containerName="cleanup"
Sep 11 18:16:09.276704 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.276505    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="495b2b07-9b01-4091-bb0e-0e0015e0b3cd" containerName="main"
Sep 11 18:16:09.276704 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.276511    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="495b2b07-9b01-4091-bb0e-0e0015e0b3cd" containerName="main"
Sep 11 18:16:09.276704 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.276527    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c968d5a2-8342-4163-a73b-d31a9d818c70" containerName="cleanup"
Sep 11 18:16:09.276704 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.276533    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c968d5a2-8342-4163-a73b-d31a9d818c70" containerName="cleanup"
Sep 11 18:16:09.276704 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.276544    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e215f20-a56e-4fb5-a9bd-f037b8819b24" containerName="cleanup"
Sep 11 18:16:09.276704 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.276552    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e215f20-a56e-4fb5-a9bd-f037b8819b24" containerName="cleanup"
Sep 11 18:16:09.276704 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.276628    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="495b2b07-9b01-4091-bb0e-0e0015e0b3cd" containerName="main"
Sep 11 18:16:09.276704 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.276638    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="000c8e16-03b0-406c-b818-355e531789a9" containerName="maas-api"
Sep 11 18:16:09.276704 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.276644    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e215f20-a56e-4fb5-a9bd-f037b8819b24" containerName="cleanup"
Sep 11 18:16:09.276704 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.276657    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c968d5a2-8342-4163-a73b-d31a9d818c70" containerName="cleanup"
Sep 11 18:16:09.276704 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.276664    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="23551345-c9aa-4b52-a6e3-cb0500a99f1b" containerName="cleanup"
Sep 11 18:16:09.281202 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.281182    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"
Sep 11 18:16:09.284319 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.284299    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-a5bbf6-serving-cert\""
Sep 11 18:16:09.293846 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.293820    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"]
Sep 11 18:16:09.343123 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.343083    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf20c90-7cf9-444e-b424-b525dd782a36-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7\" (UID: \"7bf20c90-7cf9-444e-b424-b525dd782a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"
Sep 11 18:16:09.343123 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.343129    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf20c90-7cf9-444e-b424-b525dd782a36-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7\" (UID: \"7bf20c90-7cf9-444e-b424-b525dd782a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"
Sep 11 18:16:09.343374 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.343149    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snxl6\" (UniqueName: \"kubernetes.io/projected/7bf20c90-7cf9-444e-b424-b525dd782a36-kube-api-access-snxl6\") pod \"maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7\" (UID: \"7bf20c90-7cf9-444e-b424-b525dd782a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"
Sep 11 18:16:09.343374 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.343322    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf20c90-7cf9-444e-b424-b525dd782a36-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7\" (UID: \"7bf20c90-7cf9-444e-b424-b525dd782a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"
Sep 11 18:16:09.444840 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.444806    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf20c90-7cf9-444e-b424-b525dd782a36-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7\" (UID: \"7bf20c90-7cf9-444e-b424-b525dd782a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"
Sep 11 18:16:09.445325 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.444874    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf20c90-7cf9-444e-b424-b525dd782a36-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7\" (UID: \"7bf20c90-7cf9-444e-b424-b525dd782a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"
Sep 11 18:16:09.445325 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.444906    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf20c90-7cf9-444e-b424-b525dd782a36-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7\" (UID: \"7bf20c90-7cf9-444e-b424-b525dd782a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"
Sep 11 18:16:09.445325 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.444925    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snxl6\" (UniqueName: \"kubernetes.io/projected/7bf20c90-7cf9-444e-b424-b525dd782a36-kube-api-access-snxl6\") pod \"maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7\" (UID: \"7bf20c90-7cf9-444e-b424-b525dd782a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"
Sep 11 18:16:09.445325 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:16:09.445046    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-serving-cert: secret "maas-api-e2e-shared-a-w3-a5bbf6-serving-cert" not found
Sep 11 18:16:09.445325 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:16:09.445137    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf20c90-7cf9-444e-b424-b525dd782a36-maas-api-tls podName:7bf20c90-7cf9-444e-b424-b525dd782a36 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:09.94511522 +0000 UTC m=+1327.788350645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7bf20c90-7cf9-444e-b424-b525dd782a36-maas-api-tls") pod "maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7" (UID: "7bf20c90-7cf9-444e-b424-b525dd782a36") : secret "maas-api-e2e-shared-a-w3-a5bbf6-serving-cert" not found
Sep 11 18:16:09.445690 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.445638    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf20c90-7cf9-444e-b424-b525dd782a36-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7\" (UID: \"7bf20c90-7cf9-444e-b424-b525dd782a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"
Sep 11 18:16:09.447704 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.447682    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf20c90-7cf9-444e-b424-b525dd782a36-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7\" (UID: \"7bf20c90-7cf9-444e-b424-b525dd782a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"
Sep 11 18:16:09.454164 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.454144    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snxl6\" (UniqueName: \"kubernetes.io/projected/7bf20c90-7cf9-444e-b424-b525dd782a36-kube-api-access-snxl6\") pod \"maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7\" (UID: \"7bf20c90-7cf9-444e-b424-b525dd782a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"
Sep 11 18:16:09.950469 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.950438    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf20c90-7cf9-444e-b424-b525dd782a36-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7\" (UID: \"7bf20c90-7cf9-444e-b424-b525dd782a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"
Sep 11 18:16:09.953047 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:09.953014    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf20c90-7cf9-444e-b424-b525dd782a36-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7\" (UID: \"7bf20c90-7cf9-444e-b424-b525dd782a36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"
Sep 11 18:16:10.195608 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:10.195570    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"
Sep 11 18:16:10.473280 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:10.473244    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"]
Sep 11 18:16:10.474688 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:16:10.474664    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf20c90_7cf9_444e_b424_b525dd782a36.slice/crio-61177476d1a6fb0dc3a2e6828aa81ae825df3327ed6e2e6b1a011f82d47e8a23 WatchSource:0}: Error finding container 61177476d1a6fb0dc3a2e6828aa81ae825df3327ed6e2e6b1a011f82d47e8a23: Status 404 returned error can't find the container with id 61177476d1a6fb0dc3a2e6828aa81ae825df3327ed6e2e6b1a011f82d47e8a23
Sep 11 18:16:10.476527 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:10.476507    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:16:11.388090 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:11.388050    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7" event={"ID":"7bf20c90-7cf9-444e-b424-b525dd782a36","Type":"ContainerStarted","Data":"cd78d3746fcd9e95eb3f8868679895deb27009d7a9a6782fc11686d4eaf5e27e"}
Sep 11 18:16:11.388292 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:11.388117    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"
Sep 11 18:16:11.388292 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:11.388133    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7" event={"ID":"7bf20c90-7cf9-444e-b424-b525dd782a36","Type":"ContainerStarted","Data":"61177476d1a6fb0dc3a2e6828aa81ae825df3327ed6e2e6b1a011f82d47e8a23"}
Sep 11 18:16:11.410782 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:11.410713    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7" podStartSLOduration=1.973120754 podStartE2EDuration="2.410696235s" totalImagesPullingTime="437.575481ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:11.408940683 +0000 UTC m=+1329.252176123" watchObservedRunningTime="2026-09-11 18:16:11.410696235 +0000 UTC m=+1329.253931676"
Sep 11 18:16:17.400178 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:17.400143    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"
Sep 11 18:16:37.241555 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:37.241473    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:16:44.655616 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:44.655572    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"]
Sep 11 18:16:44.659279 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:44.659257    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"
Sep 11 18:16:44.662007 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:44.661983    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-5d35e7-serving-cert\""
Sep 11 18:16:44.674296 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:44.674272    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"]
Sep 11 18:16:44.712807 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:44.712774    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg\" (UID: \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"
Sep 11 18:16:44.712807 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:44.712814    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2glfj\" (UniqueName: \"kubernetes.io/projected/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-kube-api-access-2glfj\") pod \"maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg\" (UID: \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"
Sep 11 18:16:44.713029 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:44.712892    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg\" (UID: \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"
Sep 11 18:16:44.713029 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:44.712937    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg\" (UID: \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"
Sep 11 18:16:44.814212 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:44.814174    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg\" (UID: \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"
Sep 11 18:16:44.814212 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:44.814216    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2glfj\" (UniqueName: \"kubernetes.io/projected/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-kube-api-access-2glfj\") pod \"maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg\" (UID: \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"
Sep 11 18:16:44.814502 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:44.814243    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg\" (UID: \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"
Sep 11 18:16:44.814502 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:44.814271    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg\" (UID: \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"
Sep 11 18:16:44.814502 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:16:44.814345    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-serving-cert: secret "maas-api-e2e-shared-b-w3-5d35e7-serving-cert" not found
Sep 11 18:16:44.814502 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:16:44.814434    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-maas-api-tls podName:5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a nodeName:}" failed. No retries permitted until 2026-09-11 18:16:45.314414085 +0000 UTC m=+1363.157649518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-maas-api-tls") pod "maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg" (UID: "5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a") : secret "maas-api-e2e-shared-b-w3-5d35e7-serving-cert" not found
Sep 11 18:16:44.814995 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:44.814974    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg\" (UID: \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"
Sep 11 18:16:44.816930 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:44.816907    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg\" (UID: \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"
Sep 11 18:16:44.823550 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:44.823525    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2glfj\" (UniqueName: \"kubernetes.io/projected/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-kube-api-access-2glfj\") pod \"maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg\" (UID: \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"
Sep 11 18:16:45.319280 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:45.319245    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg\" (UID: \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"
Sep 11 18:16:45.321830 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:45.321803    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg\" (UID: \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"
Sep 11 18:16:45.571441 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:45.571347    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"
Sep 11 18:16:45.883617 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:45.883594    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"]
Sep 11 18:16:45.885370 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:16:45.885343    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0b4380_1a7d_4e3d_ab98_80e5dd96a00a.slice/crio-545205c7f00ef911907b329432f432383f1ac8097e34227757b74fb94dc9d960 WatchSource:0}: Error finding container 545205c7f00ef911907b329432f432383f1ac8097e34227757b74fb94dc9d960: Status 404 returned error can't find the container with id 545205c7f00ef911907b329432f432383f1ac8097e34227757b74fb94dc9d960
Sep 11 18:16:46.588365 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:46.588277    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg" event={"ID":"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a","Type":"ContainerStarted","Data":"39dbb0679e188bf46709bf94b6e4dccfd77ad96241637ab2ab26f036003e1dea"}
Sep 11 18:16:46.588365 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:46.588345    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg" event={"ID":"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a","Type":"ContainerStarted","Data":"545205c7f00ef911907b329432f432383f1ac8097e34227757b74fb94dc9d960"}
Sep 11 18:16:46.588545 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:46.588463    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"
Sep 11 18:16:46.614119 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:46.614072    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg" podStartSLOduration=2.277246637 podStartE2EDuration="2.614056971s" totalImagesPullingTime="336.810334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:46.61246597 +0000 UTC m=+1364.455701412" watchObservedRunningTime="2026-09-11 18:16:46.614056971 +0000 UTC m=+1364.457292443"
Sep 11 18:16:52.599840 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:16:52.599811    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"
Sep 11 18:17:18.031444 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:18.031408    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:17:40.504764 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:40.504708    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"]
Sep 11 18:17:42.059462 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.059431    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"]
Sep 11 18:17:42.059903 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.059615    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw" podUID="b1ddce6d-1864-4060-a78e-9660801cd906" containerName="maas-api" containerID="cri-o://8ee1b038e80ba8552c448cf71137260b3c1a500babc0b58a29208ae67b168153" gracePeriod=30
Sep 11 18:17:42.326343 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.326277    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"
Sep 11 18:17:42.420619 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.420588    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1ddce6d-1864-4060-a78e-9660801cd906-maas-api-tls\") pod \"b1ddce6d-1864-4060-a78e-9660801cd906\" (UID: \"b1ddce6d-1864-4060-a78e-9660801cd906\") "
Sep 11 18:17:42.420812 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.420706    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ddce6d-1864-4060-a78e-9660801cd906-openshift-service-ca\") pod \"b1ddce6d-1864-4060-a78e-9660801cd906\" (UID: \"b1ddce6d-1864-4060-a78e-9660801cd906\") "
Sep 11 18:17:42.420812 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.420741    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmhv4\" (UniqueName: \"kubernetes.io/projected/b1ddce6d-1864-4060-a78e-9660801cd906-kube-api-access-gmhv4\") pod \"b1ddce6d-1864-4060-a78e-9660801cd906\" (UID: \"b1ddce6d-1864-4060-a78e-9660801cd906\") "
Sep 11 18:17:42.420899 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.420812    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1ddce6d-1864-4060-a78e-9660801cd906-metrics-certs\") pod \"b1ddce6d-1864-4060-a78e-9660801cd906\" (UID: \"b1ddce6d-1864-4060-a78e-9660801cd906\") "
Sep 11 18:17:42.421148 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.421113    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ddce6d-1864-4060-a78e-9660801cd906-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b1ddce6d-1864-4060-a78e-9660801cd906" (UID: "b1ddce6d-1864-4060-a78e-9660801cd906"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:17:42.421307 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.421215    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ddce6d-1864-4060-a78e-9660801cd906-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:17:42.423083 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.423057    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ddce6d-1864-4060-a78e-9660801cd906-kube-api-access-gmhv4" (OuterVolumeSpecName: "kube-api-access-gmhv4") pod "b1ddce6d-1864-4060-a78e-9660801cd906" (UID: "b1ddce6d-1864-4060-a78e-9660801cd906"). InnerVolumeSpecName "kube-api-access-gmhv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:17:42.423169 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.423085    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1ddce6d-1864-4060-a78e-9660801cd906-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b1ddce6d-1864-4060-a78e-9660801cd906" (UID: "b1ddce6d-1864-4060-a78e-9660801cd906"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:17:42.423169 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.423052    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1ddce6d-1864-4060-a78e-9660801cd906-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b1ddce6d-1864-4060-a78e-9660801cd906" (UID: "b1ddce6d-1864-4060-a78e-9660801cd906"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:17:42.522055 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.522011    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1ddce6d-1864-4060-a78e-9660801cd906-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:17:42.522055 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.522047    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmhv4\" (UniqueName: \"kubernetes.io/projected/b1ddce6d-1864-4060-a78e-9660801cd906-kube-api-access-gmhv4\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:17:42.522055 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.522057    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1ddce6d-1864-4060-a78e-9660801cd906-metrics-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:17:42.894006 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.893969    2560 generic.go:356] "Generic (PLEG): container finished" podID="b1ddce6d-1864-4060-a78e-9660801cd906" containerID="8ee1b038e80ba8552c448cf71137260b3c1a500babc0b58a29208ae67b168153" exitCode=0
Sep 11 18:17:42.894206 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.894032    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"
Sep 11 18:17:42.894206 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.894052    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw" event={"ID":"b1ddce6d-1864-4060-a78e-9660801cd906","Type":"ContainerDied","Data":"8ee1b038e80ba8552c448cf71137260b3c1a500babc0b58a29208ae67b168153"}
Sep 11 18:17:42.894206 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.894106    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw" event={"ID":"b1ddce6d-1864-4060-a78e-9660801cd906","Type":"ContainerDied","Data":"56ea31ba98d2c2d78ab3d432d977e2a0bef353c995e1aa68005231ccd3394192"}
Sep 11 18:17:42.894206 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.894129    2560 scope.go:117] "RemoveContainer" containerID="8ee1b038e80ba8552c448cf71137260b3c1a500babc0b58a29208ae67b168153"
Sep 11 18:17:42.903521 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.903504    2560 scope.go:117] "RemoveContainer" containerID="8ee1b038e80ba8552c448cf71137260b3c1a500babc0b58a29208ae67b168153"
Sep 11 18:17:42.903840 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:17:42.903815    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ee1b038e80ba8552c448cf71137260b3c1a500babc0b58a29208ae67b168153\": container with ID starting with 8ee1b038e80ba8552c448cf71137260b3c1a500babc0b58a29208ae67b168153 not found: ID does not exist" containerID="8ee1b038e80ba8552c448cf71137260b3c1a500babc0b58a29208ae67b168153"
Sep 11 18:17:42.903958 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.903847    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ee1b038e80ba8552c448cf71137260b3c1a500babc0b58a29208ae67b168153"} err="failed to get container status \"8ee1b038e80ba8552c448cf71137260b3c1a500babc0b58a29208ae67b168153\": rpc error: code = NotFound desc = could not find container \"8ee1b038e80ba8552c448cf71137260b3c1a500babc0b58a29208ae67b168153\": container with ID starting with 8ee1b038e80ba8552c448cf71137260b3c1a500babc0b58a29208ae67b168153 not found: ID does not exist"
Sep 11 18:17:42.928099 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.928071    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"]
Sep 11 18:17:42.935135 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:42.935110    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3a652320-5d4d4fcf79-4fpcw"]
Sep 11 18:17:44.688059 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:44.688023    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1ddce6d-1864-4060-a78e-9660801cd906" path="/var/lib/kubelet/pods/b1ddce6d-1864-4060-a78e-9660801cd906/volumes"
Sep 11 18:17:55.524616 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.524506    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh" podUID="1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f" containerName="main" containerID="cri-o://0373fcf61969aa2ad810799c2f50f1e5bc8473f72a885094e42b320e83991552" gracePeriod=45
Sep 11 18:17:55.790262 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.790200    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:17:55.960864 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.960823    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-home\") pod \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") "
Sep 11 18:17:55.961047 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.960936    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-model-cache\") pod \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") "
Sep 11 18:17:55.961047 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.960978    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-tls-certs\") pod \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") "
Sep 11 18:17:55.961047 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.961026    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-tmp-dir\") pod \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") "
Sep 11 18:17:55.961188 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.961055    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k22g9\" (UniqueName: \"kubernetes.io/projected/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-kube-api-access-k22g9\") pod \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") "
Sep 11 18:17:55.961188 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.961082    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-dshm\") pod \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\" (UID: \"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f\") "
Sep 11 18:17:55.961188 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.961146    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-home" (OuterVolumeSpecName: "home") pod "1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f" (UID: "1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:17:55.961188 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.961160    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-model-cache" (OuterVolumeSpecName: "model-cache") pod "1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f" (UID: "1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:17:55.961398 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.961337    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f" (UID: "1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:17:55.961457 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.961434    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-model-cache\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:17:55.961457 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.961447    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-tmp-dir\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:17:55.961457 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.961455    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-home\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:17:55.963444 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.963419    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f" (UID: "1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:17:55.963566 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.963462    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-dshm" (OuterVolumeSpecName: "dshm") pod "1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f" (UID: "1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:17:55.963566 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.963518    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-kube-api-access-k22g9" (OuterVolumeSpecName: "kube-api-access-k22g9") pod "1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f" (UID: "1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f"). InnerVolumeSpecName "kube-api-access-k22g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:17:55.973524 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.973495    2560 generic.go:356] "Generic (PLEG): container finished" podID="1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f" containerID="0373fcf61969aa2ad810799c2f50f1e5bc8473f72a885094e42b320e83991552" exitCode=0
Sep 11 18:17:55.973660 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.973543    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"
Sep 11 18:17:55.973660 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.973578    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh" event={"ID":"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f","Type":"ContainerDied","Data":"0373fcf61969aa2ad810799c2f50f1e5bc8473f72a885094e42b320e83991552"}
Sep 11 18:17:55.973660 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.973631    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh" event={"ID":"1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f","Type":"ContainerDied","Data":"a3c5f68c4d06c4bb5311f1989c7f25ee31ce786ee83f43f1b95753fe948afff4"}
Sep 11 18:17:55.973660 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.973655    2560 scope.go:117] "RemoveContainer" containerID="0373fcf61969aa2ad810799c2f50f1e5bc8473f72a885094e42b320e83991552"
Sep 11 18:17:55.985306 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.985284    2560 scope.go:117] "RemoveContainer" containerID="0373fcf61969aa2ad810799c2f50f1e5bc8473f72a885094e42b320e83991552"
Sep 11 18:17:55.985605 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:17:55.985589    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0373fcf61969aa2ad810799c2f50f1e5bc8473f72a885094e42b320e83991552\": container with ID starting with 0373fcf61969aa2ad810799c2f50f1e5bc8473f72a885094e42b320e83991552 not found: ID does not exist" containerID="0373fcf61969aa2ad810799c2f50f1e5bc8473f72a885094e42b320e83991552"
Sep 11 18:17:55.985656 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:55.985613    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0373fcf61969aa2ad810799c2f50f1e5bc8473f72a885094e42b320e83991552"} err="failed to get container status \"0373fcf61969aa2ad810799c2f50f1e5bc8473f72a885094e42b320e83991552\": rpc error: code = NotFound desc = could not find container \"0373fcf61969aa2ad810799c2f50f1e5bc8473f72a885094e42b320e83991552\": container with ID starting with 0373fcf61969aa2ad810799c2f50f1e5bc8473f72a885094e42b320e83991552 not found: ID does not exist"
Sep 11 18:17:56.001881 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:56.001854    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"]
Sep 11 18:17:56.005244 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:56.005220    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/e2e-conflict-a-74f751c5-kserve-7b9dccb85f-xrhkh"]
Sep 11 18:17:56.062708 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:56.062626    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-tls-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:17:56.062708 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:56.062654    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k22g9\" (UniqueName: \"kubernetes.io/projected/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-kube-api-access-k22g9\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:17:56.062708 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:56.062665    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f-dshm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:17:56.689115 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:56.689085    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f" path="/var/lib/kubelet/pods/1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f/volumes"
Sep 11 18:17:58.218019 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.217986    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"]
Sep 11 18:17:58.218416 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.218404    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f" containerName="main"
Sep 11 18:17:58.218464 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.218418    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f" containerName="main"
Sep 11 18:17:58.218464 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.218432    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1ddce6d-1864-4060-a78e-9660801cd906" containerName="maas-api"
Sep 11 18:17:58.218464 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.218437    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1ddce6d-1864-4060-a78e-9660801cd906" containerName="maas-api"
Sep 11 18:17:58.218560 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.218504    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1ddce6d-1864-4060-a78e-9660801cd906" containerName="maas-api"
Sep 11 18:17:58.218560 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.218512    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c65b43a-736f-4ca6-8bc5-7e0f14eecd8f" containerName="main"
Sep 11 18:17:58.223224 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.223206    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:17:58.226016 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.225996    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4e0b0b\"/\"distinct-w4-4e0b0b-kserve-self-signed-certs\""
Sep 11 18:17:58.234900 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.234880    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"]
Sep 11 18:17:58.386099 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.386060    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-home\") pod \"distinct-w4-4e0b0b-kserve-878d66548-dj9dn\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:17:58.386099 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.386097    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-model-cache\") pod \"distinct-w4-4e0b0b-kserve-878d66548-dj9dn\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:17:58.386325 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.386173    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-dshm\") pod \"distinct-w4-4e0b0b-kserve-878d66548-dj9dn\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:17:58.386325 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.386225    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prhcd\" (UniqueName: \"kubernetes.io/projected/828634bf-7193-4d57-aabf-98a5102993eb-kube-api-access-prhcd\") pod \"distinct-w4-4e0b0b-kserve-878d66548-dj9dn\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:17:58.386325 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.386277    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828634bf-7193-4d57-aabf-98a5102993eb-tls-certs\") pod \"distinct-w4-4e0b0b-kserve-878d66548-dj9dn\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:17:58.386437 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.386334    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-tmp-dir\") pod \"distinct-w4-4e0b0b-kserve-878d66548-dj9dn\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:17:58.487813 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.487680    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-dshm\") pod \"distinct-w4-4e0b0b-kserve-878d66548-dj9dn\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:17:58.487813 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.487778    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prhcd\" (UniqueName: \"kubernetes.io/projected/828634bf-7193-4d57-aabf-98a5102993eb-kube-api-access-prhcd\") pod \"distinct-w4-4e0b0b-kserve-878d66548-dj9dn\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:17:58.488063 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.487875    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828634bf-7193-4d57-aabf-98a5102993eb-tls-certs\") pod \"distinct-w4-4e0b0b-kserve-878d66548-dj9dn\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:17:58.488063 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.487926    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-tmp-dir\") pod \"distinct-w4-4e0b0b-kserve-878d66548-dj9dn\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:17:58.488063 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.487957    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-home\") pod \"distinct-w4-4e0b0b-kserve-878d66548-dj9dn\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:17:58.488063 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.487985    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-model-cache\") pod \"distinct-w4-4e0b0b-kserve-878d66548-dj9dn\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:17:58.488307 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.488276    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-tmp-dir\") pod \"distinct-w4-4e0b0b-kserve-878d66548-dj9dn\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:17:58.488307 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.488303    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-home\") pod \"distinct-w4-4e0b0b-kserve-878d66548-dj9dn\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:17:58.488479 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.488392    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-model-cache\") pod \"distinct-w4-4e0b0b-kserve-878d66548-dj9dn\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:17:58.490236 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.490212    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-dshm\") pod \"distinct-w4-4e0b0b-kserve-878d66548-dj9dn\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:17:58.490471 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.490453    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828634bf-7193-4d57-aabf-98a5102993eb-tls-certs\") pod \"distinct-w4-4e0b0b-kserve-878d66548-dj9dn\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:17:58.497691 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.497669    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prhcd\" (UniqueName: \"kubernetes.io/projected/828634bf-7193-4d57-aabf-98a5102993eb-kube-api-access-prhcd\") pod \"distinct-w4-4e0b0b-kserve-878d66548-dj9dn\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") " pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:17:58.532478 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.532445    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:17:58.824965 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.824941    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"]
Sep 11 18:17:58.827499 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:17:58.827472    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod828634bf_7193_4d57_aabf_98a5102993eb.slice/crio-2d09c3f23312ac95c1da112628b69ec488cd300634b80d1d39f4f44f4e04e2c6 WatchSource:0}: Error finding container 2d09c3f23312ac95c1da112628b69ec488cd300634b80d1d39f4f44f4e04e2c6: Status 404 returned error can't find the container with id 2d09c3f23312ac95c1da112628b69ec488cd300634b80d1d39f4f44f4e04e2c6
Sep 11 18:17:58.998247 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.998210    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn" event={"ID":"828634bf-7193-4d57-aabf-98a5102993eb","Type":"ContainerStarted","Data":"78dce24db6a5a4bd05b02db769e878e9d9d59ab96bfa9b54283a20342e3a0363"}
Sep 11 18:17:58.998427 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:58.998262    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn" event={"ID":"828634bf-7193-4d57-aabf-98a5102993eb","Type":"ContainerStarted","Data":"2d09c3f23312ac95c1da112628b69ec488cd300634b80d1d39f4f44f4e04e2c6"}
Sep 11 18:17:59.034332 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:17:59.034268    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn" podStartSLOduration=1.034250926 podStartE2EDuration="1.034250926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:59.03221276 +0000 UTC m=+1436.875448201" watchObservedRunningTime="2026-09-11 18:17:59.034250926 +0000 UTC m=+1436.877486422"
Sep 11 18:18:02.151210 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:02.151172    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:18:03.221506 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.221475    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"]
Sep 11 18:18:03.226414 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.226398    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:03.229670 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.229635    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5d35e7\"/\"openshift-service-ca.crt\""
Sep 11 18:18:03.229670 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.229640    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5d35e7\"/\"auth-test-model-5d35e7-kserve-self-signed-certs\""
Sep 11 18:18:03.230834 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.230813    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5d35e7\"/\"default-dockercfg-bktdb\""
Sep 11 18:18:03.230971 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.230844    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5d35e7\"/\"kube-root-ca.crt\""
Sep 11 18:18:03.238002 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.237973    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"]
Sep 11 18:18:03.338180 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.338142    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-model-cache\") pod \"auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:03.338393 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.338204    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96aa235d-5d81-4486-a2b9-0ca3152bddf7-tls-certs\") pod \"auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:03.338393 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.338346    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-home\") pod \"auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:03.338506 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.338394    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn2t8\" (UniqueName: \"kubernetes.io/projected/96aa235d-5d81-4486-a2b9-0ca3152bddf7-kube-api-access-kn2t8\") pod \"auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:03.338506 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.338469    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-tmp-dir\") pod \"auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:03.338506 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.338497    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-dshm\") pod \"auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:03.439210 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.439171    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn2t8\" (UniqueName: \"kubernetes.io/projected/96aa235d-5d81-4486-a2b9-0ca3152bddf7-kube-api-access-kn2t8\") pod \"auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:03.439405 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.439228    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-tmp-dir\") pod \"auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:03.439405 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.439352    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-dshm\") pod \"auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:03.439523 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.439431    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-model-cache\") pod \"auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:03.439523 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.439512    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96aa235d-5d81-4486-a2b9-0ca3152bddf7-tls-certs\") pod \"auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:03.439684 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.439549    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-tmp-dir\") pod \"auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:03.439684 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.439635    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-home\") pod \"auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:03.439848 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.439745    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-model-cache\") pod \"auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:03.439909 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.439883    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-home\") pod \"auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:03.441706 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.441685    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-dshm\") pod \"auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:03.442031 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.442014    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96aa235d-5d81-4486-a2b9-0ca3152bddf7-tls-certs\") pod \"auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:03.448477 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.448453    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn2t8\" (UniqueName: \"kubernetes.io/projected/96aa235d-5d81-4486-a2b9-0ca3152bddf7-kube-api-access-kn2t8\") pod \"auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") " pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:03.537007 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.536906    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:03.818682 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:03.818659    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"]
Sep 11 18:18:03.820557 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:18:03.820532    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96aa235d_5d81_4486_a2b9_0ca3152bddf7.slice/crio-6a5c1e05751f56e856017900f0dad3f524b679274122819acba818d2e2b5578b WatchSource:0}: Error finding container 6a5c1e05751f56e856017900f0dad3f524b679274122819acba818d2e2b5578b: Status 404 returned error can't find the container with id 6a5c1e05751f56e856017900f0dad3f524b679274122819acba818d2e2b5578b
Sep 11 18:18:04.029323 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:04.029290    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp" event={"ID":"96aa235d-5d81-4486-a2b9-0ca3152bddf7","Type":"ContainerStarted","Data":"84648309bd047d1cb0ab905971855cc6ac4652437397cce95adb43e033281880"}
Sep 11 18:18:04.029516 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:04.029335    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp" event={"ID":"96aa235d-5d81-4486-a2b9-0ca3152bddf7","Type":"ContainerStarted","Data":"6a5c1e05751f56e856017900f0dad3f524b679274122819acba818d2e2b5578b"}
Sep 11 18:18:04.052743 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:04.052688    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp" podStartSLOduration=1.052671111 podStartE2EDuration="1.052671111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:04.049964447 +0000 UTC m=+1441.893199888" watchObservedRunningTime="2026-09-11 18:18:04.052671111 +0000 UTC m=+1441.895906548"
Sep 11 18:18:08.532576 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:08.532539    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:18:08.533016 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:08.532634    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:18:08.544978 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:08.544956    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:18:09.068728 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:09.068697    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:18:13.538073 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:13.538035    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:13.538575 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:13.538094    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:13.550703 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:13.550679    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:14.097836 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:14.097805    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:18:37.421381 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:18:37.421344    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:19:02.713965 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:02.713937    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/2.log"
Sep 11 18:19:02.727224 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:02.727198    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/2.log"
Sep 11 18:19:10.319704 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:10.319665    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:19:48.128859 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:48.128815    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:19:50.298337 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:50.298303    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"]
Sep 11 18:19:50.302703 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:50.302673    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"
Sep 11 18:19:50.305995 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:50.305967    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-684bbd-serving-cert\""
Sep 11 18:19:50.315883 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:50.315846    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"]
Sep 11 18:19:50.339629 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:50.339601    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-maas-api-tls\") pod \"maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj\" (UID: \"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"
Sep 11 18:19:50.339808 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:50.339641    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-metrics-certs\") pod \"maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj\" (UID: \"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"
Sep 11 18:19:50.339808 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:50.339704    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-openshift-service-ca\") pod \"maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj\" (UID: \"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"
Sep 11 18:19:50.339896 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:50.339830    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88r8w\" (UniqueName: \"kubernetes.io/projected/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-kube-api-access-88r8w\") pod \"maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj\" (UID: \"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"
Sep 11 18:19:50.441295 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:50.441260    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-maas-api-tls\") pod \"maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj\" (UID: \"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"
Sep 11 18:19:50.441295 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:50.441300    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-metrics-certs\") pod \"maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj\" (UID: \"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"
Sep 11 18:19:50.441564 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:50.441331    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-openshift-service-ca\") pod \"maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj\" (UID: \"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"
Sep 11 18:19:50.441564 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:50.441400    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88r8w\" (UniqueName: \"kubernetes.io/projected/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-kube-api-access-88r8w\") pod \"maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj\" (UID: \"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"
Sep 11 18:19:50.442121 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:50.442097    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-openshift-service-ca\") pod \"maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj\" (UID: \"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"
Sep 11 18:19:50.444153 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:50.444124    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-metrics-certs\") pod \"maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj\" (UID: \"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"
Sep 11 18:19:50.444242 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:50.444124    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-maas-api-tls\") pod \"maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj\" (UID: \"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"
Sep 11 18:19:50.450516 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:50.450497    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88r8w\" (UniqueName: \"kubernetes.io/projected/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-kube-api-access-88r8w\") pod \"maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj\" (UID: \"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"
Sep 11 18:19:50.615011 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:50.614923    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"
Sep 11 18:19:50.893765 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:50.893721    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"]
Sep 11 18:19:50.895023 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:19:50.894993    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aab02fb_d668_4ef8_8a3d_b1b5c85eb2a5.slice/crio-050598612a5ffba2f2d0ff2ab9c4beebff3e5d5e046e4490918a3f618b8c0a80 WatchSource:0}: Error finding container 050598612a5ffba2f2d0ff2ab9c4beebff3e5d5e046e4490918a3f618b8c0a80: Status 404 returned error can't find the container with id 050598612a5ffba2f2d0ff2ab9c4beebff3e5d5e046e4490918a3f618b8c0a80
Sep 11 18:19:51.622319 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:51.622283    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj" event={"ID":"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5","Type":"ContainerStarted","Data":"c1c56b628bfef17d8943bfa863a33b2688bf12f58bdc2cf3873f9afc7cb0917b"}
Sep 11 18:19:51.622732 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:51.622375    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj" event={"ID":"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5","Type":"ContainerStarted","Data":"050598612a5ffba2f2d0ff2ab9c4beebff3e5d5e046e4490918a3f618b8c0a80"}
Sep 11 18:19:51.622732 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:51.622507    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"
Sep 11 18:19:51.642672 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:51.642620    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj" podStartSLOduration=1.005418583 podStartE2EDuration="1.642602779s" totalImagesPullingTime="637.184196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:19:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:19:51.641068895 +0000 UTC m=+1549.484304363" watchObservedRunningTime="2026-09-11 18:19:51.642602779 +0000 UTC m=+1549.485838219"
Sep 11 18:19:57.633929 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:19:57.633898    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"
Sep 11 18:20:20.824095 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:20.824056    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"]
Sep 11 18:20:20.828595 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:20.828575    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"
Sep 11 18:20:20.831341 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:20.831316    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-794d1c-serving-cert\""
Sep 11 18:20:20.844833 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:20.844793    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"]
Sep 11 18:20:20.871894 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:20.871864    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-maas-api-tls\") pod \"maas-api-e2e-api-b-794d1c-567d997df5-vpff7\" (UID: \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"
Sep 11 18:20:20.872071 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:20.871909    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr99z\" (UniqueName: \"kubernetes.io/projected/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-kube-api-access-kr99z\") pod \"maas-api-e2e-api-b-794d1c-567d997df5-vpff7\" (UID: \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"
Sep 11 18:20:20.872071 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:20.871986    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-openshift-service-ca\") pod \"maas-api-e2e-api-b-794d1c-567d997df5-vpff7\" (UID: \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"
Sep 11 18:20:20.872071 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:20.872030    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-metrics-certs\") pod \"maas-api-e2e-api-b-794d1c-567d997df5-vpff7\" (UID: \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"
Sep 11 18:20:20.973278 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:20.973240    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-maas-api-tls\") pod \"maas-api-e2e-api-b-794d1c-567d997df5-vpff7\" (UID: \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"
Sep 11 18:20:20.973470 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:20.973290    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr99z\" (UniqueName: \"kubernetes.io/projected/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-kube-api-access-kr99z\") pod \"maas-api-e2e-api-b-794d1c-567d997df5-vpff7\" (UID: \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"
Sep 11 18:20:20.973470 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:20.973337    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-openshift-service-ca\") pod \"maas-api-e2e-api-b-794d1c-567d997df5-vpff7\" (UID: \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"
Sep 11 18:20:20.973470 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:20.973368    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-metrics-certs\") pod \"maas-api-e2e-api-b-794d1c-567d997df5-vpff7\" (UID: \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"
Sep 11 18:20:20.973470 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:20:20.973410    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-serving-cert: secret "maas-api-e2e-api-b-794d1c-serving-cert" not found
Sep 11 18:20:20.973634 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:20:20.973485    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-maas-api-tls podName:4727935a-3579-4e9c-a9ba-4d0dec1cf07b nodeName:}" failed. No retries permitted until 2026-09-11 18:20:21.473468669 +0000 UTC m=+1579.316704087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-maas-api-tls") pod "maas-api-e2e-api-b-794d1c-567d997df5-vpff7" (UID: "4727935a-3579-4e9c-a9ba-4d0dec1cf07b") : secret "maas-api-e2e-api-b-794d1c-serving-cert" not found
Sep 11 18:20:20.974108 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:20.974090    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-openshift-service-ca\") pod \"maas-api-e2e-api-b-794d1c-567d997df5-vpff7\" (UID: \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"
Sep 11 18:20:20.976076 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:20.976058    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-metrics-certs\") pod \"maas-api-e2e-api-b-794d1c-567d997df5-vpff7\" (UID: \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"
Sep 11 18:20:20.982582 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:20.982564    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr99z\" (UniqueName: \"kubernetes.io/projected/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-kube-api-access-kr99z\") pod \"maas-api-e2e-api-b-794d1c-567d997df5-vpff7\" (UID: \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"
Sep 11 18:20:21.478145 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:21.478112    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-maas-api-tls\") pod \"maas-api-e2e-api-b-794d1c-567d997df5-vpff7\" (UID: \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"
Sep 11 18:20:21.480744 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:21.480715    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-maas-api-tls\") pod \"maas-api-e2e-api-b-794d1c-567d997df5-vpff7\" (UID: \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"
Sep 11 18:20:21.749968 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:21.749866    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"
Sep 11 18:20:22.027672 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:22.027642    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"]
Sep 11 18:20:22.029565 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:20:22.029535    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4727935a_3579_4e9c_a9ba_4d0dec1cf07b.slice/crio-0818e99473985a1be09ddda7c381b5948a5bd2fb37943f2c46899430fa1c89e5 WatchSource:0}: Error finding container 0818e99473985a1be09ddda7c381b5948a5bd2fb37943f2c46899430fa1c89e5: Status 404 returned error can't find the container with id 0818e99473985a1be09ddda7c381b5948a5bd2fb37943f2c46899430fa1c89e5
Sep 11 18:20:22.788618 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:22.788580    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7" event={"ID":"4727935a-3579-4e9c-a9ba-4d0dec1cf07b","Type":"ContainerStarted","Data":"43a0078930de3e727e680c62d1ebe5918ed7c795219c02a1491289b6d32a91c0"}
Sep 11 18:20:22.788874 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:22.788646    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7" event={"ID":"4727935a-3579-4e9c-a9ba-4d0dec1cf07b","Type":"ContainerStarted","Data":"0818e99473985a1be09ddda7c381b5948a5bd2fb37943f2c46899430fa1c89e5"}
Sep 11 18:20:22.788874 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:22.788684    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"
Sep 11 18:20:22.811473 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:22.811415    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7" podStartSLOduration=2.195905612 podStartE2EDuration="2.811398091s" totalImagesPullingTime="615.492479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:20:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:20:22.809149978 +0000 UTC m=+1580.652385418" watchObservedRunningTime="2026-09-11 18:20:22.811398091 +0000 UTC m=+1580.654633532"
Sep 11 18:20:28.800044 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:28.800017    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"
Sep 11 18:20:37.917137 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:37.917103    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:20:50.583285 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:50.583247    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"]
Sep 11 18:20:50.607824 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:50.607651    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"]
Sep 11 18:20:50.620872 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:20:50.620838    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"]
Sep 11 18:21:05.604029 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:05.603976    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9" podUID="82d24234-b1d5-4ec4-abba-aac23e2176f2" containerName="main" containerID="cri-o://dbfa6f3ab227e214d6d34b32f863552489431b1f47b62a35cf2997c1b2590da1" gracePeriod=45
Sep 11 18:21:05.628568 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:05.628531    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd" podUID="05acfa53-02f8-4afc-8159-a495849827da" containerName="main" containerID="cri-o://e081b62c8db910a305fdb776f5650dbf69396a1ded0217b0e802e43f53895c9f" gracePeriod=45
Sep 11 18:21:05.644597 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:05.644558    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl" podUID="35248822-aefa-4202-a5d2-da92d6b5fc51" containerName="main" containerID="cri-o://b6c223beda3975f9845426bb5b062ff2f7dcaeb6c83b99ff66e350d171d0dc35" gracePeriod=45
Sep 11 18:21:05.918873 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:05.918844    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:21:05.949470 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:05.949438    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:21:05.953736 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:05.953711    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:21:06.029951 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.029911    2560 generic.go:356] "Generic (PLEG): container finished" podID="05acfa53-02f8-4afc-8159-a495849827da" containerID="e081b62c8db910a305fdb776f5650dbf69396a1ded0217b0e802e43f53895c9f" exitCode=0
Sep 11 18:21:06.030119 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.030011    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"
Sep 11 18:21:06.030119 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.030043    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd" event={"ID":"05acfa53-02f8-4afc-8159-a495849827da","Type":"ContainerDied","Data":"e081b62c8db910a305fdb776f5650dbf69396a1ded0217b0e802e43f53895c9f"}
Sep 11 18:21:06.030119 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.030085    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd" event={"ID":"05acfa53-02f8-4afc-8159-a495849827da","Type":"ContainerDied","Data":"1d1635426586b5cb4aafd7e2d96ef58ac623c28d6ff42fd44ba88979536809dd"}
Sep 11 18:21:06.030119 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.030110    2560 scope.go:117] "RemoveContainer" containerID="e081b62c8db910a305fdb776f5650dbf69396a1ded0217b0e802e43f53895c9f"
Sep 11 18:21:06.031648 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.031608    2560 generic.go:356] "Generic (PLEG): container finished" podID="82d24234-b1d5-4ec4-abba-aac23e2176f2" containerID="dbfa6f3ab227e214d6d34b32f863552489431b1f47b62a35cf2997c1b2590da1" exitCode=0
Sep 11 18:21:06.031784 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.031666    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"
Sep 11 18:21:06.031784 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.031705    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9" event={"ID":"82d24234-b1d5-4ec4-abba-aac23e2176f2","Type":"ContainerDied","Data":"dbfa6f3ab227e214d6d34b32f863552489431b1f47b62a35cf2997c1b2590da1"}
Sep 11 18:21:06.031784 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.031743    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9" event={"ID":"82d24234-b1d5-4ec4-abba-aac23e2176f2","Type":"ContainerDied","Data":"297151e6de60b63c3f99254115c9910ba79979741e9d9b8bf3d6ae8c2586c38f"}
Sep 11 18:21:06.033501 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.033477    2560 generic.go:356] "Generic (PLEG): container finished" podID="35248822-aefa-4202-a5d2-da92d6b5fc51" containerID="b6c223beda3975f9845426bb5b062ff2f7dcaeb6c83b99ff66e350d171d0dc35" exitCode=0
Sep 11 18:21:06.033607 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.033524    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"
Sep 11 18:21:06.033607 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.033560    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl" event={"ID":"35248822-aefa-4202-a5d2-da92d6b5fc51","Type":"ContainerDied","Data":"b6c223beda3975f9845426bb5b062ff2f7dcaeb6c83b99ff66e350d171d0dc35"}
Sep 11 18:21:06.033607 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.033597    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl" event={"ID":"35248822-aefa-4202-a5d2-da92d6b5fc51","Type":"ContainerDied","Data":"b1e96305c53881b12fb71c0333f809108eb7052cf737246df7b5399d2181da57"}
Sep 11 18:21:06.040406 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.040389    2560 scope.go:117] "RemoveContainer" containerID="e081b62c8db910a305fdb776f5650dbf69396a1ded0217b0e802e43f53895c9f"
Sep 11 18:21:06.040703 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:21:06.040683    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e081b62c8db910a305fdb776f5650dbf69396a1ded0217b0e802e43f53895c9f\": container with ID starting with e081b62c8db910a305fdb776f5650dbf69396a1ded0217b0e802e43f53895c9f not found: ID does not exist" containerID="e081b62c8db910a305fdb776f5650dbf69396a1ded0217b0e802e43f53895c9f"
Sep 11 18:21:06.040767 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.040711    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e081b62c8db910a305fdb776f5650dbf69396a1ded0217b0e802e43f53895c9f"} err="failed to get container status \"e081b62c8db910a305fdb776f5650dbf69396a1ded0217b0e802e43f53895c9f\": rpc error: code = NotFound desc = could not find container \"e081b62c8db910a305fdb776f5650dbf69396a1ded0217b0e802e43f53895c9f\": container with ID starting with e081b62c8db910a305fdb776f5650dbf69396a1ded0217b0e802e43f53895c9f not found: ID does not exist"
Sep 11 18:21:06.040767 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.040726    2560 scope.go:117] "RemoveContainer" containerID="dbfa6f3ab227e214d6d34b32f863552489431b1f47b62a35cf2997c1b2590da1"
Sep 11 18:21:06.051239 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.051220    2560 scope.go:117] "RemoveContainer" containerID="dbfa6f3ab227e214d6d34b32f863552489431b1f47b62a35cf2997c1b2590da1"
Sep 11 18:21:06.051564 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:21:06.051545    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbfa6f3ab227e214d6d34b32f863552489431b1f47b62a35cf2997c1b2590da1\": container with ID starting with dbfa6f3ab227e214d6d34b32f863552489431b1f47b62a35cf2997c1b2590da1 not found: ID does not exist" containerID="dbfa6f3ab227e214d6d34b32f863552489431b1f47b62a35cf2997c1b2590da1"
Sep 11 18:21:06.051608 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.051572    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbfa6f3ab227e214d6d34b32f863552489431b1f47b62a35cf2997c1b2590da1"} err="failed to get container status \"dbfa6f3ab227e214d6d34b32f863552489431b1f47b62a35cf2997c1b2590da1\": rpc error: code = NotFound desc = could not find container \"dbfa6f3ab227e214d6d34b32f863552489431b1f47b62a35cf2997c1b2590da1\": container with ID starting with dbfa6f3ab227e214d6d34b32f863552489431b1f47b62a35cf2997c1b2590da1 not found: ID does not exist"
Sep 11 18:21:06.051608 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.051590    2560 scope.go:117] "RemoveContainer" containerID="b6c223beda3975f9845426bb5b062ff2f7dcaeb6c83b99ff66e350d171d0dc35"
Sep 11 18:21:06.056918 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.056894    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-dshm\") pod \"82d24234-b1d5-4ec4-abba-aac23e2176f2\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") "
Sep 11 18:21:06.057031 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.056926    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-dshm\") pod \"35248822-aefa-4202-a5d2-da92d6b5fc51\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") "
Sep 11 18:21:06.057031 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.056970    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tbqs\" (UniqueName: \"kubernetes.io/projected/35248822-aefa-4202-a5d2-da92d6b5fc51-kube-api-access-6tbqs\") pod \"35248822-aefa-4202-a5d2-da92d6b5fc51\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") "
Sep 11 18:21:06.057031 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.056985    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-model-cache\") pod \"82d24234-b1d5-4ec4-abba-aac23e2176f2\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") "
Sep 11 18:21:06.057031 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.057011    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxzk4\" (UniqueName: \"kubernetes.io/projected/05acfa53-02f8-4afc-8159-a495849827da-kube-api-access-bxzk4\") pod \"05acfa53-02f8-4afc-8159-a495849827da\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") "
Sep 11 18:21:06.057194 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.057060    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-home\") pod \"05acfa53-02f8-4afc-8159-a495849827da\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") "
Sep 11 18:21:06.057194 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.057082    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05acfa53-02f8-4afc-8159-a495849827da-tls-certs\") pod \"05acfa53-02f8-4afc-8159-a495849827da\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") "
Sep 11 18:21:06.057194 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.057105    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-home\") pod \"35248822-aefa-4202-a5d2-da92d6b5fc51\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") "
Sep 11 18:21:06.057194 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.057139    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82d24234-b1d5-4ec4-abba-aac23e2176f2-tls-certs\") pod \"82d24234-b1d5-4ec4-abba-aac23e2176f2\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") "
Sep 11 18:21:06.057194 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.057171    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-tmp-dir\") pod \"82d24234-b1d5-4ec4-abba-aac23e2176f2\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") "
Sep 11 18:21:06.057451 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.057211    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-model-cache\") pod \"35248822-aefa-4202-a5d2-da92d6b5fc51\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") "
Sep 11 18:21:06.057451 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.057251    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-dshm\") pod \"05acfa53-02f8-4afc-8159-a495849827da\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") "
Sep 11 18:21:06.057451 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.057289    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-model-cache\") pod \"05acfa53-02f8-4afc-8159-a495849827da\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") "
Sep 11 18:21:06.057451 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.057347    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35248822-aefa-4202-a5d2-da92d6b5fc51-tls-certs\") pod \"35248822-aefa-4202-a5d2-da92d6b5fc51\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") "
Sep 11 18:21:06.057833 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.057523    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-home" (OuterVolumeSpecName: "home") pod "35248822-aefa-4202-a5d2-da92d6b5fc51" (UID: "35248822-aefa-4202-a5d2-da92d6b5fc51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:06.057833 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.057533    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "82d24234-b1d5-4ec4-abba-aac23e2176f2" (UID: "82d24234-b1d5-4ec4-abba-aac23e2176f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:06.057833 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.057638    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-model-cache" (OuterVolumeSpecName: "model-cache") pod "35248822-aefa-4202-a5d2-da92d6b5fc51" (UID: "35248822-aefa-4202-a5d2-da92d6b5fc51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:06.057833 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.057797    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-model-cache" (OuterVolumeSpecName: "model-cache") pod "05acfa53-02f8-4afc-8159-a495849827da" (UID: "05acfa53-02f8-4afc-8159-a495849827da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:06.058065 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.057993    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-home\") pod \"82d24234-b1d5-4ec4-abba-aac23e2176f2\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") "
Sep 11 18:21:06.058065 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.057999    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "82d24234-b1d5-4ec4-abba-aac23e2176f2" (UID: "82d24234-b1d5-4ec4-abba-aac23e2176f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:06.058065 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.058044    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-tmp-dir\") pod \"05acfa53-02f8-4afc-8159-a495849827da\" (UID: \"05acfa53-02f8-4afc-8159-a495849827da\") "
Sep 11 18:21:06.058218 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.058071    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjxmt\" (UniqueName: \"kubernetes.io/projected/82d24234-b1d5-4ec4-abba-aac23e2176f2-kube-api-access-gjxmt\") pod \"82d24234-b1d5-4ec4-abba-aac23e2176f2\" (UID: \"82d24234-b1d5-4ec4-abba-aac23e2176f2\") "
Sep 11 18:21:06.058218 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.058118    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-tmp-dir\") pod \"35248822-aefa-4202-a5d2-da92d6b5fc51\" (UID: \"35248822-aefa-4202-a5d2-da92d6b5fc51\") "
Sep 11 18:21:06.058218 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.058186    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-home" (OuterVolumeSpecName: "home") pod "05acfa53-02f8-4afc-8159-a495849827da" (UID: "05acfa53-02f8-4afc-8159-a495849827da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:06.058380 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.058350    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-home" (OuterVolumeSpecName: "home") pod "82d24234-b1d5-4ec4-abba-aac23e2176f2" (UID: "82d24234-b1d5-4ec4-abba-aac23e2176f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:06.058798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.058774    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-home\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.058930 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.058803    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-model-cache\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.058930 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.058817    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-home\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.058930 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.058829    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-home\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.058930 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.058841    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-tmp-dir\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.058930 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.058851    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-model-cache\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.058930 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.058862    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-model-cache\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.058930 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.058898    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "05acfa53-02f8-4afc-8159-a495849827da" (UID: "05acfa53-02f8-4afc-8159-a495849827da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:06.059549 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.059030    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "35248822-aefa-4202-a5d2-da92d6b5fc51" (UID: "35248822-aefa-4202-a5d2-da92d6b5fc51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:06.060280 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.060255    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-dshm" (OuterVolumeSpecName: "dshm") pod "82d24234-b1d5-4ec4-abba-aac23e2176f2" (UID: "82d24234-b1d5-4ec4-abba-aac23e2176f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:06.060377 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.060359    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-dshm" (OuterVolumeSpecName: "dshm") pod "35248822-aefa-4202-a5d2-da92d6b5fc51" (UID: "35248822-aefa-4202-a5d2-da92d6b5fc51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:06.060660 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.060630    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35248822-aefa-4202-a5d2-da92d6b5fc51-kube-api-access-6tbqs" (OuterVolumeSpecName: "kube-api-access-6tbqs") pod "35248822-aefa-4202-a5d2-da92d6b5fc51" (UID: "35248822-aefa-4202-a5d2-da92d6b5fc51"). InnerVolumeSpecName "kube-api-access-6tbqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:21:06.060906 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.060882    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-dshm" (OuterVolumeSpecName: "dshm") pod "05acfa53-02f8-4afc-8159-a495849827da" (UID: "05acfa53-02f8-4afc-8159-a495849827da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:21:06.061428 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.061402    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82d24234-b1d5-4ec4-abba-aac23e2176f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82d24234-b1d5-4ec4-abba-aac23e2176f2" (UID: "82d24234-b1d5-4ec4-abba-aac23e2176f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:21:06.061631 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.061610    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05acfa53-02f8-4afc-8159-a495849827da-kube-api-access-bxzk4" (OuterVolumeSpecName: "kube-api-access-bxzk4") pod "05acfa53-02f8-4afc-8159-a495849827da" (UID: "05acfa53-02f8-4afc-8159-a495849827da"). InnerVolumeSpecName "kube-api-access-bxzk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:21:06.061696 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.061676    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35248822-aefa-4202-a5d2-da92d6b5fc51-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35248822-aefa-4202-a5d2-da92d6b5fc51" (UID: "35248822-aefa-4202-a5d2-da92d6b5fc51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:21:06.061923 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.061905    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d24234-b1d5-4ec4-abba-aac23e2176f2-kube-api-access-gjxmt" (OuterVolumeSpecName: "kube-api-access-gjxmt") pod "82d24234-b1d5-4ec4-abba-aac23e2176f2" (UID: "82d24234-b1d5-4ec4-abba-aac23e2176f2"). InnerVolumeSpecName "kube-api-access-gjxmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:21:06.062071 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.062048    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05acfa53-02f8-4afc-8159-a495849827da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05acfa53-02f8-4afc-8159-a495849827da" (UID: "05acfa53-02f8-4afc-8159-a495849827da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:21:06.062503 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.062488    2560 scope.go:117] "RemoveContainer" containerID="b6c223beda3975f9845426bb5b062ff2f7dcaeb6c83b99ff66e350d171d0dc35"
Sep 11 18:21:06.062839 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:21:06.062817    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6c223beda3975f9845426bb5b062ff2f7dcaeb6c83b99ff66e350d171d0dc35\": container with ID starting with b6c223beda3975f9845426bb5b062ff2f7dcaeb6c83b99ff66e350d171d0dc35 not found: ID does not exist" containerID="b6c223beda3975f9845426bb5b062ff2f7dcaeb6c83b99ff66e350d171d0dc35"
Sep 11 18:21:06.062903 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.062849    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c223beda3975f9845426bb5b062ff2f7dcaeb6c83b99ff66e350d171d0dc35"} err="failed to get container status \"b6c223beda3975f9845426bb5b062ff2f7dcaeb6c83b99ff66e350d171d0dc35\": rpc error: code = NotFound desc = could not find container \"b6c223beda3975f9845426bb5b062ff2f7dcaeb6c83b99ff66e350d171d0dc35\": container with ID starting with b6c223beda3975f9845426bb5b062ff2f7dcaeb6c83b99ff66e350d171d0dc35 not found: ID does not exist"
Sep 11 18:21:06.159793 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.159673    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-dshm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.159793 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.159704    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35248822-aefa-4202-a5d2-da92d6b5fc51-tls-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.159793 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.159715    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05acfa53-02f8-4afc-8159-a495849827da-tmp-dir\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.159793 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.159724    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjxmt\" (UniqueName: \"kubernetes.io/projected/82d24234-b1d5-4ec4-abba-aac23e2176f2-kube-api-access-gjxmt\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.159793 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.159734    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-tmp-dir\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.159793 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.159742    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82d24234-b1d5-4ec4-abba-aac23e2176f2-dshm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.159793 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.159781    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35248822-aefa-4202-a5d2-da92d6b5fc51-dshm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.159793 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.159789    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tbqs\" (UniqueName: \"kubernetes.io/projected/35248822-aefa-4202-a5d2-da92d6b5fc51-kube-api-access-6tbqs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.159793 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.159798    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxzk4\" (UniqueName: \"kubernetes.io/projected/05acfa53-02f8-4afc-8159-a495849827da-kube-api-access-bxzk4\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.160198 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.159806    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05acfa53-02f8-4afc-8159-a495849827da-tls-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.160198 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.159814    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82d24234-b1d5-4ec4-abba-aac23e2176f2-tls-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:06.354889 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.354856    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"]
Sep 11 18:21:06.360621 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.360597    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/premium-w0-f8a2d4-kserve-66f694d96c-cvmmd"]
Sep 11 18:21:06.373985 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.373950    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"]
Sep 11 18:21:06.376087 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.376065    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/distinct-w0-f8a2d4-kserve-64ff4959fd-6vfl9"]
Sep 11 18:21:06.389862 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.389817    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"]
Sep 11 18:21:06.396163 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.396133    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f8a2d4/simulator-w0-f8a2d4-kserve-59bb9f7998-j4csl"]
Sep 11 18:21:06.689963 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.689925    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05acfa53-02f8-4afc-8159-a495849827da" path="/var/lib/kubelet/pods/05acfa53-02f8-4afc-8159-a495849827da/volumes"
Sep 11 18:21:06.690349 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.690335    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35248822-aefa-4202-a5d2-da92d6b5fc51" path="/var/lib/kubelet/pods/35248822-aefa-4202-a5d2-da92d6b5fc51/volumes"
Sep 11 18:21:06.690724 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:06.690711    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82d24234-b1d5-4ec4-abba-aac23e2176f2" path="/var/lib/kubelet/pods/82d24234-b1d5-4ec4-abba-aac23e2176f2/volumes"
Sep 11 18:21:07.764271 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:07.764239    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"]
Sep 11 18:21:07.764690 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:07.764507    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj" podUID="1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5" containerName="maas-api" containerID="cri-o://c1c56b628bfef17d8943bfa863a33b2688bf12f58bdc2cf3873f9afc7cb0917b" gracePeriod=30
Sep 11 18:21:08.022325 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.022260    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"
Sep 11 18:21:08.051706 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.051674    2560 generic.go:356] "Generic (PLEG): container finished" podID="1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5" containerID="c1c56b628bfef17d8943bfa863a33b2688bf12f58bdc2cf3873f9afc7cb0917b" exitCode=0
Sep 11 18:21:08.051915 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.051721    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"
Sep 11 18:21:08.051915 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.051781    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj" event={"ID":"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5","Type":"ContainerDied","Data":"c1c56b628bfef17d8943bfa863a33b2688bf12f58bdc2cf3873f9afc7cb0917b"}
Sep 11 18:21:08.051915 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.051832    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj" event={"ID":"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5","Type":"ContainerDied","Data":"050598612a5ffba2f2d0ff2ab9c4beebff3e5d5e046e4490918a3f618b8c0a80"}
Sep 11 18:21:08.051915 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.051856    2560 scope.go:117] "RemoveContainer" containerID="c1c56b628bfef17d8943bfa863a33b2688bf12f58bdc2cf3873f9afc7cb0917b"
Sep 11 18:21:08.062387 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.062366    2560 scope.go:117] "RemoveContainer" containerID="c1c56b628bfef17d8943bfa863a33b2688bf12f58bdc2cf3873f9afc7cb0917b"
Sep 11 18:21:08.062708 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:21:08.062681    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c56b628bfef17d8943bfa863a33b2688bf12f58bdc2cf3873f9afc7cb0917b\": container with ID starting with c1c56b628bfef17d8943bfa863a33b2688bf12f58bdc2cf3873f9afc7cb0917b not found: ID does not exist" containerID="c1c56b628bfef17d8943bfa863a33b2688bf12f58bdc2cf3873f9afc7cb0917b"
Sep 11 18:21:08.062787 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.062721    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c56b628bfef17d8943bfa863a33b2688bf12f58bdc2cf3873f9afc7cb0917b"} err="failed to get container status \"c1c56b628bfef17d8943bfa863a33b2688bf12f58bdc2cf3873f9afc7cb0917b\": rpc error: code = NotFound desc = could not find container \"c1c56b628bfef17d8943bfa863a33b2688bf12f58bdc2cf3873f9afc7cb0917b\": container with ID starting with c1c56b628bfef17d8943bfa863a33b2688bf12f58bdc2cf3873f9afc7cb0917b not found: ID does not exist"
Sep 11 18:21:08.080142 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.080110    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88r8w\" (UniqueName: \"kubernetes.io/projected/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-kube-api-access-88r8w\") pod \"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5\" (UID: \"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5\") "
Sep 11 18:21:08.080282 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.080161    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-metrics-certs\") pod \"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5\" (UID: \"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5\") "
Sep 11 18:21:08.080282 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.080222    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-maas-api-tls\") pod \"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5\" (UID: \"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5\") "
Sep 11 18:21:08.080282 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.080244    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-openshift-service-ca\") pod \"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5\" (UID: \"1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5\") "
Sep 11 18:21:08.080636 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.080613    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5" (UID: "1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:21:08.082330 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.082309    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-kube-api-access-88r8w" (OuterVolumeSpecName: "kube-api-access-88r8w") pod "1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5" (UID: "1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5"). InnerVolumeSpecName "kube-api-access-88r8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:21:08.082497 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.082473    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5" (UID: "1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:21:08.082590 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.082532    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5" (UID: "1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:21:08.181355 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.181311    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88r8w\" (UniqueName: \"kubernetes.io/projected/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-kube-api-access-88r8w\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:08.181355 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.181349    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-metrics-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:08.181355 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.181362    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:08.181611 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.181372    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:08.379178 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.379145    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"]
Sep 11 18:21:08.383155 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.383126    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-684bbd-85dc69979b-7v4bj"]
Sep 11 18:21:08.688113 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:08.688022    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5" path="/var/lib/kubelet/pods/1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5/volumes"
Sep 11 18:21:13.419281 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:13.419241    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:21:31.838843 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:31.838809    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"]
Sep 11 18:21:31.839324 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:31.839029    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7" podUID="4727935a-3579-4e9c-a9ba-4d0dec1cf07b" containerName="maas-api" containerID="cri-o://43a0078930de3e727e680c62d1ebe5918ed7c795219c02a1491289b6d32a91c0" gracePeriod=30
Sep 11 18:21:32.095619 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.095559    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"
Sep 11 18:21:32.122022 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.121995    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kr99z\" (UniqueName: \"kubernetes.io/projected/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-kube-api-access-kr99z\") pod \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\" (UID: \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\") "
Sep 11 18:21:32.122187 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.122140    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-openshift-service-ca\") pod \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\" (UID: \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\") "
Sep 11 18:21:32.122187 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.122169    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-maas-api-tls\") pod \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\" (UID: \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\") "
Sep 11 18:21:32.122187 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.122185    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-metrics-certs\") pod \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\" (UID: \"4727935a-3579-4e9c-a9ba-4d0dec1cf07b\") "
Sep 11 18:21:32.122563 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.122538    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4727935a-3579-4e9c-a9ba-4d0dec1cf07b" (UID: "4727935a-3579-4e9c-a9ba-4d0dec1cf07b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:21:32.124382 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.124356    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-kube-api-access-kr99z" (OuterVolumeSpecName: "kube-api-access-kr99z") pod "4727935a-3579-4e9c-a9ba-4d0dec1cf07b" (UID: "4727935a-3579-4e9c-a9ba-4d0dec1cf07b"). InnerVolumeSpecName "kube-api-access-kr99z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:21:32.124487 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.124430    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4727935a-3579-4e9c-a9ba-4d0dec1cf07b" (UID: "4727935a-3579-4e9c-a9ba-4d0dec1cf07b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:21:32.124606 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.124589    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4727935a-3579-4e9c-a9ba-4d0dec1cf07b" (UID: "4727935a-3579-4e9c-a9ba-4d0dec1cf07b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:21:32.183204 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.183173    2560 generic.go:356] "Generic (PLEG): container finished" podID="4727935a-3579-4e9c-a9ba-4d0dec1cf07b" containerID="43a0078930de3e727e680c62d1ebe5918ed7c795219c02a1491289b6d32a91c0" exitCode=0
Sep 11 18:21:32.183390 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.183222    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"
Sep 11 18:21:32.183390 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.183259    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7" event={"ID":"4727935a-3579-4e9c-a9ba-4d0dec1cf07b","Type":"ContainerDied","Data":"43a0078930de3e727e680c62d1ebe5918ed7c795219c02a1491289b6d32a91c0"}
Sep 11 18:21:32.183390 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.183315    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7" event={"ID":"4727935a-3579-4e9c-a9ba-4d0dec1cf07b","Type":"ContainerDied","Data":"0818e99473985a1be09ddda7c381b5948a5bd2fb37943f2c46899430fa1c89e5"}
Sep 11 18:21:32.183390 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.183340    2560 scope.go:117] "RemoveContainer" containerID="43a0078930de3e727e680c62d1ebe5918ed7c795219c02a1491289b6d32a91c0"
Sep 11 18:21:32.196332 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.196313    2560 scope.go:117] "RemoveContainer" containerID="43a0078930de3e727e680c62d1ebe5918ed7c795219c02a1491289b6d32a91c0"
Sep 11 18:21:32.196637 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:21:32.196618    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43a0078930de3e727e680c62d1ebe5918ed7c795219c02a1491289b6d32a91c0\": container with ID starting with 43a0078930de3e727e680c62d1ebe5918ed7c795219c02a1491289b6d32a91c0 not found: ID does not exist" containerID="43a0078930de3e727e680c62d1ebe5918ed7c795219c02a1491289b6d32a91c0"
Sep 11 18:21:32.196705 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.196648    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a0078930de3e727e680c62d1ebe5918ed7c795219c02a1491289b6d32a91c0"} err="failed to get container status \"43a0078930de3e727e680c62d1ebe5918ed7c795219c02a1491289b6d32a91c0\": rpc error: code = NotFound desc = could not find container \"43a0078930de3e727e680c62d1ebe5918ed7c795219c02a1491289b6d32a91c0\": container with ID starting with 43a0078930de3e727e680c62d1ebe5918ed7c795219c02a1491289b6d32a91c0 not found: ID does not exist"
Sep 11 18:21:32.207667 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.207639    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"]
Sep 11 18:21:32.209580 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.209562    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-794d1c-567d997df5-vpff7"]
Sep 11 18:21:32.223906 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.223873    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:32.223906 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.223905    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-metrics-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:32.223906 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.223916    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr99z\" (UniqueName: \"kubernetes.io/projected/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-kube-api-access-kr99z\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:32.224050 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.223925    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4727935a-3579-4e9c-a9ba-4d0dec1cf07b-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:21:32.687688 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:32.687640    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4727935a-3579-4e9c-a9ba-4d0dec1cf07b" path="/var/lib/kubelet/pods/4727935a-3579-4e9c-a9ba-4d0dec1cf07b/volumes"
Sep 11 18:21:41.814085 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:41.814052    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:21:49.850008 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:49.849974    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b05610-29819175-gsvl7"]
Sep 11 18:21:49.855046 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:49.855022    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b05610-29819175-gsvl7"]
Sep 11 18:21:50.688354 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:50.688317    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c968d5a2-8342-4163-a73b-d31a9d818c70" path="/var/lib/kubelet/pods/c968d5a2-8342-4163-a73b-d31a9d818c70/volumes"
Sep 11 18:21:53.652426 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:53.652390    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"]
Sep 11 18:21:53.657194 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:53.657165    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"]
Sep 11 18:21:53.661840 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:21:53.661814    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"]
Sep 11 18:22:02.831910 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:02.831874    2560 scope.go:117] "RemoveContainer" containerID="fe5bfe49ea0316d817479d5e49a1441ea7771db3309f5d16e07c2eca6bf83a7b"
Sep 11 18:22:02.841098 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:02.841077    2560 scope.go:117] "RemoveContainer" containerID="480e886044e43d24489c7ebcfbb6d7e32dca0f7498b0925d3e0e6dd1ea24e8b2"
Sep 11 18:22:02.850936 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:02.850019    2560 scope.go:117] "RemoveContainer" containerID="66ac85702f445a22d422145a28d052e02a9f8d1e1007ddbf915b259139599161"
Sep 11 18:22:06.611483 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:06.611447    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:22:08.497407 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:08.497374    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"]
Sep 11 18:22:08.497972 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:08.497624    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt" podUID="dd879040-5d7d-4c11-9a8f-9c60ec980169" containerName="maas-api" containerID="cri-o://a38dd7af42f53005307476bd2ebd3f344834e168568ab92335623173a3ba1041" gracePeriod=30
Sep 11 18:22:08.674007 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:08.673951    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn" podUID="828634bf-7193-4d57-aabf-98a5102993eb" containerName="main" containerID="cri-o://78dce24db6a5a4bd05b02db769e878e9d9d59ab96bfa9b54283a20342e3a0363" gracePeriod=45
Sep 11 18:22:08.678385 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:08.678344    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr" podUID="35d12f6a-c0bf-4cdf-ac77-c78e773253c9" containerName="main" containerID="cri-o://3b051e92c0b1fb29c43b53808a7a8ffc7c87a22a83e09e4f44b55d369c5dfa05" gracePeriod=45
Sep 11 18:22:08.683155 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:08.683098    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5" podUID="221f6e7a-44be-4e66-82db-4b852b2ef7e2" containerName="main" containerID="cri-o://8400d7c80dccdc0c767bf7242d7c4184b27a02f9a363d388dd93709f9327c7e6" gracePeriod=45
Sep 11 18:22:08.887884 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:08.887856    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"
Sep 11 18:22:08.964816 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:08.964792    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:22:08.968024 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:08.967997    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:22:08.973538 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:08.973518    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:22:09.000687 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.000657    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd879040-5d7d-4c11-9a8f-9c60ec980169-openshift-service-ca\") pod \"dd879040-5d7d-4c11-9a8f-9c60ec980169\" (UID: \"dd879040-5d7d-4c11-9a8f-9c60ec980169\") "
Sep 11 18:22:09.000896 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.000705    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-model-cache\") pod \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") "
Sep 11 18:22:09.000896 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.000733    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-model-cache\") pod \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") "
Sep 11 18:22:09.000896 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.000802    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-home\") pod \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") "
Sep 11 18:22:09.000896 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.000831    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dc4cj\" (UniqueName: \"kubernetes.io/projected/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-kube-api-access-dc4cj\") pod \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") "
Sep 11 18:22:09.000896 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.000871    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-home\") pod \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") "
Sep 11 18:22:09.001191 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.000933    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdmdt\" (UniqueName: \"kubernetes.io/projected/221f6e7a-44be-4e66-82db-4b852b2ef7e2-kube-api-access-hdmdt\") pod \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") "
Sep 11 18:22:09.001191 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.000962    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828634bf-7193-4d57-aabf-98a5102993eb-tls-certs\") pod \"828634bf-7193-4d57-aabf-98a5102993eb\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") "
Sep 11 18:22:09.001961 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.001939    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-tmp-dir\") pod \"828634bf-7193-4d57-aabf-98a5102993eb\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") "
Sep 11 18:22:09.002202 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.002184    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/221f6e7a-44be-4e66-82db-4b852b2ef7e2-tls-certs\") pod \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") "
Sep 11 18:22:09.002371 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.002344    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-model-cache\") pod \"828634bf-7193-4d57-aabf-98a5102993eb\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") "
Sep 11 18:22:09.002945 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.001498    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-home" (OuterVolumeSpecName: "home") pod "221f6e7a-44be-4e66-82db-4b852b2ef7e2" (UID: "221f6e7a-44be-4e66-82db-4b852b2ef7e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:09.002945 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.001498    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-home" (OuterVolumeSpecName: "home") pod "35d12f6a-c0bf-4cdf-ac77-c78e773253c9" (UID: "35d12f6a-c0bf-4cdf-ac77-c78e773253c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:09.002945 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.001517    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "221f6e7a-44be-4e66-82db-4b852b2ef7e2" (UID: "221f6e7a-44be-4e66-82db-4b852b2ef7e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:09.002945 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.001783    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd879040-5d7d-4c11-9a8f-9c60ec980169-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dd879040-5d7d-4c11-9a8f-9c60ec980169" (UID: "dd879040-5d7d-4c11-9a8f-9c60ec980169"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:22:09.002945 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.002128    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "35d12f6a-c0bf-4cdf-ac77-c78e773253c9" (UID: "35d12f6a-c0bf-4cdf-ac77-c78e773253c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:09.002945 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.002208    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "828634bf-7193-4d57-aabf-98a5102993eb" (UID: "828634bf-7193-4d57-aabf-98a5102993eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:09.002945 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.002682    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "828634bf-7193-4d57-aabf-98a5102993eb" (UID: "828634bf-7193-4d57-aabf-98a5102993eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:09.003412 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.003394    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prhcd\" (UniqueName: \"kubernetes.io/projected/828634bf-7193-4d57-aabf-98a5102993eb-kube-api-access-prhcd\") pod \"828634bf-7193-4d57-aabf-98a5102993eb\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") "
Sep 11 18:22:09.003536 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.003523    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-tmp-dir\") pod \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") "
Sep 11 18:22:09.003642 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.003628    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-tls-certs\") pod \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") "
Sep 11 18:22:09.003812 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.003741    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6l8m4\" (UniqueName: \"kubernetes.io/projected/dd879040-5d7d-4c11-9a8f-9c60ec980169-kube-api-access-6l8m4\") pod \"dd879040-5d7d-4c11-9a8f-9c60ec980169\" (UID: \"dd879040-5d7d-4c11-9a8f-9c60ec980169\") "
Sep 11 18:22:09.003939 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.003925    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-dshm\") pod \"828634bf-7193-4d57-aabf-98a5102993eb\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") "
Sep 11 18:22:09.004051 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.004038    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-home\") pod \"828634bf-7193-4d57-aabf-98a5102993eb\" (UID: \"828634bf-7193-4d57-aabf-98a5102993eb\") "
Sep 11 18:22:09.004167 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.004154    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd879040-5d7d-4c11-9a8f-9c60ec980169-metrics-certs\") pod \"dd879040-5d7d-4c11-9a8f-9c60ec980169\" (UID: \"dd879040-5d7d-4c11-9a8f-9c60ec980169\") "
Sep 11 18:22:09.004265 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.004253    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-tmp-dir\") pod \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") "
Sep 11 18:22:09.004363 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.004349    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-dshm\") pod \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\" (UID: \"221f6e7a-44be-4e66-82db-4b852b2ef7e2\") "
Sep 11 18:22:09.004477 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.004464    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-dshm\") pod \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\" (UID: \"35d12f6a-c0bf-4cdf-ac77-c78e773253c9\") "
Sep 11 18:22:09.004598 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.004583    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd879040-5d7d-4c11-9a8f-9c60ec980169-maas-api-tls\") pod \"dd879040-5d7d-4c11-9a8f-9c60ec980169\" (UID: \"dd879040-5d7d-4c11-9a8f-9c60ec980169\") "
Sep 11 18:22:09.006220 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.005651    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "221f6e7a-44be-4e66-82db-4b852b2ef7e2" (UID: "221f6e7a-44be-4e66-82db-4b852b2ef7e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:09.006220 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.005732    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-kube-api-access-dc4cj" (OuterVolumeSpecName: "kube-api-access-dc4cj") pod "35d12f6a-c0bf-4cdf-ac77-c78e773253c9" (UID: "35d12f6a-c0bf-4cdf-ac77-c78e773253c9"). InnerVolumeSpecName "kube-api-access-dc4cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:22:09.006220 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.005954    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-home" (OuterVolumeSpecName: "home") pod "828634bf-7193-4d57-aabf-98a5102993eb" (UID: "828634bf-7193-4d57-aabf-98a5102993eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:09.006532 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.006503    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/828634bf-7193-4d57-aabf-98a5102993eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "828634bf-7193-4d57-aabf-98a5102993eb" (UID: "828634bf-7193-4d57-aabf-98a5102993eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:22:09.007005 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.006721    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "35d12f6a-c0bf-4cdf-ac77-c78e773253c9" (UID: "35d12f6a-c0bf-4cdf-ac77-c78e773253c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:09.007868 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.007567    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-tmp-dir\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.007868 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.007589    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-home\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.007868 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.007601    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-tmp-dir\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.007868 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.007613    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd879040-5d7d-4c11-9a8f-9c60ec980169-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.007868 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.007627    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-model-cache\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.007868 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.007638    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-model-cache\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.007868 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.007648    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-home\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.007868 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.007658    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc4cj\" (UniqueName: \"kubernetes.io/projected/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-kube-api-access-dc4cj\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.007868 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.007669    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-home\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.007868 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.007681    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/828634bf-7193-4d57-aabf-98a5102993eb-tls-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.007868 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.007692    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-tmp-dir\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.007868 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.007703    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-model-cache\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.008595 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.008362    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35d12f6a-c0bf-4cdf-ac77-c78e773253c9" (UID: "35d12f6a-c0bf-4cdf-ac77-c78e773253c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:22:09.008595 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.008456    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/221f6e7a-44be-4e66-82db-4b852b2ef7e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "221f6e7a-44be-4e66-82db-4b852b2ef7e2" (UID: "221f6e7a-44be-4e66-82db-4b852b2ef7e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:22:09.008595 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.008538    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221f6e7a-44be-4e66-82db-4b852b2ef7e2-kube-api-access-hdmdt" (OuterVolumeSpecName: "kube-api-access-hdmdt") pod "221f6e7a-44be-4e66-82db-4b852b2ef7e2" (UID: "221f6e7a-44be-4e66-82db-4b852b2ef7e2"). InnerVolumeSpecName "kube-api-access-hdmdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:22:09.009937 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.009912    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828634bf-7193-4d57-aabf-98a5102993eb-kube-api-access-prhcd" (OuterVolumeSpecName: "kube-api-access-prhcd") pod "828634bf-7193-4d57-aabf-98a5102993eb" (UID: "828634bf-7193-4d57-aabf-98a5102993eb"). InnerVolumeSpecName "kube-api-access-prhcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:22:09.010405 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.010373    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd879040-5d7d-4c11-9a8f-9c60ec980169-kube-api-access-6l8m4" (OuterVolumeSpecName: "kube-api-access-6l8m4") pod "dd879040-5d7d-4c11-9a8f-9c60ec980169" (UID: "dd879040-5d7d-4c11-9a8f-9c60ec980169"). InnerVolumeSpecName "kube-api-access-6l8m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:22:09.010405 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.010388    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-dshm" (OuterVolumeSpecName: "dshm") pod "221f6e7a-44be-4e66-82db-4b852b2ef7e2" (UID: "221f6e7a-44be-4e66-82db-4b852b2ef7e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:09.010518 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.010409    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd879040-5d7d-4c11-9a8f-9c60ec980169-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dd879040-5d7d-4c11-9a8f-9c60ec980169" (UID: "dd879040-5d7d-4c11-9a8f-9c60ec980169"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:22:09.010518 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.010469    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-dshm" (OuterVolumeSpecName: "dshm") pod "35d12f6a-c0bf-4cdf-ac77-c78e773253c9" (UID: "35d12f6a-c0bf-4cdf-ac77-c78e773253c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:09.010805 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.010742    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd879040-5d7d-4c11-9a8f-9c60ec980169-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dd879040-5d7d-4c11-9a8f-9c60ec980169" (UID: "dd879040-5d7d-4c11-9a8f-9c60ec980169"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:22:09.010980 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.010961    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-dshm" (OuterVolumeSpecName: "dshm") pod "828634bf-7193-4d57-aabf-98a5102993eb" (UID: "828634bf-7193-4d57-aabf-98a5102993eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:09.109208 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.109171    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdmdt\" (UniqueName: \"kubernetes.io/projected/221f6e7a-44be-4e66-82db-4b852b2ef7e2-kube-api-access-hdmdt\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.109208 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.109202    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/221f6e7a-44be-4e66-82db-4b852b2ef7e2-tls-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.109208 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.109213    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prhcd\" (UniqueName: \"kubernetes.io/projected/828634bf-7193-4d57-aabf-98a5102993eb-kube-api-access-prhcd\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.109449 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.109221    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-tls-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.109449 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.109231    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l8m4\" (UniqueName: \"kubernetes.io/projected/dd879040-5d7d-4c11-9a8f-9c60ec980169-kube-api-access-6l8m4\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.109449 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.109238    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/828634bf-7193-4d57-aabf-98a5102993eb-dshm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.109449 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.109246    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd879040-5d7d-4c11-9a8f-9c60ec980169-metrics-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.109449 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.109253    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/221f6e7a-44be-4e66-82db-4b852b2ef7e2-dshm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.109449 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.109260    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35d12f6a-c0bf-4cdf-ac77-c78e773253c9-dshm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.109449 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.109267    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd879040-5d7d-4c11-9a8f-9c60ec980169-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:09.386501 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.386410    2560 generic.go:356] "Generic (PLEG): container finished" podID="828634bf-7193-4d57-aabf-98a5102993eb" containerID="78dce24db6a5a4bd05b02db769e878e9d9d59ab96bfa9b54283a20342e3a0363" exitCode=0
Sep 11 18:22:09.386501 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.386458    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"
Sep 11 18:22:09.386741 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.386497    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn" event={"ID":"828634bf-7193-4d57-aabf-98a5102993eb","Type":"ContainerDied","Data":"78dce24db6a5a4bd05b02db769e878e9d9d59ab96bfa9b54283a20342e3a0363"}
Sep 11 18:22:09.386741 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.386546    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn" event={"ID":"828634bf-7193-4d57-aabf-98a5102993eb","Type":"ContainerDied","Data":"2d09c3f23312ac95c1da112628b69ec488cd300634b80d1d39f4f44f4e04e2c6"}
Sep 11 18:22:09.386741 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.386571    2560 scope.go:117] "RemoveContainer" containerID="78dce24db6a5a4bd05b02db769e878e9d9d59ab96bfa9b54283a20342e3a0363"
Sep 11 18:22:09.388134 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.388109    2560 generic.go:356] "Generic (PLEG): container finished" podID="35d12f6a-c0bf-4cdf-ac77-c78e773253c9" containerID="3b051e92c0b1fb29c43b53808a7a8ffc7c87a22a83e09e4f44b55d369c5dfa05" exitCode=0
Sep 11 18:22:09.388219 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.388144    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"
Sep 11 18:22:09.388316 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.388297    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr" event={"ID":"35d12f6a-c0bf-4cdf-ac77-c78e773253c9","Type":"ContainerDied","Data":"3b051e92c0b1fb29c43b53808a7a8ffc7c87a22a83e09e4f44b55d369c5dfa05"}
Sep 11 18:22:09.388364 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.388335    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr" event={"ID":"35d12f6a-c0bf-4cdf-ac77-c78e773253c9","Type":"ContainerDied","Data":"26ebbbd32b8eb82aa50f0f5ab384d25479a80ab7fb55afef2934a1526f024a26"}
Sep 11 18:22:09.389882 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.389856    2560 generic.go:356] "Generic (PLEG): container finished" podID="221f6e7a-44be-4e66-82db-4b852b2ef7e2" containerID="8400d7c80dccdc0c767bf7242d7c4184b27a02f9a363d388dd93709f9327c7e6" exitCode=0
Sep 11 18:22:09.389996 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.389892    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5" event={"ID":"221f6e7a-44be-4e66-82db-4b852b2ef7e2","Type":"ContainerDied","Data":"8400d7c80dccdc0c767bf7242d7c4184b27a02f9a363d388dd93709f9327c7e6"}
Sep 11 18:22:09.389996 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.389908    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"
Sep 11 18:22:09.389996 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.389945    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5" event={"ID":"221f6e7a-44be-4e66-82db-4b852b2ef7e2","Type":"ContainerDied","Data":"a8d60aad90cc30f7dc860202522d47a4675599cc6297a814b63b65db7d5e9bd3"}
Sep 11 18:22:09.391525 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.391500    2560 generic.go:356] "Generic (PLEG): container finished" podID="dd879040-5d7d-4c11-9a8f-9c60ec980169" containerID="a38dd7af42f53005307476bd2ebd3f344834e168568ab92335623173a3ba1041" exitCode=0
Sep 11 18:22:09.391668 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.391544    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"
Sep 11 18:22:09.391668 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.391560    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt" event={"ID":"dd879040-5d7d-4c11-9a8f-9c60ec980169","Type":"ContainerDied","Data":"a38dd7af42f53005307476bd2ebd3f344834e168568ab92335623173a3ba1041"}
Sep 11 18:22:09.391668 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.391600    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt" event={"ID":"dd879040-5d7d-4c11-9a8f-9c60ec980169","Type":"ContainerDied","Data":"72e3af50598b4be1fde934dbc484cb065f55f4f853df1fb7956afaf5635a3438"}
Sep 11 18:22:09.397537 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.397443    2560 scope.go:117] "RemoveContainer" containerID="78dce24db6a5a4bd05b02db769e878e9d9d59ab96bfa9b54283a20342e3a0363"
Sep 11 18:22:09.397839 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:22:09.397816    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78dce24db6a5a4bd05b02db769e878e9d9d59ab96bfa9b54283a20342e3a0363\": container with ID starting with 78dce24db6a5a4bd05b02db769e878e9d9d59ab96bfa9b54283a20342e3a0363 not found: ID does not exist" containerID="78dce24db6a5a4bd05b02db769e878e9d9d59ab96bfa9b54283a20342e3a0363"
Sep 11 18:22:09.397943 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.397845    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78dce24db6a5a4bd05b02db769e878e9d9d59ab96bfa9b54283a20342e3a0363"} err="failed to get container status \"78dce24db6a5a4bd05b02db769e878e9d9d59ab96bfa9b54283a20342e3a0363\": rpc error: code = NotFound desc = could not find container \"78dce24db6a5a4bd05b02db769e878e9d9d59ab96bfa9b54283a20342e3a0363\": container with ID starting with 78dce24db6a5a4bd05b02db769e878e9d9d59ab96bfa9b54283a20342e3a0363 not found: ID does not exist"
Sep 11 18:22:09.397943 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.397867    2560 scope.go:117] "RemoveContainer" containerID="3b051e92c0b1fb29c43b53808a7a8ffc7c87a22a83e09e4f44b55d369c5dfa05"
Sep 11 18:22:09.407930 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.407911    2560 scope.go:117] "RemoveContainer" containerID="3b051e92c0b1fb29c43b53808a7a8ffc7c87a22a83e09e4f44b55d369c5dfa05"
Sep 11 18:22:09.408203 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:22:09.408183    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b051e92c0b1fb29c43b53808a7a8ffc7c87a22a83e09e4f44b55d369c5dfa05\": container with ID starting with 3b051e92c0b1fb29c43b53808a7a8ffc7c87a22a83e09e4f44b55d369c5dfa05 not found: ID does not exist" containerID="3b051e92c0b1fb29c43b53808a7a8ffc7c87a22a83e09e4f44b55d369c5dfa05"
Sep 11 18:22:09.408254 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.408213    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b051e92c0b1fb29c43b53808a7a8ffc7c87a22a83e09e4f44b55d369c5dfa05"} err="failed to get container status \"3b051e92c0b1fb29c43b53808a7a8ffc7c87a22a83e09e4f44b55d369c5dfa05\": rpc error: code = NotFound desc = could not find container \"3b051e92c0b1fb29c43b53808a7a8ffc7c87a22a83e09e4f44b55d369c5dfa05\": container with ID starting with 3b051e92c0b1fb29c43b53808a7a8ffc7c87a22a83e09e4f44b55d369c5dfa05 not found: ID does not exist"
Sep 11 18:22:09.408254 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.408234    2560 scope.go:117] "RemoveContainer" containerID="8400d7c80dccdc0c767bf7242d7c4184b27a02f9a363d388dd93709f9327c7e6"
Sep 11 18:22:09.414690 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.414664    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"]
Sep 11 18:22:09.416657 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.416638    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/distinct-w4-4e0b0b-kserve-878d66548-dj9dn"]
Sep 11 18:22:09.417189 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.417173    2560 scope.go:117] "RemoveContainer" containerID="8400d7c80dccdc0c767bf7242d7c4184b27a02f9a363d388dd93709f9327c7e6"
Sep 11 18:22:09.417505 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:22:09.417487    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8400d7c80dccdc0c767bf7242d7c4184b27a02f9a363d388dd93709f9327c7e6\": container with ID starting with 8400d7c80dccdc0c767bf7242d7c4184b27a02f9a363d388dd93709f9327c7e6 not found: ID does not exist" containerID="8400d7c80dccdc0c767bf7242d7c4184b27a02f9a363d388dd93709f9327c7e6"
Sep 11 18:22:09.417551 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.417512    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8400d7c80dccdc0c767bf7242d7c4184b27a02f9a363d388dd93709f9327c7e6"} err="failed to get container status \"8400d7c80dccdc0c767bf7242d7c4184b27a02f9a363d388dd93709f9327c7e6\": rpc error: code = NotFound desc = could not find container \"8400d7c80dccdc0c767bf7242d7c4184b27a02f9a363d388dd93709f9327c7e6\": container with ID starting with 8400d7c80dccdc0c767bf7242d7c4184b27a02f9a363d388dd93709f9327c7e6 not found: ID does not exist"
Sep 11 18:22:09.417551 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.417531    2560 scope.go:117] "RemoveContainer" containerID="a38dd7af42f53005307476bd2ebd3f344834e168568ab92335623173a3ba1041"
Sep 11 18:22:09.428066 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.428038    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"]
Sep 11 18:22:09.429646 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.429618    2560 scope.go:117] "RemoveContainer" containerID="a38dd7af42f53005307476bd2ebd3f344834e168568ab92335623173a3ba1041"
Sep 11 18:22:09.430214 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:22:09.430185    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a38dd7af42f53005307476bd2ebd3f344834e168568ab92335623173a3ba1041\": container with ID starting with a38dd7af42f53005307476bd2ebd3f344834e168568ab92335623173a3ba1041 not found: ID does not exist" containerID="a38dd7af42f53005307476bd2ebd3f344834e168568ab92335623173a3ba1041"
Sep 11 18:22:09.430294 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.430222    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a38dd7af42f53005307476bd2ebd3f344834e168568ab92335623173a3ba1041"} err="failed to get container status \"a38dd7af42f53005307476bd2ebd3f344834e168568ab92335623173a3ba1041\": rpc error: code = NotFound desc = could not find container \"a38dd7af42f53005307476bd2ebd3f344834e168568ab92335623173a3ba1041\": container with ID starting with a38dd7af42f53005307476bd2ebd3f344834e168568ab92335623173a3ba1041 not found: ID does not exist"
Sep 11 18:22:09.430959 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.430938    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-90057a-6f68d477bb-pd8gt"]
Sep 11 18:22:09.440976 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.440949    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"]
Sep 11 18:22:09.444409 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.444388    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/premium-w4-4e0b0b-kserve-696cd48cb9-lrtvr"]
Sep 11 18:22:09.454278 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.454251    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"]
Sep 11 18:22:09.458462 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:09.458438    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4e0b0b/simulator-w4-4e0b0b-kserve-64cdb4d788-c7jr5"]
Sep 11 18:22:10.688777 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:10.688726    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="221f6e7a-44be-4e66-82db-4b852b2ef7e2" path="/var/lib/kubelet/pods/221f6e7a-44be-4e66-82db-4b852b2ef7e2/volumes"
Sep 11 18:22:10.689224 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:10.689209    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35d12f6a-c0bf-4cdf-ac77-c78e773253c9" path="/var/lib/kubelet/pods/35d12f6a-c0bf-4cdf-ac77-c78e773253c9/volumes"
Sep 11 18:22:10.689590 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:10.689576    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="828634bf-7193-4d57-aabf-98a5102993eb" path="/var/lib/kubelet/pods/828634bf-7193-4d57-aabf-98a5102993eb/volumes"
Sep 11 18:22:10.690293 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:10.690277    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd879040-5d7d-4c11-9a8f-9c60ec980169" path="/var/lib/kubelet/pods/dd879040-5d7d-4c11-9a8f-9c60ec980169/volumes"
Sep 11 18:22:25.807661 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:25.807569    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:22:38.900540 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:38.900508    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"]
Sep 11 18:22:40.317637 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:40.317596    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:22:53.920764 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:53.920695    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp" podUID="96aa235d-5d81-4486-a2b9-0ca3152bddf7" containerName="main" containerID="cri-o://84648309bd047d1cb0ab905971855cc6ac4652437397cce95adb43e033281880" gracePeriod=45
Sep 11 18:22:54.167262 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.167230    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:22:54.208311 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.208231    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:22:54.263969 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.263940    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-tmp-dir\") pod \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") "
Sep 11 18:22:54.264111 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.263977    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-model-cache\") pod \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") "
Sep 11 18:22:54.264111 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.264010    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn2t8\" (UniqueName: \"kubernetes.io/projected/96aa235d-5d81-4486-a2b9-0ca3152bddf7-kube-api-access-kn2t8\") pod \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") "
Sep 11 18:22:54.264111 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.264054    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96aa235d-5d81-4486-a2b9-0ca3152bddf7-tls-certs\") pod \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") "
Sep 11 18:22:54.264244 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.264112    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-home\") pod \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") "
Sep 11 18:22:54.264244 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.264169    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-dshm\") pod \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\" (UID: \"96aa235d-5d81-4486-a2b9-0ca3152bddf7\") "
Sep 11 18:22:54.264361 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.264243    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-model-cache" (OuterVolumeSpecName: "model-cache") pod "96aa235d-5d81-4486-a2b9-0ca3152bddf7" (UID: "96aa235d-5d81-4486-a2b9-0ca3152bddf7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:54.264361 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.264263    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96aa235d-5d81-4486-a2b9-0ca3152bddf7" (UID: "96aa235d-5d81-4486-a2b9-0ca3152bddf7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:54.264470 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.264446    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-home" (OuterVolumeSpecName: "home") pod "96aa235d-5d81-4486-a2b9-0ca3152bddf7" (UID: "96aa235d-5d81-4486-a2b9-0ca3152bddf7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:54.264587 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.264570    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-home\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:54.264648 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.264592    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-tmp-dir\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:54.264648 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.264618    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-model-cache\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:54.266471 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.266447    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-dshm" (OuterVolumeSpecName: "dshm") pod "96aa235d-5d81-4486-a2b9-0ca3152bddf7" (UID: "96aa235d-5d81-4486-a2b9-0ca3152bddf7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:22:54.266543 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.266487    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96aa235d-5d81-4486-a2b9-0ca3152bddf7-kube-api-access-kn2t8" (OuterVolumeSpecName: "kube-api-access-kn2t8") pod "96aa235d-5d81-4486-a2b9-0ca3152bddf7" (UID: "96aa235d-5d81-4486-a2b9-0ca3152bddf7"). InnerVolumeSpecName "kube-api-access-kn2t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:22:54.266581 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.266554    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96aa235d-5d81-4486-a2b9-0ca3152bddf7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96aa235d-5d81-4486-a2b9-0ca3152bddf7" (UID: "96aa235d-5d81-4486-a2b9-0ca3152bddf7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:22:54.366072 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.366043    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96aa235d-5d81-4486-a2b9-0ca3152bddf7-dshm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:54.366072 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.366065    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn2t8\" (UniqueName: \"kubernetes.io/projected/96aa235d-5d81-4486-a2b9-0ca3152bddf7-kube-api-access-kn2t8\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:54.366072 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.366075    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96aa235d-5d81-4486-a2b9-0ca3152bddf7-tls-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:22:54.625575 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.625543    2560 generic.go:356] "Generic (PLEG): container finished" podID="96aa235d-5d81-4486-a2b9-0ca3152bddf7" containerID="84648309bd047d1cb0ab905971855cc6ac4652437397cce95adb43e033281880" exitCode=0
Sep 11 18:22:54.625770 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.625588    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"
Sep 11 18:22:54.625770 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.625635    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp" event={"ID":"96aa235d-5d81-4486-a2b9-0ca3152bddf7","Type":"ContainerDied","Data":"84648309bd047d1cb0ab905971855cc6ac4652437397cce95adb43e033281880"}
Sep 11 18:22:54.625770 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.625674    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp" event={"ID":"96aa235d-5d81-4486-a2b9-0ca3152bddf7","Type":"ContainerDied","Data":"6a5c1e05751f56e856017900f0dad3f524b679274122819acba818d2e2b5578b"}
Sep 11 18:22:54.625770 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.625691    2560 scope.go:117] "RemoveContainer" containerID="84648309bd047d1cb0ab905971855cc6ac4652437397cce95adb43e033281880"
Sep 11 18:22:54.635128 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.635111    2560 scope.go:117] "RemoveContainer" containerID="84648309bd047d1cb0ab905971855cc6ac4652437397cce95adb43e033281880"
Sep 11 18:22:54.635421 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:22:54.635399    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84648309bd047d1cb0ab905971855cc6ac4652437397cce95adb43e033281880\": container with ID starting with 84648309bd047d1cb0ab905971855cc6ac4652437397cce95adb43e033281880 not found: ID does not exist" containerID="84648309bd047d1cb0ab905971855cc6ac4652437397cce95adb43e033281880"
Sep 11 18:22:54.635527 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.635427    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84648309bd047d1cb0ab905971855cc6ac4652437397cce95adb43e033281880"} err="failed to get container status \"84648309bd047d1cb0ab905971855cc6ac4652437397cce95adb43e033281880\": rpc error: code = NotFound desc = could not find container \"84648309bd047d1cb0ab905971855cc6ac4652437397cce95adb43e033281880\": container with ID starting with 84648309bd047d1cb0ab905971855cc6ac4652437397cce95adb43e033281880 not found: ID does not exist"
Sep 11 18:22:54.647201 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.647174    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"]
Sep 11 18:22:54.650489 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.650470    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp"]
Sep 11 18:22:54.687685 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:54.687658    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96aa235d-5d81-4486-a2b9-0ca3152bddf7" path="/var/lib/kubelet/pods/96aa235d-5d81-4486-a2b9-0ca3152bddf7/volumes"
Sep 11 18:22:59.088209 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:22:59.088145    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-5d35e7/auth-test-model-5d35e7-kserve-5db74fcd7-mr4qp" podUID="96aa235d-5d81-4486-a2b9-0ca3152bddf7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/ready\": context deadline exceeded"
Sep 11 18:23:22.015319 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:22.015276    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:23:41.816638 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:41.816603    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:23:54.506466 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:54.506371    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:23:56.406585 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.406535    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7499-5a2c78"]
Sep 11 18:23:56.407428 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407405    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5" containerName="maas-api"
Sep 11 18:23:56.407531 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407432    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5" containerName="maas-api"
Sep 11 18:23:56.407531 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407469    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82d24234-b1d5-4ec4-abba-aac23e2176f2" containerName="main"
Sep 11 18:23:56.407531 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407477    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="82d24234-b1d5-4ec4-abba-aac23e2176f2" containerName="main"
Sep 11 18:23:56.407531 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407497    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96aa235d-5d81-4486-a2b9-0ca3152bddf7" containerName="main"
Sep 11 18:23:56.407531 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407504    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="96aa235d-5d81-4486-a2b9-0ca3152bddf7" containerName="main"
Sep 11 18:23:56.407798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407535    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4727935a-3579-4e9c-a9ba-4d0dec1cf07b" containerName="maas-api"
Sep 11 18:23:56.407798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407544    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4727935a-3579-4e9c-a9ba-4d0dec1cf07b" containerName="maas-api"
Sep 11 18:23:56.407798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407563    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05acfa53-02f8-4afc-8159-a495849827da" containerName="main"
Sep 11 18:23:56.407798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407577    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="05acfa53-02f8-4afc-8159-a495849827da" containerName="main"
Sep 11 18:23:56.407798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407597    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="828634bf-7193-4d57-aabf-98a5102993eb" containerName="main"
Sep 11 18:23:56.407798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407605    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="828634bf-7193-4d57-aabf-98a5102993eb" containerName="main"
Sep 11 18:23:56.407798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407622    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd879040-5d7d-4c11-9a8f-9c60ec980169" containerName="maas-api"
Sep 11 18:23:56.407798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407630    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd879040-5d7d-4c11-9a8f-9c60ec980169" containerName="maas-api"
Sep 11 18:23:56.407798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407646    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35248822-aefa-4202-a5d2-da92d6b5fc51" containerName="main"
Sep 11 18:23:56.407798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407654    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="35248822-aefa-4202-a5d2-da92d6b5fc51" containerName="main"
Sep 11 18:23:56.407798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407664    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="221f6e7a-44be-4e66-82db-4b852b2ef7e2" containerName="main"
Sep 11 18:23:56.407798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407671    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="221f6e7a-44be-4e66-82db-4b852b2ef7e2" containerName="main"
Sep 11 18:23:56.407798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407687    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35d12f6a-c0bf-4cdf-ac77-c78e773253c9" containerName="main"
Sep 11 18:23:56.407798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407694    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="35d12f6a-c0bf-4cdf-ac77-c78e773253c9" containerName="main"
Sep 11 18:23:56.408499 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407901    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="82d24234-b1d5-4ec4-abba-aac23e2176f2" containerName="main"
Sep 11 18:23:56.408499 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407923    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="96aa235d-5d81-4486-a2b9-0ca3152bddf7" containerName="main"
Sep 11 18:23:56.408499 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407932    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="221f6e7a-44be-4e66-82db-4b852b2ef7e2" containerName="main"
Sep 11 18:23:56.408499 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407947    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="828634bf-7193-4d57-aabf-98a5102993eb" containerName="main"
Sep 11 18:23:56.408499 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407956    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="35d12f6a-c0bf-4cdf-ac77-c78e773253c9" containerName="main"
Sep 11 18:23:56.408499 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407972    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd879040-5d7d-4c11-9a8f-9c60ec980169" containerName="maas-api"
Sep 11 18:23:56.408499 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407991    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="35248822-aefa-4202-a5d2-da92d6b5fc51" containerName="main"
Sep 11 18:23:56.408499 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.407998    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4727935a-3579-4e9c-a9ba-4d0dec1cf07b" containerName="maas-api"
Sep 11 18:23:56.408499 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.408006    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="05acfa53-02f8-4afc-8159-a495849827da" containerName="main"
Sep 11 18:23:56.408499 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.408021    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1aab02fb-d668-4ef8-8a3d-b1b5c85eb2a5" containerName="maas-api"
Sep 11 18:23:56.412083 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.412056    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7499-5a2c78"
Sep 11 18:23:56.414918 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.414893    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7499-5a2c78"]
Sep 11 18:23:56.415291 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.415265    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q6zqg\""
Sep 11 18:23:56.467443 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.467401    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvxpc\" (UniqueName: \"kubernetes.io/projected/a6bbfd28-3a39-491e-b436-ab8a0ee86c77-kube-api-access-gvxpc\") pod \"test-curl-7499-5a2c78\" (UID: \"a6bbfd28-3a39-491e-b436-ab8a0ee86c77\") " pod="opendatahub/test-curl-7499-5a2c78"
Sep 11 18:23:56.569049 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.569003    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvxpc\" (UniqueName: \"kubernetes.io/projected/a6bbfd28-3a39-491e-b436-ab8a0ee86c77-kube-api-access-gvxpc\") pod \"test-curl-7499-5a2c78\" (UID: \"a6bbfd28-3a39-491e-b436-ab8a0ee86c77\") " pod="opendatahub/test-curl-7499-5a2c78"
Sep 11 18:23:56.578838 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.578810    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvxpc\" (UniqueName: \"kubernetes.io/projected/a6bbfd28-3a39-491e-b436-ab8a0ee86c77-kube-api-access-gvxpc\") pod \"test-curl-7499-5a2c78\" (UID: \"a6bbfd28-3a39-491e-b436-ab8a0ee86c77\") " pod="opendatahub/test-curl-7499-5a2c78"
Sep 11 18:23:56.722958 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.722932    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7499-5a2c78"
Sep 11 18:23:56.995929 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.995906    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7499-5a2c78"]
Sep 11 18:23:56.997988 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:23:56.997958    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6bbfd28_3a39_491e_b436_ab8a0ee86c77.slice/crio-cb151bac6765ed917782e2683a4ad7c3a10aac6489431660cbf56208fe767d09 WatchSource:0}: Error finding container cb151bac6765ed917782e2683a4ad7c3a10aac6489431660cbf56208fe767d09: Status 404 returned error can't find the container with id cb151bac6765ed917782e2683a4ad7c3a10aac6489431660cbf56208fe767d09
Sep 11 18:23:56.999713 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:56.999697    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:23:57.943072 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:57.943036    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7499-5a2c78" event={"ID":"a6bbfd28-3a39-491e-b436-ab8a0ee86c77","Type":"ContainerStarted","Data":"cb151bac6765ed917782e2683a4ad7c3a10aac6489431660cbf56208fe767d09"}
Sep 11 18:23:58.949456 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:58.949429    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7499-5a2c78" event={"ID":"a6bbfd28-3a39-491e-b436-ab8a0ee86c77","Type":"ContainerStarted","Data":"04e012bb89a713b69aeaf92e0d999b733b56d7aa99cecb1c0b54c7f3843f2ab6"}
Sep 11 18:23:58.966299 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:58.966254    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7499-5a2c78" podStartSLOduration=1.097858407 podStartE2EDuration="2.966238643s" totalImagesPullingTime="1.868380236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:58.965185033 +0000 UTC m=+1796.808420497" watchObservedRunningTime="2026-09-11 18:23:58.966238643 +0000 UTC m=+1796.809474087"
Sep 11 18:23:59.550702 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:59.550668    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7499-5a2c78"]
Sep 11 18:23:59.554585 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:59.554560    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7499-5a2c78"]
Sep 11 18:23:59.671091 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:59.671016    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7499-fd3a1b"]
Sep 11 18:23:59.671395 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:59.671383    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6bbfd28-3a39-491e-b436-ab8a0ee86c77" containerName="test-curl-7499-5a2c78"
Sep 11 18:23:59.671438 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:59.671407    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6bbfd28-3a39-491e-b436-ab8a0ee86c77" containerName="test-curl-7499-5a2c78"
Sep 11 18:23:59.671484 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:59.671476    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6bbfd28-3a39-491e-b436-ab8a0ee86c77" containerName="test-curl-7499-5a2c78"
Sep 11 18:23:59.676557 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:59.676284    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7499-fd3a1b"
Sep 11 18:23:59.682147 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:59.682115    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7499-fd3a1b"]
Sep 11 18:23:59.799605 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:59.799572    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl48l\" (UniqueName: \"kubernetes.io/projected/290a1d0a-ef2f-4c32-89cb-d5e754c810d4-kube-api-access-bl48l\") pod \"test-curl-7499-fd3a1b\" (UID: \"290a1d0a-ef2f-4c32-89cb-d5e754c810d4\") " pod="opendatahub/test-curl-7499-fd3a1b"
Sep 11 18:23:59.900588 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:59.900552    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl48l\" (UniqueName: \"kubernetes.io/projected/290a1d0a-ef2f-4c32-89cb-d5e754c810d4-kube-api-access-bl48l\") pod \"test-curl-7499-fd3a1b\" (UID: \"290a1d0a-ef2f-4c32-89cb-d5e754c810d4\") " pod="opendatahub/test-curl-7499-fd3a1b"
Sep 11 18:23:59.909999 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:59.909971    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl48l\" (UniqueName: \"kubernetes.io/projected/290a1d0a-ef2f-4c32-89cb-d5e754c810d4-kube-api-access-bl48l\") pod \"test-curl-7499-fd3a1b\" (UID: \"290a1d0a-ef2f-4c32-89cb-d5e754c810d4\") " pod="opendatahub/test-curl-7499-fd3a1b"
Sep 11 18:23:59.955193 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:23:59.955121    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7499-5a2c78" container="test-curl-7499-5a2c78"
Sep 11 18:23:59.987840 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:23:59.987815    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7499-fd3a1b"
Sep 11 18:24:00.261317 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:00.261293    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7499-fd3a1b"]
Sep 11 18:24:00.262811 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:24:00.262779    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod290a1d0a_ef2f_4c32_89cb_d5e754c810d4.slice/crio-b11a4ee2e3950c3432101cc83d948b7625bb850109fedbea31ed730be68d4bac WatchSource:0}: Error finding container b11a4ee2e3950c3432101cc83d948b7625bb850109fedbea31ed730be68d4bac: Status 404 returned error can't find the container with id b11a4ee2e3950c3432101cc83d948b7625bb850109fedbea31ed730be68d4bac
Sep 11 18:24:00.962581 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:00.962544    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7499-fd3a1b" event={"ID":"290a1d0a-ef2f-4c32-89cb-d5e754c810d4","Type":"ContainerStarted","Data":"55a624f3e833b0071bce90d947c0854ebb00c867a05e4c1c619e6853fc8846c7"}
Sep 11 18:24:00.962982 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:00.962592    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7499-fd3a1b" event={"ID":"290a1d0a-ef2f-4c32-89cb-d5e754c810d4","Type":"ContainerStarted","Data":"b11a4ee2e3950c3432101cc83d948b7625bb850109fedbea31ed730be68d4bac"}
Sep 11 18:24:00.962982 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:00.962716    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7499-5a2c78" podUID="a6bbfd28-3a39-491e-b436-ab8a0ee86c77" containerName="test-curl-7499-5a2c78" containerID="cri-o://04e012bb89a713b69aeaf92e0d999b733b56d7aa99cecb1c0b54c7f3843f2ab6" gracePeriod=2
Sep 11 18:24:00.978619 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:00.978567    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7499-fd3a1b" podStartSLOduration=1.978551801 podStartE2EDuration="1.978551801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:00.976981958 +0000 UTC m=+1798.820217399" watchObservedRunningTime="2026-09-11 18:24:00.978551801 +0000 UTC m=+1798.821787241"
Sep 11 18:24:01.560528 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:01.560478    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7499-fd3a1b"]
Sep 11 18:24:01.566336 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:01.566306    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7499-fd3a1b"]
Sep 11 18:24:01.967951 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:24:01.967923    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7499-fd3a1b" container="test-curl-7499-fd3a1b"
Sep 11 18:24:02.753299 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:02.753269    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/2.log"
Sep 11 18:24:02.761628 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:02.761602    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/2.log"
Sep 11 18:24:02.974287 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:02.974232    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7499-fd3a1b" podUID="290a1d0a-ef2f-4c32-89cb-d5e754c810d4" containerName="test-curl-7499-fd3a1b" containerID="cri-o://55a624f3e833b0071bce90d947c0854ebb00c867a05e4c1c619e6853fc8846c7" gracePeriod=2
Sep 11 18:24:03.156338 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:03.156313    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7499-5a2c78"
Sep 11 18:24:03.234079 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:03.234046    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvxpc\" (UniqueName: \"kubernetes.io/projected/a6bbfd28-3a39-491e-b436-ab8a0ee86c77-kube-api-access-gvxpc\") pod \"a6bbfd28-3a39-491e-b436-ab8a0ee86c77\" (UID: \"a6bbfd28-3a39-491e-b436-ab8a0ee86c77\") "
Sep 11 18:24:03.236639 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:03.236607    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6bbfd28-3a39-491e-b436-ab8a0ee86c77-kube-api-access-gvxpc" (OuterVolumeSpecName: "kube-api-access-gvxpc") pod "a6bbfd28-3a39-491e-b436-ab8a0ee86c77" (UID: "a6bbfd28-3a39-491e-b436-ab8a0ee86c77"). InnerVolumeSpecName "kube-api-access-gvxpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:24:03.335058 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:03.334974    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvxpc\" (UniqueName: \"kubernetes.io/projected/a6bbfd28-3a39-491e-b436-ab8a0ee86c77-kube-api-access-gvxpc\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:24:03.980265 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:03.980227    2560 generic.go:356] "Generic (PLEG): container finished" podID="a6bbfd28-3a39-491e-b436-ab8a0ee86c77" containerID="04e012bb89a713b69aeaf92e0d999b733b56d7aa99cecb1c0b54c7f3843f2ab6" exitCode=137
Sep 11 18:24:03.980675 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:03.980272    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7499-5a2c78"
Sep 11 18:24:03.980675 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:03.980299    2560 scope.go:117] "RemoveContainer" containerID="04e012bb89a713b69aeaf92e0d999b733b56d7aa99cecb1c0b54c7f3843f2ab6"
Sep 11 18:24:03.989598 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:03.989576    2560 scope.go:117] "RemoveContainer" containerID="04e012bb89a713b69aeaf92e0d999b733b56d7aa99cecb1c0b54c7f3843f2ab6"
Sep 11 18:24:03.989906 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:24:03.989886    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e012bb89a713b69aeaf92e0d999b733b56d7aa99cecb1c0b54c7f3843f2ab6\": container with ID starting with 04e012bb89a713b69aeaf92e0d999b733b56d7aa99cecb1c0b54c7f3843f2ab6 not found: ID does not exist" containerID="04e012bb89a713b69aeaf92e0d999b733b56d7aa99cecb1c0b54c7f3843f2ab6"
Sep 11 18:24:03.989981 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:03.989914    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e012bb89a713b69aeaf92e0d999b733b56d7aa99cecb1c0b54c7f3843f2ab6"} err="failed to get container status \"04e012bb89a713b69aeaf92e0d999b733b56d7aa99cecb1c0b54c7f3843f2ab6\": rpc error: code = NotFound desc = could not find container \"04e012bb89a713b69aeaf92e0d999b733b56d7aa99cecb1c0b54c7f3843f2ab6\": container with ID starting with 04e012bb89a713b69aeaf92e0d999b733b56d7aa99cecb1c0b54c7f3843f2ab6 not found: ID does not exist"
Sep 11 18:24:03.991200 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:24:03.991179    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7499-5a2c78\" is forbidden: User \"system:node:ip-10-0-128-29.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-29.ec2.internal' and this object" podUID="a6bbfd28-3a39-491e-b436-ab8a0ee86c77" pod="opendatahub/test-curl-7499-5a2c78"
Sep 11 18:24:04.687809 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:04.687740    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6bbfd28-3a39-491e-b436-ab8a0ee86c77" path="/var/lib/kubelet/pods/a6bbfd28-3a39-491e-b436-ab8a0ee86c77/volumes"
Sep 11 18:24:05.160090 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:05.160067    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7499-fd3a1b"
Sep 11 18:24:05.253992 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:05.253913    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl48l\" (UniqueName: \"kubernetes.io/projected/290a1d0a-ef2f-4c32-89cb-d5e754c810d4-kube-api-access-bl48l\") pod \"290a1d0a-ef2f-4c32-89cb-d5e754c810d4\" (UID: \"290a1d0a-ef2f-4c32-89cb-d5e754c810d4\") "
Sep 11 18:24:05.256169 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:05.256136    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290a1d0a-ef2f-4c32-89cb-d5e754c810d4-kube-api-access-bl48l" (OuterVolumeSpecName: "kube-api-access-bl48l") pod "290a1d0a-ef2f-4c32-89cb-d5e754c810d4" (UID: "290a1d0a-ef2f-4c32-89cb-d5e754c810d4"). InnerVolumeSpecName "kube-api-access-bl48l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:24:05.355093 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:05.355065    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl48l\" (UniqueName: \"kubernetes.io/projected/290a1d0a-ef2f-4c32-89cb-d5e754c810d4-kube-api-access-bl48l\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:24:05.992401 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:05.992366    2560 generic.go:356] "Generic (PLEG): container finished" podID="290a1d0a-ef2f-4c32-89cb-d5e754c810d4" containerID="55a624f3e833b0071bce90d947c0854ebb00c867a05e4c1c619e6853fc8846c7" exitCode=137
Sep 11 18:24:05.992643 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:05.992414    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7499-fd3a1b"
Sep 11 18:24:05.992643 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:05.992431    2560 scope.go:117] "RemoveContainer" containerID="55a624f3e833b0071bce90d947c0854ebb00c867a05e4c1c619e6853fc8846c7"
Sep 11 18:24:06.001447 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:06.001429    2560 scope.go:117] "RemoveContainer" containerID="55a624f3e833b0071bce90d947c0854ebb00c867a05e4c1c619e6853fc8846c7"
Sep 11 18:24:06.001687 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:24:06.001670    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a624f3e833b0071bce90d947c0854ebb00c867a05e4c1c619e6853fc8846c7\": container with ID starting with 55a624f3e833b0071bce90d947c0854ebb00c867a05e4c1c619e6853fc8846c7 not found: ID does not exist" containerID="55a624f3e833b0071bce90d947c0854ebb00c867a05e4c1c619e6853fc8846c7"
Sep 11 18:24:06.001774 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:06.001693    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a624f3e833b0071bce90d947c0854ebb00c867a05e4c1c619e6853fc8846c7"} err="failed to get container status \"55a624f3e833b0071bce90d947c0854ebb00c867a05e4c1c619e6853fc8846c7\": rpc error: code = NotFound desc = could not find container \"55a624f3e833b0071bce90d947c0854ebb00c867a05e4c1c619e6853fc8846c7\": container with ID starting with 55a624f3e833b0071bce90d947c0854ebb00c867a05e4c1c619e6853fc8846c7 not found: ID does not exist"
Sep 11 18:24:06.003076 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:24:06.003053    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7499-fd3a1b\" is forbidden: User \"system:node:ip-10-0-128-29.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-29.ec2.internal' and this object" podUID="290a1d0a-ef2f-4c32-89cb-d5e754c810d4" pod="opendatahub/test-curl-7499-fd3a1b"
Sep 11 18:24:06.687001 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:06.686972    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="290a1d0a-ef2f-4c32-89cb-d5e754c810d4" path="/var/lib/kubelet/pods/290a1d0a-ef2f-4c32-89cb-d5e754c810d4/volumes"
Sep 11 18:24:07.913023 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:07.912990    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:24:25.421791 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:24:25.421741    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:25:02.322999 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:25:02.322960    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:25:17.622348 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:25:17.622316    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:25:27.111191 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:25:27.111115    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:25:37.408031 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:25:37.407996    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:25:50.705179 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:25:50.705144    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:26:10.639581 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:10.639547    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"]
Sep 11 18:26:10.640198 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:10.640177    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="290a1d0a-ef2f-4c32-89cb-d5e754c810d4" containerName="test-curl-7499-fd3a1b"
Sep 11 18:26:10.640244 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:10.640204    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="290a1d0a-ef2f-4c32-89cb-d5e754c810d4" containerName="test-curl-7499-fd3a1b"
Sep 11 18:26:10.640368 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:10.640354    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="290a1d0a-ef2f-4c32-89cb-d5e754c810d4" containerName="test-curl-7499-fd3a1b"
Sep 11 18:26:10.642717 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:10.642697    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"
Sep 11 18:26:10.645461 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:10.645439    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-74a302-serving-cert\""
Sep 11 18:26:10.652360 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:10.651656    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"]
Sep 11 18:26:10.752027 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:10.751985    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n2tv\" (UniqueName: \"kubernetes.io/projected/7efa298b-9c9e-4411-9924-188267b3d93c-kube-api-access-6n2tv\") pod \"maas-api-e2e-ipp-b-74a302-56947db679-m29b2\" (UID: \"7efa298b-9c9e-4411-9924-188267b3d93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"
Sep 11 18:26:10.752027 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:10.752026    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7efa298b-9c9e-4411-9924-188267b3d93c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-74a302-56947db679-m29b2\" (UID: \"7efa298b-9c9e-4411-9924-188267b3d93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"
Sep 11 18:26:10.752251 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:10.752044    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7efa298b-9c9e-4411-9924-188267b3d93c-metrics-certs\") pod \"maas-api-e2e-ipp-b-74a302-56947db679-m29b2\" (UID: \"7efa298b-9c9e-4411-9924-188267b3d93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"
Sep 11 18:26:10.752251 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:10.752114    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7efa298b-9c9e-4411-9924-188267b3d93c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-74a302-56947db679-m29b2\" (UID: \"7efa298b-9c9e-4411-9924-188267b3d93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"
Sep 11 18:26:10.853446 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:10.853412    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n2tv\" (UniqueName: \"kubernetes.io/projected/7efa298b-9c9e-4411-9924-188267b3d93c-kube-api-access-6n2tv\") pod \"maas-api-e2e-ipp-b-74a302-56947db679-m29b2\" (UID: \"7efa298b-9c9e-4411-9924-188267b3d93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"
Sep 11 18:26:10.853446 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:10.853447    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7efa298b-9c9e-4411-9924-188267b3d93c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-74a302-56947db679-m29b2\" (UID: \"7efa298b-9c9e-4411-9924-188267b3d93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"
Sep 11 18:26:10.853647 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:10.853465    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7efa298b-9c9e-4411-9924-188267b3d93c-metrics-certs\") pod \"maas-api-e2e-ipp-b-74a302-56947db679-m29b2\" (UID: \"7efa298b-9c9e-4411-9924-188267b3d93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"
Sep 11 18:26:10.853647 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:10.853485    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7efa298b-9c9e-4411-9924-188267b3d93c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-74a302-56947db679-m29b2\" (UID: \"7efa298b-9c9e-4411-9924-188267b3d93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"
Sep 11 18:26:10.853647 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:26:10.853628    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-serving-cert: secret "maas-api-e2e-ipp-b-74a302-serving-cert" not found
Sep 11 18:26:10.853800 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:26:10.853703    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7efa298b-9c9e-4411-9924-188267b3d93c-maas-api-tls podName:7efa298b-9c9e-4411-9924-188267b3d93c nodeName:}" failed. No retries permitted until 2026-09-11 18:26:11.353683135 +0000 UTC m=+1929.196918554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7efa298b-9c9e-4411-9924-188267b3d93c-maas-api-tls") pod "maas-api-e2e-ipp-b-74a302-56947db679-m29b2" (UID: "7efa298b-9c9e-4411-9924-188267b3d93c") : secret "maas-api-e2e-ipp-b-74a302-serving-cert" not found
Sep 11 18:26:10.854234 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:10.854213    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7efa298b-9c9e-4411-9924-188267b3d93c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-74a302-56947db679-m29b2\" (UID: \"7efa298b-9c9e-4411-9924-188267b3d93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"
Sep 11 18:26:10.856056 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:10.856038    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7efa298b-9c9e-4411-9924-188267b3d93c-metrics-certs\") pod \"maas-api-e2e-ipp-b-74a302-56947db679-m29b2\" (UID: \"7efa298b-9c9e-4411-9924-188267b3d93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"
Sep 11 18:26:10.864995 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:10.864970    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n2tv\" (UniqueName: \"kubernetes.io/projected/7efa298b-9c9e-4411-9924-188267b3d93c-kube-api-access-6n2tv\") pod \"maas-api-e2e-ipp-b-74a302-56947db679-m29b2\" (UID: \"7efa298b-9c9e-4411-9924-188267b3d93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"
Sep 11 18:26:11.358292 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:11.358261    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7efa298b-9c9e-4411-9924-188267b3d93c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-74a302-56947db679-m29b2\" (UID: \"7efa298b-9c9e-4411-9924-188267b3d93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"
Sep 11 18:26:11.360901 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:11.360883    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7efa298b-9c9e-4411-9924-188267b3d93c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-74a302-56947db679-m29b2\" (UID: \"7efa298b-9c9e-4411-9924-188267b3d93c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"
Sep 11 18:26:11.555780 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:11.555705    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"
Sep 11 18:26:11.827266 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:11.827235    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"]
Sep 11 18:26:11.828652 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:26:11.828626    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7efa298b_9c9e_4411_9924_188267b3d93c.slice/crio-213adcafb2c31ae7cd9f9af1872cb515adcd9e822d932983e827d0f15907bd84 WatchSource:0}: Error finding container 213adcafb2c31ae7cd9f9af1872cb515adcd9e822d932983e827d0f15907bd84: Status 404 returned error can't find the container with id 213adcafb2c31ae7cd9f9af1872cb515adcd9e822d932983e827d0f15907bd84
Sep 11 18:26:12.629783 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:12.629668    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2" event={"ID":"7efa298b-9c9e-4411-9924-188267b3d93c","Type":"ContainerStarted","Data":"031f2b28b2a5e059bc1641d6749e8ff9d3c604d74c0ba81e91d5544e91528720"}
Sep 11 18:26:12.629783 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:12.629723    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2" event={"ID":"7efa298b-9c9e-4411-9924-188267b3d93c","Type":"ContainerStarted","Data":"213adcafb2c31ae7cd9f9af1872cb515adcd9e822d932983e827d0f15907bd84"}
Sep 11 18:26:12.629983 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:12.629784    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"
Sep 11 18:26:12.648153 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:12.648099    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2" podStartSLOduration=2.247701832 podStartE2EDuration="2.648083301s" totalImagesPullingTime="400.381469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:26:12.646985187 +0000 UTC m=+1930.490220629" watchObservedRunningTime="2026-09-11 18:26:12.648083301 +0000 UTC m=+1930.491318739"
Sep 11 18:26:15.913226 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:15.913188    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:26:18.640839 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:18.640812    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"
Sep 11 18:26:44.896320 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:44.896290    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"]
Sep 11 18:26:44.899990 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:44.899972    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:44.904112 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:44.904090    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-be394b\"/\"openshift-service-ca.crt\""
Sep 11 18:26:44.904112 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:44.904095    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-be394b\"/\"ipp-route-be394b-kserve-self-signed-certs\""
Sep 11 18:26:44.904298 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:44.904111    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-be394b\"/\"default-dockercfg-nl8qz\""
Sep 11 18:26:44.904298 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:44.904116    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-be394b\"/\"kube-root-ca.crt\""
Sep 11 18:26:44.911784 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:44.911030    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"]
Sep 11 18:26:44.968884 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:44.968853    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-model-cache\") pod \"ipp-route-be394b-kserve-64745dddf5-rg45r\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") " pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:44.969058 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:44.968889    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-home\") pod \"ipp-route-be394b-kserve-64745dddf5-rg45r\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") " pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:44.969058 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:44.968915    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn5pl\" (UniqueName: \"kubernetes.io/projected/5ced1884-b4ca-45f9-9000-a25780703893-kube-api-access-zn5pl\") pod \"ipp-route-be394b-kserve-64745dddf5-rg45r\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") " pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:44.969058 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:44.968939    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-dshm\") pod \"ipp-route-be394b-kserve-64745dddf5-rg45r\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") " pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:44.969058 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:44.968996    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-tmp-dir\") pod \"ipp-route-be394b-kserve-64745dddf5-rg45r\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") " pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:44.969058 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:44.969024    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ced1884-b4ca-45f9-9000-a25780703893-tls-certs\") pod \"ipp-route-be394b-kserve-64745dddf5-rg45r\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") " pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:45.070387 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:45.070350    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-tmp-dir\") pod \"ipp-route-be394b-kserve-64745dddf5-rg45r\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") " pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:45.070578 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:45.070402    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ced1884-b4ca-45f9-9000-a25780703893-tls-certs\") pod \"ipp-route-be394b-kserve-64745dddf5-rg45r\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") " pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:45.070578 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:45.070445    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-model-cache\") pod \"ipp-route-be394b-kserve-64745dddf5-rg45r\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") " pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:45.070578 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:45.070463    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-home\") pod \"ipp-route-be394b-kserve-64745dddf5-rg45r\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") " pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:45.070578 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:45.070491    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn5pl\" (UniqueName: \"kubernetes.io/projected/5ced1884-b4ca-45f9-9000-a25780703893-kube-api-access-zn5pl\") pod \"ipp-route-be394b-kserve-64745dddf5-rg45r\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") " pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:45.070578 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:45.070511    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-dshm\") pod \"ipp-route-be394b-kserve-64745dddf5-rg45r\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") " pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:45.070893 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:45.070840    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-tmp-dir\") pod \"ipp-route-be394b-kserve-64745dddf5-rg45r\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") " pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:45.070893 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:45.070868    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-model-cache\") pod \"ipp-route-be394b-kserve-64745dddf5-rg45r\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") " pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:45.070964 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:45.070887    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-home\") pod \"ipp-route-be394b-kserve-64745dddf5-rg45r\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") " pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:45.072952 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:45.072933    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-dshm\") pod \"ipp-route-be394b-kserve-64745dddf5-rg45r\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") " pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:45.073201 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:45.073183    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ced1884-b4ca-45f9-9000-a25780703893-tls-certs\") pod \"ipp-route-be394b-kserve-64745dddf5-rg45r\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") " pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:45.078578 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:45.078556    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn5pl\" (UniqueName: \"kubernetes.io/projected/5ced1884-b4ca-45f9-9000-a25780703893-kube-api-access-zn5pl\") pod \"ipp-route-be394b-kserve-64745dddf5-rg45r\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") " pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:45.216722 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:45.216688    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:45.489742 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:45.489712    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"]
Sep 11 18:26:45.491347 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:26:45.491322    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ced1884_b4ca_45f9_9000_a25780703893.slice/crio-26002d3c1a7ccc9fb72699fafcc989be69295b603a4409e23f13ff81ae01aa50 WatchSource:0}: Error finding container 26002d3c1a7ccc9fb72699fafcc989be69295b603a4409e23f13ff81ae01aa50: Status 404 returned error can't find the container with id 26002d3c1a7ccc9fb72699fafcc989be69295b603a4409e23f13ff81ae01aa50
Sep 11 18:26:45.792356 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:45.792265    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r" event={"ID":"5ced1884-b4ca-45f9-9000-a25780703893","Type":"ContainerStarted","Data":"ee2bb9446b6e03644fb3ef893bbac250c65663aca53b7dc5454b0a367a35073f"}
Sep 11 18:26:45.792356 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:45.792313    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r" event={"ID":"5ced1884-b4ca-45f9-9000-a25780703893","Type":"ContainerStarted","Data":"26002d3c1a7ccc9fb72699fafcc989be69295b603a4409e23f13ff81ae01aa50"}
Sep 11 18:26:45.812480 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:45.812431    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r" podStartSLOduration=1.812415526 podStartE2EDuration="1.812415526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:26:45.810199603 +0000 UTC m=+1963.653435052" watchObservedRunningTime="2026-09-11 18:26:45.812415526 +0000 UTC m=+1963.655650966"
Sep 11 18:26:48.712510 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:48.712473    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:26:55.217718 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:55.217633    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:55.218240 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:55.217792    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:55.230818 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:55.230789    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:26:55.856012 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:26:55.855979    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:27:03.413692 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:03.413660    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:27:16.576547 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:16.576508    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"]
Sep 11 18:27:16.580245 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:16.580225    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"
Sep 11 18:27:16.583108 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:16.583089    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-b2aace-serving-cert\""
Sep 11 18:27:16.590737 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:16.590714    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"]
Sep 11 18:27:16.679405 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:16.679372    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb61a39-a94b-4a47-9967-3a78f325897c-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686\" (UID: \"2fb61a39-a94b-4a47-9967-3a78f325897c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"
Sep 11 18:27:16.679550 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:16.679412    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fb61a39-a94b-4a47-9967-3a78f325897c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686\" (UID: \"2fb61a39-a94b-4a47-9967-3a78f325897c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"
Sep 11 18:27:16.679550 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:16.679439    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhjbl\" (UniqueName: \"kubernetes.io/projected/2fb61a39-a94b-4a47-9967-3a78f325897c-kube-api-access-xhjbl\") pod \"maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686\" (UID: \"2fb61a39-a94b-4a47-9967-3a78f325897c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"
Sep 11 18:27:16.679550 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:16.679484    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fb61a39-a94b-4a47-9967-3a78f325897c-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686\" (UID: \"2fb61a39-a94b-4a47-9967-3a78f325897c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"
Sep 11 18:27:16.780150 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:16.780117    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb61a39-a94b-4a47-9967-3a78f325897c-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686\" (UID: \"2fb61a39-a94b-4a47-9967-3a78f325897c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"
Sep 11 18:27:16.780321 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:16.780165    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fb61a39-a94b-4a47-9967-3a78f325897c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686\" (UID: \"2fb61a39-a94b-4a47-9967-3a78f325897c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"
Sep 11 18:27:16.780321 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:27:16.780246    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-serving-cert: secret "maas-api-e2e-ipp-cleanup-b2aace-serving-cert" not found
Sep 11 18:27:16.780321 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:27:16.780307    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fb61a39-a94b-4a47-9967-3a78f325897c-maas-api-tls podName:2fb61a39-a94b-4a47-9967-3a78f325897c nodeName:}" failed. No retries permitted until 2026-09-11 18:27:17.280288527 +0000 UTC m=+1995.123523967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2fb61a39-a94b-4a47-9967-3a78f325897c-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686" (UID: "2fb61a39-a94b-4a47-9967-3a78f325897c") : secret "maas-api-e2e-ipp-cleanup-b2aace-serving-cert" not found
Sep 11 18:27:16.780581 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:16.780346    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhjbl\" (UniqueName: \"kubernetes.io/projected/2fb61a39-a94b-4a47-9967-3a78f325897c-kube-api-access-xhjbl\") pod \"maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686\" (UID: \"2fb61a39-a94b-4a47-9967-3a78f325897c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"
Sep 11 18:27:16.780581 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:16.780406    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fb61a39-a94b-4a47-9967-3a78f325897c-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686\" (UID: \"2fb61a39-a94b-4a47-9967-3a78f325897c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"
Sep 11 18:27:16.780940 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:16.780913    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb61a39-a94b-4a47-9967-3a78f325897c-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686\" (UID: \"2fb61a39-a94b-4a47-9967-3a78f325897c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"
Sep 11 18:27:16.782892 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:16.782867    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fb61a39-a94b-4a47-9967-3a78f325897c-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686\" (UID: \"2fb61a39-a94b-4a47-9967-3a78f325897c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"
Sep 11 18:27:16.789024 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:16.789005    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhjbl\" (UniqueName: \"kubernetes.io/projected/2fb61a39-a94b-4a47-9967-3a78f325897c-kube-api-access-xhjbl\") pod \"maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686\" (UID: \"2fb61a39-a94b-4a47-9967-3a78f325897c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"
Sep 11 18:27:17.286207 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:17.286171    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fb61a39-a94b-4a47-9967-3a78f325897c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686\" (UID: \"2fb61a39-a94b-4a47-9967-3a78f325897c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"
Sep 11 18:27:17.288733 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:17.288711    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fb61a39-a94b-4a47-9967-3a78f325897c-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686\" (UID: \"2fb61a39-a94b-4a47-9967-3a78f325897c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"
Sep 11 18:27:17.491650 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:17.491617    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"
Sep 11 18:27:17.764818 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:17.764789    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"]
Sep 11 18:27:17.766167 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:27:17.766143    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb61a39_a94b_4a47_9967_3a78f325897c.slice/crio-cc42a501b667055f0800b9eec9c58566f56e88c8974bb0e6a507c14278b5bf75 WatchSource:0}: Error finding container cc42a501b667055f0800b9eec9c58566f56e88c8974bb0e6a507c14278b5bf75: Status 404 returned error can't find the container with id cc42a501b667055f0800b9eec9c58566f56e88c8974bb0e6a507c14278b5bf75
Sep 11 18:27:17.951975 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:17.951884    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686" event={"ID":"2fb61a39-a94b-4a47-9967-3a78f325897c","Type":"ContainerStarted","Data":"cc42a501b667055f0800b9eec9c58566f56e88c8974bb0e6a507c14278b5bf75"}
Sep 11 18:27:18.958685 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:18.958653    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686" event={"ID":"2fb61a39-a94b-4a47-9967-3a78f325897c","Type":"ContainerStarted","Data":"c28103dd92e378fdbdcca91473bd78219fc6a6fcbe5b470e51d89663b49804e1"}
Sep 11 18:27:18.959127 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:18.958802    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"
Sep 11 18:27:18.978112 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:18.978057    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686" podStartSLOduration=2.572204555 podStartE2EDuration="2.978038505s" totalImagesPullingTime="405.83395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:18.977559666 +0000 UTC m=+1996.820795107" watchObservedRunningTime="2026-09-11 18:27:18.978038505 +0000 UTC m=+1996.821273946"
Sep 11 18:27:23.117026 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:23.116988    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:27:24.969806 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:24.969776    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"
Sep 11 18:27:55.410260 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:55.410227    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"]
Sep 11 18:27:55.410795 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:55.410454    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686" podUID="2fb61a39-a94b-4a47-9967-3a78f325897c" containerName="maas-api" containerID="cri-o://c28103dd92e378fdbdcca91473bd78219fc6a6fcbe5b470e51d89663b49804e1" gracePeriod=30
Sep 11 18:27:55.658101 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:55.658075    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"
Sep 11 18:27:55.746094 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:55.746064    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhjbl\" (UniqueName: \"kubernetes.io/projected/2fb61a39-a94b-4a47-9967-3a78f325897c-kube-api-access-xhjbl\") pod \"2fb61a39-a94b-4a47-9967-3a78f325897c\" (UID: \"2fb61a39-a94b-4a47-9967-3a78f325897c\") "
Sep 11 18:27:55.746275 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:55.746146    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fb61a39-a94b-4a47-9967-3a78f325897c-maas-api-tls\") pod \"2fb61a39-a94b-4a47-9967-3a78f325897c\" (UID: \"2fb61a39-a94b-4a47-9967-3a78f325897c\") "
Sep 11 18:27:55.746275 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:55.746168    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fb61a39-a94b-4a47-9967-3a78f325897c-metrics-certs\") pod \"2fb61a39-a94b-4a47-9967-3a78f325897c\" (UID: \"2fb61a39-a94b-4a47-9967-3a78f325897c\") "
Sep 11 18:27:55.746275 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:55.746208    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb61a39-a94b-4a47-9967-3a78f325897c-openshift-service-ca\") pod \"2fb61a39-a94b-4a47-9967-3a78f325897c\" (UID: \"2fb61a39-a94b-4a47-9967-3a78f325897c\") "
Sep 11 18:27:55.746626 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:55.746599    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fb61a39-a94b-4a47-9967-3a78f325897c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2fb61a39-a94b-4a47-9967-3a78f325897c" (UID: "2fb61a39-a94b-4a47-9967-3a78f325897c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:27:55.748397 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:55.748371    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb61a39-a94b-4a47-9967-3a78f325897c-kube-api-access-xhjbl" (OuterVolumeSpecName: "kube-api-access-xhjbl") pod "2fb61a39-a94b-4a47-9967-3a78f325897c" (UID: "2fb61a39-a94b-4a47-9967-3a78f325897c"). InnerVolumeSpecName "kube-api-access-xhjbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:27:55.748872 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:55.748848    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb61a39-a94b-4a47-9967-3a78f325897c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2fb61a39-a94b-4a47-9967-3a78f325897c" (UID: "2fb61a39-a94b-4a47-9967-3a78f325897c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:27:55.748945 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:55.748868    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fb61a39-a94b-4a47-9967-3a78f325897c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2fb61a39-a94b-4a47-9967-3a78f325897c" (UID: "2fb61a39-a94b-4a47-9967-3a78f325897c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:27:55.848219 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:55.848183    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fb61a39-a94b-4a47-9967-3a78f325897c-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:27:55.848219 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:55.848223    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fb61a39-a94b-4a47-9967-3a78f325897c-metrics-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:27:55.848396 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:55.848238    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fb61a39-a94b-4a47-9967-3a78f325897c-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:27:55.848396 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:55.848252    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhjbl\" (UniqueName: \"kubernetes.io/projected/2fb61a39-a94b-4a47-9967-3a78f325897c-kube-api-access-xhjbl\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:27:56.146285 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:56.146188    2560 generic.go:356] "Generic (PLEG): container finished" podID="2fb61a39-a94b-4a47-9967-3a78f325897c" containerID="c28103dd92e378fdbdcca91473bd78219fc6a6fcbe5b470e51d89663b49804e1" exitCode=0
Sep 11 18:27:56.146285 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:56.146236    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"
Sep 11 18:27:56.146485 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:56.146277    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686" event={"ID":"2fb61a39-a94b-4a47-9967-3a78f325897c","Type":"ContainerDied","Data":"c28103dd92e378fdbdcca91473bd78219fc6a6fcbe5b470e51d89663b49804e1"}
Sep 11 18:27:56.146485 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:56.146340    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686" event={"ID":"2fb61a39-a94b-4a47-9967-3a78f325897c","Type":"ContainerDied","Data":"cc42a501b667055f0800b9eec9c58566f56e88c8974bb0e6a507c14278b5bf75"}
Sep 11 18:27:56.146485 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:56.146368    2560 scope.go:117] "RemoveContainer" containerID="c28103dd92e378fdbdcca91473bd78219fc6a6fcbe5b470e51d89663b49804e1"
Sep 11 18:27:56.156082 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:56.156066    2560 scope.go:117] "RemoveContainer" containerID="c28103dd92e378fdbdcca91473bd78219fc6a6fcbe5b470e51d89663b49804e1"
Sep 11 18:27:56.156354 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:27:56.156334    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c28103dd92e378fdbdcca91473bd78219fc6a6fcbe5b470e51d89663b49804e1\": container with ID starting with c28103dd92e378fdbdcca91473bd78219fc6a6fcbe5b470e51d89663b49804e1 not found: ID does not exist" containerID="c28103dd92e378fdbdcca91473bd78219fc6a6fcbe5b470e51d89663b49804e1"
Sep 11 18:27:56.156426 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:56.156366    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c28103dd92e378fdbdcca91473bd78219fc6a6fcbe5b470e51d89663b49804e1"} err="failed to get container status \"c28103dd92e378fdbdcca91473bd78219fc6a6fcbe5b470e51d89663b49804e1\": rpc error: code = NotFound desc = could not find container \"c28103dd92e378fdbdcca91473bd78219fc6a6fcbe5b470e51d89663b49804e1\": container with ID starting with c28103dd92e378fdbdcca91473bd78219fc6a6fcbe5b470e51d89663b49804e1 not found: ID does not exist"
Sep 11 18:27:56.167517 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:56.167491    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"]
Sep 11 18:27:56.171274 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:56.171253    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-b2aace-74f464579b-n2686"]
Sep 11 18:27:56.688343 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:27:56.688314    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fb61a39-a94b-4a47-9967-3a78f325897c" path="/var/lib/kubelet/pods/2fb61a39-a94b-4a47-9967-3a78f325897c/volumes"
Sep 11 18:28:00.108878 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:00.108836    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:28:19.217357 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:19.217318    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:28:27.267919 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:27.267836    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"]
Sep 11 18:28:32.016090 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:32.016051    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:28:42.288255 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:42.288209    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r" podUID="5ced1884-b4ca-45f9-9000-a25780703893" containerName="main" containerID="cri-o://ee2bb9446b6e03644fb3ef893bbac250c65663aca53b7dc5454b0a367a35073f" gracePeriod=45
Sep 11 18:28:42.544640 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:42.544578    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:28:42.707236 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:42.707204    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-tmp-dir\") pod \"5ced1884-b4ca-45f9-9000-a25780703893\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") "
Sep 11 18:28:42.707424 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:42.707244    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn5pl\" (UniqueName: \"kubernetes.io/projected/5ced1884-b4ca-45f9-9000-a25780703893-kube-api-access-zn5pl\") pod \"5ced1884-b4ca-45f9-9000-a25780703893\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") "
Sep 11 18:28:42.707424 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:42.707274    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-home\") pod \"5ced1884-b4ca-45f9-9000-a25780703893\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") "
Sep 11 18:28:42.707424 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:42.707294    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ced1884-b4ca-45f9-9000-a25780703893-tls-certs\") pod \"5ced1884-b4ca-45f9-9000-a25780703893\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") "
Sep 11 18:28:42.707424 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:42.707313    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-dshm\") pod \"5ced1884-b4ca-45f9-9000-a25780703893\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") "
Sep 11 18:28:42.707424 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:42.707347    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-model-cache\") pod \"5ced1884-b4ca-45f9-9000-a25780703893\" (UID: \"5ced1884-b4ca-45f9-9000-a25780703893\") "
Sep 11 18:28:42.707708 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:42.707528    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5ced1884-b4ca-45f9-9000-a25780703893" (UID: "5ced1884-b4ca-45f9-9000-a25780703893"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:28:42.707708 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:42.707580    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-home" (OuterVolumeSpecName: "home") pod "5ced1884-b4ca-45f9-9000-a25780703893" (UID: "5ced1884-b4ca-45f9-9000-a25780703893"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:28:42.707708 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:42.707670    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-model-cache" (OuterVolumeSpecName: "model-cache") pod "5ced1884-b4ca-45f9-9000-a25780703893" (UID: "5ced1884-b4ca-45f9-9000-a25780703893"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:28:42.707875 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:42.707839    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-tmp-dir\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:28:42.707875 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:42.707854    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-home\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:28:42.707875 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:42.707864    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-model-cache\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:28:42.709798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:42.709777    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-dshm" (OuterVolumeSpecName: "dshm") pod "5ced1884-b4ca-45f9-9000-a25780703893" (UID: "5ced1884-b4ca-45f9-9000-a25780703893"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:28:42.709798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:42.709784    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ced1884-b4ca-45f9-9000-a25780703893-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5ced1884-b4ca-45f9-9000-a25780703893" (UID: "5ced1884-b4ca-45f9-9000-a25780703893"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:28:42.709918 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:42.709846    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ced1884-b4ca-45f9-9000-a25780703893-kube-api-access-zn5pl" (OuterVolumeSpecName: "kube-api-access-zn5pl") pod "5ced1884-b4ca-45f9-9000-a25780703893" (UID: "5ced1884-b4ca-45f9-9000-a25780703893"). InnerVolumeSpecName "kube-api-access-zn5pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:28:42.808824 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:42.808712    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn5pl\" (UniqueName: \"kubernetes.io/projected/5ced1884-b4ca-45f9-9000-a25780703893-kube-api-access-zn5pl\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:28:42.808824 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:42.808745    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ced1884-b4ca-45f9-9000-a25780703893-tls-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:28:42.808824 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:42.808782    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ced1884-b4ca-45f9-9000-a25780703893-dshm\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:28:43.382898 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:43.382861    2560 generic.go:356] "Generic (PLEG): container finished" podID="5ced1884-b4ca-45f9-9000-a25780703893" containerID="ee2bb9446b6e03644fb3ef893bbac250c65663aca53b7dc5454b0a367a35073f" exitCode=0
Sep 11 18:28:43.383301 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:43.382910    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"
Sep 11 18:28:43.383301 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:43.382941    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r" event={"ID":"5ced1884-b4ca-45f9-9000-a25780703893","Type":"ContainerDied","Data":"ee2bb9446b6e03644fb3ef893bbac250c65663aca53b7dc5454b0a367a35073f"}
Sep 11 18:28:43.383301 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:43.382986    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r" event={"ID":"5ced1884-b4ca-45f9-9000-a25780703893","Type":"ContainerDied","Data":"26002d3c1a7ccc9fb72699fafcc989be69295b603a4409e23f13ff81ae01aa50"}
Sep 11 18:28:43.383301 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:43.383003    2560 scope.go:117] "RemoveContainer" containerID="ee2bb9446b6e03644fb3ef893bbac250c65663aca53b7dc5454b0a367a35073f"
Sep 11 18:28:43.400186 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:43.400165    2560 scope.go:117] "RemoveContainer" containerID="ee2bb9446b6e03644fb3ef893bbac250c65663aca53b7dc5454b0a367a35073f"
Sep 11 18:28:43.400477 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:28:43.400459    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee2bb9446b6e03644fb3ef893bbac250c65663aca53b7dc5454b0a367a35073f\": container with ID starting with ee2bb9446b6e03644fb3ef893bbac250c65663aca53b7dc5454b0a367a35073f not found: ID does not exist" containerID="ee2bb9446b6e03644fb3ef893bbac250c65663aca53b7dc5454b0a367a35073f"
Sep 11 18:28:43.400538 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:43.400484    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee2bb9446b6e03644fb3ef893bbac250c65663aca53b7dc5454b0a367a35073f"} err="failed to get container status \"ee2bb9446b6e03644fb3ef893bbac250c65663aca53b7dc5454b0a367a35073f\": rpc error: code = NotFound desc = could not find container \"ee2bb9446b6e03644fb3ef893bbac250c65663aca53b7dc5454b0a367a35073f\": container with ID starting with ee2bb9446b6e03644fb3ef893bbac250c65663aca53b7dc5454b0a367a35073f not found: ID does not exist"
Sep 11 18:28:43.408527 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:43.408504    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"]
Sep 11 18:28:43.412035 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:43.412013    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-be394b/ipp-route-be394b-kserve-64745dddf5-rg45r"]
Sep 11 18:28:44.687592 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:28:44.687560    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ced1884-b4ca-45f9-9000-a25780703893" path="/var/lib/kubelet/pods/5ced1884-b4ca-45f9-9000-a25780703893/volumes"
Sep 11 18:29:02.787568 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:02.787539    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/2.log"
Sep 11 18:29:02.794654 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:02.794625    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/2.log"
Sep 11 18:29:27.709862 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:27.709828    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"]
Sep 11 18:29:27.710370 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:27.710034    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2" podUID="7efa298b-9c9e-4411-9924-188267b3d93c" containerName="maas-api" containerID="cri-o://031f2b28b2a5e059bc1641d6749e8ff9d3c604d74c0ba81e91d5544e91528720" gracePeriod=30
Sep 11 18:29:27.965307 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:27.965282    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"
Sep 11 18:29:28.056821 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.056777    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n2tv\" (UniqueName: \"kubernetes.io/projected/7efa298b-9c9e-4411-9924-188267b3d93c-kube-api-access-6n2tv\") pod \"7efa298b-9c9e-4411-9924-188267b3d93c\" (UID: \"7efa298b-9c9e-4411-9924-188267b3d93c\") "
Sep 11 18:29:28.057009 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.056835    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7efa298b-9c9e-4411-9924-188267b3d93c-openshift-service-ca\") pod \"7efa298b-9c9e-4411-9924-188267b3d93c\" (UID: \"7efa298b-9c9e-4411-9924-188267b3d93c\") "
Sep 11 18:29:28.057009 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.056962    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7efa298b-9c9e-4411-9924-188267b3d93c-metrics-certs\") pod \"7efa298b-9c9e-4411-9924-188267b3d93c\" (UID: \"7efa298b-9c9e-4411-9924-188267b3d93c\") "
Sep 11 18:29:28.057108 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.057055    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7efa298b-9c9e-4411-9924-188267b3d93c-maas-api-tls\") pod \"7efa298b-9c9e-4411-9924-188267b3d93c\" (UID: \"7efa298b-9c9e-4411-9924-188267b3d93c\") "
Sep 11 18:29:28.057265 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.057241    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7efa298b-9c9e-4411-9924-188267b3d93c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7efa298b-9c9e-4411-9924-188267b3d93c" (UID: "7efa298b-9c9e-4411-9924-188267b3d93c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:29:28.057418 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.057403    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7efa298b-9c9e-4411-9924-188267b3d93c-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:29:28.059218 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.059187    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7efa298b-9c9e-4411-9924-188267b3d93c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7efa298b-9c9e-4411-9924-188267b3d93c" (UID: "7efa298b-9c9e-4411-9924-188267b3d93c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:29:28.059349 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.059220    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7efa298b-9c9e-4411-9924-188267b3d93c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7efa298b-9c9e-4411-9924-188267b3d93c" (UID: "7efa298b-9c9e-4411-9924-188267b3d93c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:29:28.059349 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.059256    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7efa298b-9c9e-4411-9924-188267b3d93c-kube-api-access-6n2tv" (OuterVolumeSpecName: "kube-api-access-6n2tv") pod "7efa298b-9c9e-4411-9924-188267b3d93c" (UID: "7efa298b-9c9e-4411-9924-188267b3d93c"). InnerVolumeSpecName "kube-api-access-6n2tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:29:28.157894 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.157859    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7efa298b-9c9e-4411-9924-188267b3d93c-metrics-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:29:28.157894 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.157883    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7efa298b-9c9e-4411-9924-188267b3d93c-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:29:28.157894 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.157891    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n2tv\" (UniqueName: \"kubernetes.io/projected/7efa298b-9c9e-4411-9924-188267b3d93c-kube-api-access-6n2tv\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:29:28.610320 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.610280    2560 generic.go:356] "Generic (PLEG): container finished" podID="7efa298b-9c9e-4411-9924-188267b3d93c" containerID="031f2b28b2a5e059bc1641d6749e8ff9d3c604d74c0ba81e91d5544e91528720" exitCode=0
Sep 11 18:29:28.610542 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.610327    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"
Sep 11 18:29:28.610542 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.610360    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2" event={"ID":"7efa298b-9c9e-4411-9924-188267b3d93c","Type":"ContainerDied","Data":"031f2b28b2a5e059bc1641d6749e8ff9d3c604d74c0ba81e91d5544e91528720"}
Sep 11 18:29:28.610542 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.610428    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2" event={"ID":"7efa298b-9c9e-4411-9924-188267b3d93c","Type":"ContainerDied","Data":"213adcafb2c31ae7cd9f9af1872cb515adcd9e822d932983e827d0f15907bd84"}
Sep 11 18:29:28.610542 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.610457    2560 scope.go:117] "RemoveContainer" containerID="031f2b28b2a5e059bc1641d6749e8ff9d3c604d74c0ba81e91d5544e91528720"
Sep 11 18:29:28.620965 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.620944    2560 scope.go:117] "RemoveContainer" containerID="031f2b28b2a5e059bc1641d6749e8ff9d3c604d74c0ba81e91d5544e91528720"
Sep 11 18:29:28.621254 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:29:28.621235    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031f2b28b2a5e059bc1641d6749e8ff9d3c604d74c0ba81e91d5544e91528720\": container with ID starting with 031f2b28b2a5e059bc1641d6749e8ff9d3c604d74c0ba81e91d5544e91528720 not found: ID does not exist" containerID="031f2b28b2a5e059bc1641d6749e8ff9d3c604d74c0ba81e91d5544e91528720"
Sep 11 18:29:28.621325 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.621268    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031f2b28b2a5e059bc1641d6749e8ff9d3c604d74c0ba81e91d5544e91528720"} err="failed to get container status \"031f2b28b2a5e059bc1641d6749e8ff9d3c604d74c0ba81e91d5544e91528720\": rpc error: code = NotFound desc = could not find container \"031f2b28b2a5e059bc1641d6749e8ff9d3c604d74c0ba81e91d5544e91528720\": container with ID starting with 031f2b28b2a5e059bc1641d6749e8ff9d3c604d74c0ba81e91d5544e91528720 not found: ID does not exist"
Sep 11 18:29:28.634005 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.633978    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"]
Sep 11 18:29:28.637730 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.637709    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-74a302-56947db679-m29b2"]
Sep 11 18:29:28.687497 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:28.687467    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7efa298b-9c9e-4411-9924-188267b3d93c" path="/var/lib/kubelet/pods/7efa298b-9c9e-4411-9924-188267b3d93c/volumes"
Sep 11 18:29:30.924125 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:30.924090    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:29:46.029590 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.029557    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"]
Sep 11 18:29:46.030177 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.029791    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7" podUID="7bf20c90-7cf9-444e-b424-b525dd782a36" containerName="maas-api" containerID="cri-o://cd78d3746fcd9e95eb3f8868679895deb27009d7a9a6782fc11686d4eaf5e27e" gracePeriod=30
Sep 11 18:29:46.278918 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.278887    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"
Sep 11 18:29:46.333187 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.333096    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-snxl6\" (UniqueName: \"kubernetes.io/projected/7bf20c90-7cf9-444e-b424-b525dd782a36-kube-api-access-snxl6\") pod \"7bf20c90-7cf9-444e-b424-b525dd782a36\" (UID: \"7bf20c90-7cf9-444e-b424-b525dd782a36\") "
Sep 11 18:29:46.333338 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.333199    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf20c90-7cf9-444e-b424-b525dd782a36-metrics-certs\") pod \"7bf20c90-7cf9-444e-b424-b525dd782a36\" (UID: \"7bf20c90-7cf9-444e-b424-b525dd782a36\") "
Sep 11 18:29:46.333338 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.333253    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf20c90-7cf9-444e-b424-b525dd782a36-maas-api-tls\") pod \"7bf20c90-7cf9-444e-b424-b525dd782a36\" (UID: \"7bf20c90-7cf9-444e-b424-b525dd782a36\") "
Sep 11 18:29:46.333338 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.333303    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf20c90-7cf9-444e-b424-b525dd782a36-openshift-service-ca\") pod \"7bf20c90-7cf9-444e-b424-b525dd782a36\" (UID: \"7bf20c90-7cf9-444e-b424-b525dd782a36\") "
Sep 11 18:29:46.333778 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.333732    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf20c90-7cf9-444e-b424-b525dd782a36-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7bf20c90-7cf9-444e-b424-b525dd782a36" (UID: "7bf20c90-7cf9-444e-b424-b525dd782a36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:29:46.335464 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.335434    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bf20c90-7cf9-444e-b424-b525dd782a36-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7bf20c90-7cf9-444e-b424-b525dd782a36" (UID: "7bf20c90-7cf9-444e-b424-b525dd782a36"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:29:46.335574 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.335491    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bf20c90-7cf9-444e-b424-b525dd782a36-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7bf20c90-7cf9-444e-b424-b525dd782a36" (UID: "7bf20c90-7cf9-444e-b424-b525dd782a36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:29:46.335574 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.335530    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf20c90-7cf9-444e-b424-b525dd782a36-kube-api-access-snxl6" (OuterVolumeSpecName: "kube-api-access-snxl6") pod "7bf20c90-7cf9-444e-b424-b525dd782a36" (UID: "7bf20c90-7cf9-444e-b424-b525dd782a36"). InnerVolumeSpecName "kube-api-access-snxl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:29:46.434910 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.434877    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf20c90-7cf9-444e-b424-b525dd782a36-metrics-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:29:46.434910 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.434904    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf20c90-7cf9-444e-b424-b525dd782a36-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:29:46.434910 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.434913    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf20c90-7cf9-444e-b424-b525dd782a36-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:29:46.435147 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.434922    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snxl6\" (UniqueName: \"kubernetes.io/projected/7bf20c90-7cf9-444e-b424-b525dd782a36-kube-api-access-snxl6\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:29:46.700401 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.700362    2560 generic.go:356] "Generic (PLEG): container finished" podID="7bf20c90-7cf9-444e-b424-b525dd782a36" containerID="cd78d3746fcd9e95eb3f8868679895deb27009d7a9a6782fc11686d4eaf5e27e" exitCode=0
Sep 11 18:29:46.700547 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.700399    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"
Sep 11 18:29:46.700547 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.700442    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7" event={"ID":"7bf20c90-7cf9-444e-b424-b525dd782a36","Type":"ContainerDied","Data":"cd78d3746fcd9e95eb3f8868679895deb27009d7a9a6782fc11686d4eaf5e27e"}
Sep 11 18:29:46.700547 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.700505    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7" event={"ID":"7bf20c90-7cf9-444e-b424-b525dd782a36","Type":"ContainerDied","Data":"61177476d1a6fb0dc3a2e6828aa81ae825df3327ed6e2e6b1a011f82d47e8a23"}
Sep 11 18:29:46.700547 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.700537    2560 scope.go:117] "RemoveContainer" containerID="cd78d3746fcd9e95eb3f8868679895deb27009d7a9a6782fc11686d4eaf5e27e"
Sep 11 18:29:46.710576 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.710540    2560 scope.go:117] "RemoveContainer" containerID="cd78d3746fcd9e95eb3f8868679895deb27009d7a9a6782fc11686d4eaf5e27e"
Sep 11 18:29:46.710915 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:29:46.710894    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd78d3746fcd9e95eb3f8868679895deb27009d7a9a6782fc11686d4eaf5e27e\": container with ID starting with cd78d3746fcd9e95eb3f8868679895deb27009d7a9a6782fc11686d4eaf5e27e not found: ID does not exist" containerID="cd78d3746fcd9e95eb3f8868679895deb27009d7a9a6782fc11686d4eaf5e27e"
Sep 11 18:29:46.710987 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.710923    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd78d3746fcd9e95eb3f8868679895deb27009d7a9a6782fc11686d4eaf5e27e"} err="failed to get container status \"cd78d3746fcd9e95eb3f8868679895deb27009d7a9a6782fc11686d4eaf5e27e\": rpc error: code = NotFound desc = could not find container \"cd78d3746fcd9e95eb3f8868679895deb27009d7a9a6782fc11686d4eaf5e27e\": container with ID starting with cd78d3746fcd9e95eb3f8868679895deb27009d7a9a6782fc11686d4eaf5e27e not found: ID does not exist"
Sep 11 18:29:46.718597 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.718571    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"]
Sep 11 18:29:46.723975 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:46.723952    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-a5bbf6-5c89c899f4-z2ps7"]
Sep 11 18:29:47.924034 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:47.923995    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:29:48.688654 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:29:48.688607    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bf20c90-7cf9-444e-b424-b525dd782a36" path="/var/lib/kubelet/pods/7bf20c90-7cf9-444e-b424-b525dd782a36/volumes"
Sep 11 18:30:04.224121 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.224086    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"]
Sep 11 18:30:04.224664 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.224379    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg" podUID="5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a" containerName="maas-api" containerID="cri-o://39dbb0679e188bf46709bf94b6e4dccfd77ad96241637ab2ab26f036003e1dea" gracePeriod=30
Sep 11 18:30:04.471499 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.471471    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"
Sep 11 18:30:04.618419 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.618330    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-openshift-service-ca\") pod \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\" (UID: \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\") "
Sep 11 18:30:04.618419 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.618386    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-metrics-certs\") pod \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\" (UID: \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\") "
Sep 11 18:30:04.618635 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.618452    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-maas-api-tls\") pod \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\" (UID: \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\") "
Sep 11 18:30:04.618635 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.618519    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2glfj\" (UniqueName: \"kubernetes.io/projected/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-kube-api-access-2glfj\") pod \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\" (UID: \"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a\") "
Sep 11 18:30:04.618807 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.618786    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a" (UID: "5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:30:04.618878 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.618860    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:30:04.620840 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.620813    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a" (UID: "5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:30:04.620936 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.620843    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a" (UID: "5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:30:04.620971 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.620938    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-kube-api-access-2glfj" (OuterVolumeSpecName: "kube-api-access-2glfj") pod "5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a" (UID: "5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a"). InnerVolumeSpecName "kube-api-access-2glfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:30:04.720293 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.720264    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-metrics-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:30:04.720293 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.720289    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:30:04.720293 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.720299    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2glfj\" (UniqueName: \"kubernetes.io/projected/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a-kube-api-access-2glfj\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:30:04.795665 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.795634    2560 generic.go:356] "Generic (PLEG): container finished" podID="5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a" containerID="39dbb0679e188bf46709bf94b6e4dccfd77ad96241637ab2ab26f036003e1dea" exitCode=0
Sep 11 18:30:04.795665 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.795672    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg" event={"ID":"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a","Type":"ContainerDied","Data":"39dbb0679e188bf46709bf94b6e4dccfd77ad96241637ab2ab26f036003e1dea"}
Sep 11 18:30:04.795916 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.795680    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"
Sep 11 18:30:04.795916 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.795709    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg" event={"ID":"5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a","Type":"ContainerDied","Data":"545205c7f00ef911907b329432f432383f1ac8097e34227757b74fb94dc9d960"}
Sep 11 18:30:04.795916 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.795729    2560 scope.go:117] "RemoveContainer" containerID="39dbb0679e188bf46709bf94b6e4dccfd77ad96241637ab2ab26f036003e1dea"
Sep 11 18:30:04.804913 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.804861    2560 scope.go:117] "RemoveContainer" containerID="39dbb0679e188bf46709bf94b6e4dccfd77ad96241637ab2ab26f036003e1dea"
Sep 11 18:30:04.805131 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:30:04.805112    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39dbb0679e188bf46709bf94b6e4dccfd77ad96241637ab2ab26f036003e1dea\": container with ID starting with 39dbb0679e188bf46709bf94b6e4dccfd77ad96241637ab2ab26f036003e1dea not found: ID does not exist" containerID="39dbb0679e188bf46709bf94b6e4dccfd77ad96241637ab2ab26f036003e1dea"
Sep 11 18:30:04.805195 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.805139    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39dbb0679e188bf46709bf94b6e4dccfd77ad96241637ab2ab26f036003e1dea"} err="failed to get container status \"39dbb0679e188bf46709bf94b6e4dccfd77ad96241637ab2ab26f036003e1dea\": rpc error: code = NotFound desc = could not find container \"39dbb0679e188bf46709bf94b6e4dccfd77ad96241637ab2ab26f036003e1dea\": container with ID starting with 39dbb0679e188bf46709bf94b6e4dccfd77ad96241637ab2ab26f036003e1dea not found: ID does not exist"
Sep 11 18:30:04.814927 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.814903    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"]
Sep 11 18:30:04.820101 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:04.820078    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-5d35e7-79cf6c5556-v4rbg"]
Sep 11 18:30:06.689475 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:06.689436    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a" path="/var/lib/kubelet/pods/5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a/volumes"
Sep 11 18:30:06.810101 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:06.810067    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:30:15.612537 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:30:15.612502    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:31:38.322629 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:31:38.322545    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:32:02.633689 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:32:02.633650    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:32:10.518017 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:32:10.517977    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:32:18.632822 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:32:18.632790    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:32:36.515343 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:32:36.515309    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:32:42.607179 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:32:42.607145    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:34:02.822816 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:34:02.822783    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/2.log"
Sep 11 18:34:02.825156 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:34:02.825125    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/2.log"
Sep 11 18:34:07.422886 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:34:07.422848    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:34:16.325694 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:34:16.325656    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:34:31.825620 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:34:31.825537    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:34:43.915080 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:34:43.915044    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:34:52.412976 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:34:52.412940    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:35:00.026323 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:00.026287    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:35:06.621355 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:06.621317    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:35:07.123324 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.123291    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"]
Sep 11 18:35:07.123675 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.123663    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a" containerName="maas-api"
Sep 11 18:35:07.123721 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.123677    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a" containerName="maas-api"
Sep 11 18:35:07.123721 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.123689    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fb61a39-a94b-4a47-9967-3a78f325897c" containerName="maas-api"
Sep 11 18:35:07.123721 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.123695    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fb61a39-a94b-4a47-9967-3a78f325897c" containerName="maas-api"
Sep 11 18:35:07.123721 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.123703    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bf20c90-7cf9-444e-b424-b525dd782a36" containerName="maas-api"
Sep 11 18:35:07.123721 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.123708    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bf20c90-7cf9-444e-b424-b525dd782a36" containerName="maas-api"
Sep 11 18:35:07.123721 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.123714    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ced1884-b4ca-45f9-9000-a25780703893" containerName="main"
Sep 11 18:35:07.123721 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.123720    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ced1884-b4ca-45f9-9000-a25780703893" containerName="main"
Sep 11 18:35:07.123978 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.123728    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7efa298b-9c9e-4411-9924-188267b3d93c" containerName="maas-api"
Sep 11 18:35:07.123978 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.123733    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7efa298b-9c9e-4411-9924-188267b3d93c" containerName="maas-api"
Sep 11 18:35:07.123978 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.123818    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ced1884-b4ca-45f9-9000-a25780703893" containerName="main"
Sep 11 18:35:07.123978 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.123827    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7efa298b-9c9e-4411-9924-188267b3d93c" containerName="maas-api"
Sep 11 18:35:07.123978 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.123833    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b0b4380-1a7d-4e3d-ab98-80e5dd96a00a" containerName="maas-api"
Sep 11 18:35:07.123978 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.123840    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fb61a39-a94b-4a47-9967-3a78f325897c" containerName="maas-api"
Sep 11 18:35:07.123978 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.123846    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bf20c90-7cf9-444e-b424-b525dd782a36" containerName="maas-api"
Sep 11 18:35:07.126790 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.126774    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:07.129511 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:35:07.129478    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-66ff6b6fc9-69mj9\" is forbidden: User \"system:node:ip-10-0-128-29.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-29.ec2.internal' and this object" podUID="c65ab244-5d42-45a8-97f8-ae90c206bac3" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:07.129660 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:35:07.129500    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-api-dockercfg-2sf5b\" is forbidden: User \"system:node:ip-10-0-128-29.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-29.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2sf5b\"" type="*v1.Secret"
Sep 11 18:35:07.130264 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:35:07.130235    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-29.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-29.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 11 18:35:07.130264 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:35:07.130246    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-api-metrics-tls\" is forbidden: User \"system:node:ip-10-0-128-29.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-29.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" type="*v1.Secret"
Sep 11 18:35:07.130420 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:35:07.130293    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-api-serving-cert\" is forbidden: User \"system:node:ip-10-0-128-29.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-29.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" type="*v1.Secret"
Sep 11 18:35:07.132356 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:35:07.132328    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-29.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-128-29.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 11 18:35:07.144552 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.144522    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"]
Sep 11 18:35:07.186314 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.186279    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c65ab244-5d42-45a8-97f8-ae90c206bac3-metrics-certs\") pod \"maas-api-66ff6b6fc9-69mj9\" (UID: \"c65ab244-5d42-45a8-97f8-ae90c206bac3\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:07.186314 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.186319    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65ab244-5d42-45a8-97f8-ae90c206bac3-openshift-service-ca\") pod \"maas-api-66ff6b6fc9-69mj9\" (UID: \"c65ab244-5d42-45a8-97f8-ae90c206bac3\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:07.186562 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.186375    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c65ab244-5d42-45a8-97f8-ae90c206bac3-maas-api-tls\") pod \"maas-api-66ff6b6fc9-69mj9\" (UID: \"c65ab244-5d42-45a8-97f8-ae90c206bac3\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:07.186562 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.186436    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9jn4\" (UniqueName: \"kubernetes.io/projected/c65ab244-5d42-45a8-97f8-ae90c206bac3-kube-api-access-w9jn4\") pod \"maas-api-66ff6b6fc9-69mj9\" (UID: \"c65ab244-5d42-45a8-97f8-ae90c206bac3\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:07.287148 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.287109    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c65ab244-5d42-45a8-97f8-ae90c206bac3-metrics-certs\") pod \"maas-api-66ff6b6fc9-69mj9\" (UID: \"c65ab244-5d42-45a8-97f8-ae90c206bac3\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:07.287148 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.287157    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65ab244-5d42-45a8-97f8-ae90c206bac3-openshift-service-ca\") pod \"maas-api-66ff6b6fc9-69mj9\" (UID: \"c65ab244-5d42-45a8-97f8-ae90c206bac3\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:07.287372 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.287201    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c65ab244-5d42-45a8-97f8-ae90c206bac3-maas-api-tls\") pod \"maas-api-66ff6b6fc9-69mj9\" (UID: \"c65ab244-5d42-45a8-97f8-ae90c206bac3\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:07.287372 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.287247    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9jn4\" (UniqueName: \"kubernetes.io/projected/c65ab244-5d42-45a8-97f8-ae90c206bac3-kube-api-access-w9jn4\") pod \"maas-api-66ff6b6fc9-69mj9\" (UID: \"c65ab244-5d42-45a8-97f8-ae90c206bac3\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:07.954043 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:07.954013    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2sf5b\""
Sep 11 18:35:08.288031 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:35:08.287992    2560 configmap.go:193] Couldn't get configMap odh-ai-gateway-infra/openshift-service-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 11 18:35:08.288031 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:35:08.288008    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: failed to sync secret cache: timed out waiting for the condition
Sep 11 18:35:08.288282 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:35:08.288080    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c65ab244-5d42-45a8-97f8-ae90c206bac3-openshift-service-ca podName:c65ab244-5d42-45a8-97f8-ae90c206bac3 nodeName:}" failed. No retries permitted until 2026-09-11 18:35:08.78806275 +0000 UTC m=+2466.631298170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-service-ca" (UniqueName: "kubernetes.io/configmap/c65ab244-5d42-45a8-97f8-ae90c206bac3-openshift-service-ca") pod "maas-api-66ff6b6fc9-69mj9" (UID: "c65ab244-5d42-45a8-97f8-ae90c206bac3") : failed to sync configmap cache: timed out waiting for the condition
Sep 11 18:35:08.288282 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:35:08.287995    2560 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: failed to sync secret cache: timed out waiting for the condition
Sep 11 18:35:08.288282 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:35:08.288095    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65ab244-5d42-45a8-97f8-ae90c206bac3-metrics-certs podName:c65ab244-5d42-45a8-97f8-ae90c206bac3 nodeName:}" failed. No retries permitted until 2026-09-11 18:35:08.788087851 +0000 UTC m=+2466.631323270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c65ab244-5d42-45a8-97f8-ae90c206bac3-metrics-certs") pod "maas-api-66ff6b6fc9-69mj9" (UID: "c65ab244-5d42-45a8-97f8-ae90c206bac3") : failed to sync secret cache: timed out waiting for the condition
Sep 11 18:35:08.288282 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:35:08.288123    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65ab244-5d42-45a8-97f8-ae90c206bac3-maas-api-tls podName:c65ab244-5d42-45a8-97f8-ae90c206bac3 nodeName:}" failed. No retries permitted until 2026-09-11 18:35:08.788104446 +0000 UTC m=+2466.631339865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c65ab244-5d42-45a8-97f8-ae90c206bac3-maas-api-tls") pod "maas-api-66ff6b6fc9-69mj9" (UID: "c65ab244-5d42-45a8-97f8-ae90c206bac3") : failed to sync secret cache: timed out waiting for the condition
Sep 11 18:35:08.297154 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:35:08.297132    2560 projected.go:291] Couldn't get configMap odh-ai-gateway-infra/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 11 18:35:08.511963 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:08.511927    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 18:35:08.554380 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:08.554298    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 18:35:08.703845 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:08.703814    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 18:35:08.732812 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:08.732778    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:35:08.738265 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:35:08.738247    2560 projected.go:196] Error preparing data for projected volume kube-api-access-w9jn4 for pod odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9: failed to sync configmap cache: timed out waiting for the condition
Sep 11 18:35:08.738353 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:35:08.738322    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c65ab244-5d42-45a8-97f8-ae90c206bac3-kube-api-access-w9jn4 podName:c65ab244-5d42-45a8-97f8-ae90c206bac3 nodeName:}" failed. No retries permitted until 2026-09-11 18:35:09.238305314 +0000 UTC m=+2467.081540732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w9jn4" (UniqueName: "kubernetes.io/projected/c65ab244-5d42-45a8-97f8-ae90c206bac3-kube-api-access-w9jn4") pod "maas-api-66ff6b6fc9-69mj9" (UID: "c65ab244-5d42-45a8-97f8-ae90c206bac3") : failed to sync configmap cache: timed out waiting for the condition
Sep 11 18:35:08.799816 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:08.799776    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c65ab244-5d42-45a8-97f8-ae90c206bac3-metrics-certs\") pod \"maas-api-66ff6b6fc9-69mj9\" (UID: \"c65ab244-5d42-45a8-97f8-ae90c206bac3\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:08.799973 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:08.799828    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65ab244-5d42-45a8-97f8-ae90c206bac3-openshift-service-ca\") pod \"maas-api-66ff6b6fc9-69mj9\" (UID: \"c65ab244-5d42-45a8-97f8-ae90c206bac3\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:08.799973 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:08.799879    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c65ab244-5d42-45a8-97f8-ae90c206bac3-maas-api-tls\") pod \"maas-api-66ff6b6fc9-69mj9\" (UID: \"c65ab244-5d42-45a8-97f8-ae90c206bac3\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:08.800571 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:08.800475    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65ab244-5d42-45a8-97f8-ae90c206bac3-openshift-service-ca\") pod \"maas-api-66ff6b6fc9-69mj9\" (UID: \"c65ab244-5d42-45a8-97f8-ae90c206bac3\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:08.802527 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:08.802503    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c65ab244-5d42-45a8-97f8-ae90c206bac3-maas-api-tls\") pod \"maas-api-66ff6b6fc9-69mj9\" (UID: \"c65ab244-5d42-45a8-97f8-ae90c206bac3\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:08.802654 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:08.802550    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c65ab244-5d42-45a8-97f8-ae90c206bac3-metrics-certs\") pod \"maas-api-66ff6b6fc9-69mj9\" (UID: \"c65ab244-5d42-45a8-97f8-ae90c206bac3\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:09.304162 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:09.304124    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9jn4\" (UniqueName: \"kubernetes.io/projected/c65ab244-5d42-45a8-97f8-ae90c206bac3-kube-api-access-w9jn4\") pod \"maas-api-66ff6b6fc9-69mj9\" (UID: \"c65ab244-5d42-45a8-97f8-ae90c206bac3\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:09.306727 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:09.306704    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9jn4\" (UniqueName: \"kubernetes.io/projected/c65ab244-5d42-45a8-97f8-ae90c206bac3-kube-api-access-w9jn4\") pod \"maas-api-66ff6b6fc9-69mj9\" (UID: \"c65ab244-5d42-45a8-97f8-ae90c206bac3\") " pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:09.537340 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:09.537303    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:09.826031 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:09.826000    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"]
Sep 11 18:35:09.828524 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:35:09.828493    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65ab244_5d42_45a8_97f8_ae90c206bac3.slice/crio-b47fda3cd30b020bafc83e27136640a359581a886286c006aa7e029039ec828f WatchSource:0}: Error finding container b47fda3cd30b020bafc83e27136640a359581a886286c006aa7e029039ec828f: Status 404 returned error can't find the container with id b47fda3cd30b020bafc83e27136640a359581a886286c006aa7e029039ec828f
Sep 11 18:35:09.830251 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:09.830233    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:35:10.320803 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:10.320772    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9" event={"ID":"c65ab244-5d42-45a8-97f8-ae90c206bac3","Type":"ContainerStarted","Data":"44d547429fc764dfab94357543e637e94e5f5499d832d4c97e05968a070c86da"}
Sep 11 18:35:10.321184 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:10.320820    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9" event={"ID":"c65ab244-5d42-45a8-97f8-ae90c206bac3","Type":"ContainerStarted","Data":"b47fda3cd30b020bafc83e27136640a359581a886286c006aa7e029039ec828f"}
Sep 11 18:35:10.321184 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:10.320900    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:10.341882 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:10.341827    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9" podStartSLOduration=2.949066828 podStartE2EDuration="3.341811711s" totalImagesPullingTime="392.744883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:35:10.340387935 +0000 UTC m=+2468.183623377" watchObservedRunningTime="2026-09-11 18:35:10.341811711 +0000 UTC m=+2468.185047152"
Sep 11 18:35:14.517480 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:14.517439    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:35:14.771239 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:14.771156    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"]
Sep 11 18:35:14.771425 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:14.771401    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9" podUID="c65ab244-5d42-45a8-97f8-ae90c206bac3" containerName="maas-api" containerID="cri-o://44d547429fc764dfab94357543e637e94e5f5499d832d4c97e05968a070c86da" gracePeriod=30
Sep 11 18:35:14.810785 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:14.810741    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj"]
Sep 11 18:35:14.814105 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:14.814089    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj"
Sep 11 18:35:14.825888 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:14.825862    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj"]
Sep 11 18:35:14.853374 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:14.853346    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49d66ef1-4929-4bf0-aee2-764ede38df5e-metrics-certs\") pod \"maas-api-bc5d8db47-d4rlj\" (UID: \"49d66ef1-4929-4bf0-aee2-764ede38df5e\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj"
Sep 11 18:35:14.853504 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:14.853391    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csmvs\" (UniqueName: \"kubernetes.io/projected/49d66ef1-4929-4bf0-aee2-764ede38df5e-kube-api-access-csmvs\") pod \"maas-api-bc5d8db47-d4rlj\" (UID: \"49d66ef1-4929-4bf0-aee2-764ede38df5e\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj"
Sep 11 18:35:14.853504 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:14.853468    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49d66ef1-4929-4bf0-aee2-764ede38df5e-maas-api-tls\") pod \"maas-api-bc5d8db47-d4rlj\" (UID: \"49d66ef1-4929-4bf0-aee2-764ede38df5e\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj"
Sep 11 18:35:14.853619 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:14.853532    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d66ef1-4929-4bf0-aee2-764ede38df5e-openshift-service-ca\") pod \"maas-api-bc5d8db47-d4rlj\" (UID: \"49d66ef1-4929-4bf0-aee2-764ede38df5e\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj"
Sep 11 18:35:14.954666 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:14.954621    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49d66ef1-4929-4bf0-aee2-764ede38df5e-metrics-certs\") pod \"maas-api-bc5d8db47-d4rlj\" (UID: \"49d66ef1-4929-4bf0-aee2-764ede38df5e\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj"
Sep 11 18:35:14.954812 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:14.954707    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csmvs\" (UniqueName: \"kubernetes.io/projected/49d66ef1-4929-4bf0-aee2-764ede38df5e-kube-api-access-csmvs\") pod \"maas-api-bc5d8db47-d4rlj\" (UID: \"49d66ef1-4929-4bf0-aee2-764ede38df5e\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj"
Sep 11 18:35:14.954812 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:14.954799    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49d66ef1-4929-4bf0-aee2-764ede38df5e-maas-api-tls\") pod \"maas-api-bc5d8db47-d4rlj\" (UID: \"49d66ef1-4929-4bf0-aee2-764ede38df5e\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj"
Sep 11 18:35:14.954907 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:14.954855    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d66ef1-4929-4bf0-aee2-764ede38df5e-openshift-service-ca\") pod \"maas-api-bc5d8db47-d4rlj\" (UID: \"49d66ef1-4929-4bf0-aee2-764ede38df5e\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj"
Sep 11 18:35:14.955618 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:14.955544    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49d66ef1-4929-4bf0-aee2-764ede38df5e-openshift-service-ca\") pod \"maas-api-bc5d8db47-d4rlj\" (UID: \"49d66ef1-4929-4bf0-aee2-764ede38df5e\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj"
Sep 11 18:35:14.957276 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:14.957247    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49d66ef1-4929-4bf0-aee2-764ede38df5e-metrics-certs\") pod \"maas-api-bc5d8db47-d4rlj\" (UID: \"49d66ef1-4929-4bf0-aee2-764ede38df5e\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj"
Sep 11 18:35:14.957359 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:14.957309    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49d66ef1-4929-4bf0-aee2-764ede38df5e-maas-api-tls\") pod \"maas-api-bc5d8db47-d4rlj\" (UID: \"49d66ef1-4929-4bf0-aee2-764ede38df5e\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj"
Sep 11 18:35:14.964052 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:14.964027    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csmvs\" (UniqueName: \"kubernetes.io/projected/49d66ef1-4929-4bf0-aee2-764ede38df5e-kube-api-access-csmvs\") pod \"maas-api-bc5d8db47-d4rlj\" (UID: \"49d66ef1-4929-4bf0-aee2-764ede38df5e\") " pod="odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj"
Sep 11 18:35:15.020998 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.020967    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:15.055556 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.055487    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9jn4\" (UniqueName: \"kubernetes.io/projected/c65ab244-5d42-45a8-97f8-ae90c206bac3-kube-api-access-w9jn4\") pod \"c65ab244-5d42-45a8-97f8-ae90c206bac3\" (UID: \"c65ab244-5d42-45a8-97f8-ae90c206bac3\") "
Sep 11 18:35:15.055556 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.055544    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c65ab244-5d42-45a8-97f8-ae90c206bac3-maas-api-tls\") pod \"c65ab244-5d42-45a8-97f8-ae90c206bac3\" (UID: \"c65ab244-5d42-45a8-97f8-ae90c206bac3\") "
Sep 11 18:35:15.055798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.055601    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65ab244-5d42-45a8-97f8-ae90c206bac3-openshift-service-ca\") pod \"c65ab244-5d42-45a8-97f8-ae90c206bac3\" (UID: \"c65ab244-5d42-45a8-97f8-ae90c206bac3\") "
Sep 11 18:35:15.055798 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.055696    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c65ab244-5d42-45a8-97f8-ae90c206bac3-metrics-certs\") pod \"c65ab244-5d42-45a8-97f8-ae90c206bac3\" (UID: \"c65ab244-5d42-45a8-97f8-ae90c206bac3\") "
Sep 11 18:35:15.056024 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.055997    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c65ab244-5d42-45a8-97f8-ae90c206bac3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c65ab244-5d42-45a8-97f8-ae90c206bac3" (UID: "c65ab244-5d42-45a8-97f8-ae90c206bac3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:35:15.057924 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.057896    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c65ab244-5d42-45a8-97f8-ae90c206bac3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c65ab244-5d42-45a8-97f8-ae90c206bac3" (UID: "c65ab244-5d42-45a8-97f8-ae90c206bac3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:35:15.058027 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.057936    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c65ab244-5d42-45a8-97f8-ae90c206bac3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c65ab244-5d42-45a8-97f8-ae90c206bac3" (UID: "c65ab244-5d42-45a8-97f8-ae90c206bac3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:35:15.058027 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.057987    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c65ab244-5d42-45a8-97f8-ae90c206bac3-kube-api-access-w9jn4" (OuterVolumeSpecName: "kube-api-access-w9jn4") pod "c65ab244-5d42-45a8-97f8-ae90c206bac3" (UID: "c65ab244-5d42-45a8-97f8-ae90c206bac3"). InnerVolumeSpecName "kube-api-access-w9jn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:15.143930 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.143898    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj"
Sep 11 18:35:15.157328 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.157295    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9jn4\" (UniqueName: \"kubernetes.io/projected/c65ab244-5d42-45a8-97f8-ae90c206bac3-kube-api-access-w9jn4\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:35:15.157328 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.157321    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c65ab244-5d42-45a8-97f8-ae90c206bac3-maas-api-tls\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:35:15.157328 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.157332    2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65ab244-5d42-45a8-97f8-ae90c206bac3-openshift-service-ca\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:35:15.157559 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.157340    2560 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c65ab244-5d42-45a8-97f8-ae90c206bac3-metrics-certs\") on node \"ip-10-0-128-29.ec2.internal\" DevicePath \"\""
Sep 11 18:35:15.346868 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.346785    2560 generic.go:356] "Generic (PLEG): container finished" podID="c65ab244-5d42-45a8-97f8-ae90c206bac3" containerID="44d547429fc764dfab94357543e637e94e5f5499d832d4c97e05968a070c86da" exitCode=0
Sep 11 18:35:15.346868 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.346825    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9" event={"ID":"c65ab244-5d42-45a8-97f8-ae90c206bac3","Type":"ContainerDied","Data":"44d547429fc764dfab94357543e637e94e5f5499d832d4c97e05968a070c86da"}
Sep 11 18:35:15.346868 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.346868    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9" event={"ID":"c65ab244-5d42-45a8-97f8-ae90c206bac3","Type":"ContainerDied","Data":"b47fda3cd30b020bafc83e27136640a359581a886286c006aa7e029039ec828f"}
Sep 11 18:35:15.347075 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.346890    2560 scope.go:117] "RemoveContainer" containerID="44d547429fc764dfab94357543e637e94e5f5499d832d4c97e05968a070c86da"
Sep 11 18:35:15.347075 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.346889    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"
Sep 11 18:35:15.356965 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.356944    2560 scope.go:117] "RemoveContainer" containerID="44d547429fc764dfab94357543e637e94e5f5499d832d4c97e05968a070c86da"
Sep 11 18:35:15.357215 ip-10-0-128-29 kubenswrapper[2560]: E0911 18:35:15.357188    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44d547429fc764dfab94357543e637e94e5f5499d832d4c97e05968a070c86da\": container with ID starting with 44d547429fc764dfab94357543e637e94e5f5499d832d4c97e05968a070c86da not found: ID does not exist" containerID="44d547429fc764dfab94357543e637e94e5f5499d832d4c97e05968a070c86da"
Sep 11 18:35:15.357327 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.357224    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44d547429fc764dfab94357543e637e94e5f5499d832d4c97e05968a070c86da"} err="failed to get container status \"44d547429fc764dfab94357543e637e94e5f5499d832d4c97e05968a070c86da\": rpc error: code = NotFound desc = could not find container \"44d547429fc764dfab94357543e637e94e5f5499d832d4c97e05968a070c86da\": container with ID starting with 44d547429fc764dfab94357543e637e94e5f5499d832d4c97e05968a070c86da not found: ID does not exist"
Sep 11 18:35:15.370725 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.370701    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"]
Sep 11 18:35:15.375819 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.375778    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-66ff6b6fc9-69mj9"]
Sep 11 18:35:15.419065 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:15.419001    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj"]
Sep 11 18:35:15.421492 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:35:15.421463    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d66ef1_4929_4bf0_aee2_764ede38df5e.slice/crio-2dd28a71bb2b40466e59311f8eff45d850dc1e66e22e24cbc66e01f327320dcb WatchSource:0}: Error finding container 2dd28a71bb2b40466e59311f8eff45d850dc1e66e22e24cbc66e01f327320dcb: Status 404 returned error can't find the container with id 2dd28a71bb2b40466e59311f8eff45d850dc1e66e22e24cbc66e01f327320dcb
Sep 11 18:35:16.354773 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:16.354704    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj" event={"ID":"49d66ef1-4929-4bf0-aee2-764ede38df5e","Type":"ContainerStarted","Data":"87784b098804dc7bc551ea226a006524d5ffc08505deec450dc37e2d6ab87d69"}
Sep 11 18:35:16.354773 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:16.354778    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj" event={"ID":"49d66ef1-4929-4bf0-aee2-764ede38df5e","Type":"ContainerStarted","Data":"2dd28a71bb2b40466e59311f8eff45d850dc1e66e22e24cbc66e01f327320dcb"}
Sep 11 18:35:16.355180 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:16.354850    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj"
Sep 11 18:35:16.379580 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:16.379526    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj" podStartSLOduration=2.095956713 podStartE2EDuration="2.379509597s" totalImagesPullingTime="283.552884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:35:16.379277716 +0000 UTC m=+2474.222513158" watchObservedRunningTime="2026-09-11 18:35:16.379509597 +0000 UTC m=+2474.222745037"
Sep 11 18:35:16.687923 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:16.687845    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c65ab244-5d42-45a8-97f8-ae90c206bac3" path="/var/lib/kubelet/pods/c65ab244-5d42-45a8-97f8-ae90c206bac3/volumes"
Sep 11 18:35:21.420949 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:21.420914    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:35:22.365257 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:22.365230    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bc5d8db47-d4rlj"
Sep 11 18:35:27.413076 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:27.413034    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:35:34.212452 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:34.212410    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-t5cm4"]
Sep 11 18:35:42.799227 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:42.799197    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7_e01fbe9e-fc87-4548-8223-64276fabad17/copy-manifests/0.log"
Sep 11 18:35:42.806461 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:42.806438    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7_e01fbe9e-fc87-4548-8223-64276fabad17/manager/0.log"
Sep 11 18:35:43.495603 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:43.495573    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-bc5d8db47-d4rlj_49d66ef1-4929-4bf0-aee2-764ede38df5e/maas-api/0.log"
Sep 11 18:35:44.967394 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:44.967366    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-68g6n_bc5d3dba-0fae-4dcf-a68f-0a0071f07ada/kuadrant-console-plugin/0.log"
Sep 11 18:35:45.199255 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:45.199225    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-t5cm4_69c6a8d9-6459-4b7e-b0b6-f2edbb2f6ec7/limitador/0.log"
Sep 11 18:35:45.883012 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:45.882987    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ff57cc6b9-bnldt_cd43172e-8f56-47b0-a780-fa9232ff2d41/kube-auth-proxy/0.log"
Sep 11 18:35:47.095581 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:47.095552    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-hr2w4_a6fcc7c5-7038-40a0-a002-6240f8be2549/main/0.log"
Sep 11 18:35:47.230496 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:47.230466    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-tsdqh_5437bb20-46f3-4188-b578-f1a027c36b91/main/0.log"
Sep 11 18:35:47.475206 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:47.475159    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-vbk42_f5a90df0-a07e-4eb4-b174-dc330824af1a/main/0.log"
Sep 11 18:35:47.592866 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:47.592836    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-b46qk_f48c605d-4dab-420f-9bf9-1301f822afe2/main/0.log"
Sep 11 18:35:55.914964 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:55.914883    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-r7cvn_c5702497-d40f-4a13-b5b8-0be72c784011/global-pull-secret-syncer/0.log"
Sep 11 18:35:56.129614 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:56.129585    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v4zkn_dbe2052d-5694-4782-815b-0975eced80b8/konnectivity-agent/0.log"
Sep 11 18:35:56.154399 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:35:56.154372    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-29.ec2.internal_fdf7e40390d26689763c696c18fb8f21/haproxy/0.log"
Sep 11 18:36:00.777015 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:00.776985    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-68g6n_bc5d3dba-0fae-4dcf-a68f-0a0071f07ada/kuadrant-console-plugin/0.log"
Sep 11 18:36:00.856266 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:00.856238    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-t5cm4_69c6a8d9-6459-4b7e-b0b6-f2edbb2f6ec7/limitador/0.log"
Sep 11 18:36:02.329230 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:02.329196    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d878d705-9b00-40f5-85f3-16a887c4192d/alertmanager/0.log"
Sep 11 18:36:02.358670 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:02.358648    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d878d705-9b00-40f5-85f3-16a887c4192d/config-reloader/0.log"
Sep 11 18:36:02.383829 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:02.383799    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d878d705-9b00-40f5-85f3-16a887c4192d/kube-rbac-proxy-web/0.log"
Sep 11 18:36:02.410656 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:02.410631    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d878d705-9b00-40f5-85f3-16a887c4192d/kube-rbac-proxy/0.log"
Sep 11 18:36:02.439151 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:02.439120    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d878d705-9b00-40f5-85f3-16a887c4192d/kube-rbac-proxy-metric/0.log"
Sep 11 18:36:02.471248 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:02.471217    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d878d705-9b00-40f5-85f3-16a887c4192d/prom-label-proxy/0.log"
Sep 11 18:36:02.501098 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:02.501077    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d878d705-9b00-40f5-85f3-16a887c4192d/init-config-reloader/0.log"
Sep 11 18:36:02.566958 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:02.566929    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-g6dh4_e99b97af-cf7c-4b4e-8fac-374449322c1e/cluster-monitoring-operator/0.log"
Sep 11 18:36:02.706643 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:02.706603    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6d8c99dcb-b5j7r_2183bc68-cb24-4a71-99f8-40cd8177e13e/metrics-server/0.log"
Sep 11 18:36:02.745543 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:02.745515    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-kngjs_4fbe05f0-4cdc-4d26-ada7-b23de0eb0dcf/monitoring-plugin/0.log"
Sep 11 18:36:02.922490 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:02.922466    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p65z7_f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6/node-exporter/0.log"
Sep 11 18:36:02.993282 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:02.993246    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p65z7_f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6/kube-rbac-proxy/0.log"
Sep 11 18:36:03.056170 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:03.056145    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p65z7_f70f4cdd-df7e-4704-8c9b-280fc6f8d5c6/init-textfile/0.log"
Sep 11 18:36:03.210004 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:03.209974    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-kdnvw_6f650bd3-55b5-4762-9ff2-772e71d1e5e7/kube-rbac-proxy-main/0.log"
Sep 11 18:36:03.235459 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:03.235429    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-kdnvw_6f650bd3-55b5-4762-9ff2-772e71d1e5e7/kube-rbac-proxy-self/0.log"
Sep 11 18:36:03.261599 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:03.261530    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-kdnvw_6f650bd3-55b5-4762-9ff2-772e71d1e5e7/openshift-state-metrics/0.log"
Sep 11 18:36:03.573602 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:03.573527    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-s7225_00cc7737-711d-4e6f-87ce-da84f6bc1684/prometheus-operator/0.log"
Sep 11 18:36:03.600665 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:03.600620    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-s7225_00cc7737-711d-4e6f-87ce-da84f6bc1684/kube-rbac-proxy/0.log"
Sep 11 18:36:03.739206 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:03.739174    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7dbf9f45f6-g9prl_262186c8-449d-4f55-8194-fe7b2e619d63/telemeter-client/0.log"
Sep 11 18:36:03.764406 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:03.764375    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7dbf9f45f6-g9prl_262186c8-449d-4f55-8194-fe7b2e619d63/reload/0.log"
Sep 11 18:36:03.793311 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:03.793283    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7dbf9f45f6-g9prl_262186c8-449d-4f55-8194-fe7b2e619d63/kube-rbac-proxy/0.log"
Sep 11 18:36:05.366432 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.366402    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-74tpj_a95d5df4-396b-404d-af3f-3787a83dea92/networking-console-plugin/0.log"
Sep 11 18:36:05.699268 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.699191    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"]
Sep 11 18:36:05.699592 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.699579    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c65ab244-5d42-45a8-97f8-ae90c206bac3" containerName="maas-api"
Sep 11 18:36:05.699634 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.699593    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c65ab244-5d42-45a8-97f8-ae90c206bac3" containerName="maas-api"
Sep 11 18:36:05.699673 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.699660    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c65ab244-5d42-45a8-97f8-ae90c206bac3" containerName="maas-api"
Sep 11 18:36:05.703934 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.703918    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"
Sep 11 18:36:05.707959 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.707933    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mt88g\"/\"kube-root-ca.crt\""
Sep 11 18:36:05.709226 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.709180    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mt88g\"/\"default-dockercfg-2kcn4\""
Sep 11 18:36:05.709358 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.709180    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mt88g\"/\"openshift-service-ca.crt\""
Sep 11 18:36:05.722181 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.722154    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"]
Sep 11 18:36:05.741965 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.741931    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95246b0b-1a60-4e86-b309-34cbf8258bdc-lib-modules\") pod \"perf-node-gather-daemonset-bs8lr\" (UID: \"95246b0b-1a60-4e86-b309-34cbf8258bdc\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"
Sep 11 18:36:05.742086 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.741981    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/95246b0b-1a60-4e86-b309-34cbf8258bdc-proc\") pod \"perf-node-gather-daemonset-bs8lr\" (UID: \"95246b0b-1a60-4e86-b309-34cbf8258bdc\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"
Sep 11 18:36:05.742086 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.742022    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95246b0b-1a60-4e86-b309-34cbf8258bdc-sys\") pod \"perf-node-gather-daemonset-bs8lr\" (UID: \"95246b0b-1a60-4e86-b309-34cbf8258bdc\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"
Sep 11 18:36:05.742086 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.742081    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pf6r\" (UniqueName: \"kubernetes.io/projected/95246b0b-1a60-4e86-b309-34cbf8258bdc-kube-api-access-4pf6r\") pod \"perf-node-gather-daemonset-bs8lr\" (UID: \"95246b0b-1a60-4e86-b309-34cbf8258bdc\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"
Sep 11 18:36:05.742205 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.742129    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/95246b0b-1a60-4e86-b309-34cbf8258bdc-podres\") pod \"perf-node-gather-daemonset-bs8lr\" (UID: \"95246b0b-1a60-4e86-b309-34cbf8258bdc\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"
Sep 11 18:36:05.843214 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.843174    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95246b0b-1a60-4e86-b309-34cbf8258bdc-lib-modules\") pod \"perf-node-gather-daemonset-bs8lr\" (UID: \"95246b0b-1a60-4e86-b309-34cbf8258bdc\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"
Sep 11 18:36:05.843375 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.843225    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/95246b0b-1a60-4e86-b309-34cbf8258bdc-proc\") pod \"perf-node-gather-daemonset-bs8lr\" (UID: \"95246b0b-1a60-4e86-b309-34cbf8258bdc\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"
Sep 11 18:36:05.843375 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.843244    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95246b0b-1a60-4e86-b309-34cbf8258bdc-sys\") pod \"perf-node-gather-daemonset-bs8lr\" (UID: \"95246b0b-1a60-4e86-b309-34cbf8258bdc\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"
Sep 11 18:36:05.843375 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.843280    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pf6r\" (UniqueName: \"kubernetes.io/projected/95246b0b-1a60-4e86-b309-34cbf8258bdc-kube-api-access-4pf6r\") pod \"perf-node-gather-daemonset-bs8lr\" (UID: \"95246b0b-1a60-4e86-b309-34cbf8258bdc\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"
Sep 11 18:36:05.843375 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.843313    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/95246b0b-1a60-4e86-b309-34cbf8258bdc-podres\") pod \"perf-node-gather-daemonset-bs8lr\" (UID: \"95246b0b-1a60-4e86-b309-34cbf8258bdc\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"
Sep 11 18:36:05.843375 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.843317    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/95246b0b-1a60-4e86-b309-34cbf8258bdc-proc\") pod \"perf-node-gather-daemonset-bs8lr\" (UID: \"95246b0b-1a60-4e86-b309-34cbf8258bdc\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"
Sep 11 18:36:05.843375 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.843354    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/95246b0b-1a60-4e86-b309-34cbf8258bdc-lib-modules\") pod \"perf-node-gather-daemonset-bs8lr\" (UID: \"95246b0b-1a60-4e86-b309-34cbf8258bdc\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"
Sep 11 18:36:05.843574 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.843377    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/95246b0b-1a60-4e86-b309-34cbf8258bdc-sys\") pod \"perf-node-gather-daemonset-bs8lr\" (UID: \"95246b0b-1a60-4e86-b309-34cbf8258bdc\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"
Sep 11 18:36:05.843574 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.843423    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/95246b0b-1a60-4e86-b309-34cbf8258bdc-podres\") pod \"perf-node-gather-daemonset-bs8lr\" (UID: \"95246b0b-1a60-4e86-b309-34cbf8258bdc\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"
Sep 11 18:36:05.859128 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:05.859100    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pf6r\" (UniqueName: \"kubernetes.io/projected/95246b0b-1a60-4e86-b309-34cbf8258bdc-kube-api-access-4pf6r\") pod \"perf-node-gather-daemonset-bs8lr\" (UID: \"95246b0b-1a60-4e86-b309-34cbf8258bdc\") " pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"
Sep 11 18:36:06.013618 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:06.013583    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"
Sep 11 18:36:06.177152 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:06.177122    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/2.log"
Sep 11 18:36:06.181168 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:06.181148    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-k8qz2_84f48f44-bac5-4f61-bcc5-7dc52f0d1628/console-operator/3.log"
Sep 11 18:36:06.291294 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:06.291265    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"]
Sep 11 18:36:06.292202 ip-10-0-128-29 kubenswrapper[2560]: W0911 18:36:06.292178    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod95246b0b_1a60_4e86_b309_34cbf8258bdc.slice/crio-cc5129141f3f89f67cd0b409ffbfb91d126e3aa5a91ccd78799df50e07bbddfd WatchSource:0}: Error finding container cc5129141f3f89f67cd0b409ffbfb91d126e3aa5a91ccd78799df50e07bbddfd: Status 404 returned error can't find the container with id cc5129141f3f89f67cd0b409ffbfb91d126e3aa5a91ccd78799df50e07bbddfd
Sep 11 18:36:06.622220 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:06.622126    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr" event={"ID":"95246b0b-1a60-4e86-b309-34cbf8258bdc","Type":"ContainerStarted","Data":"7ecc21fa5d6d245e3e3e00d444a4a5aa71b0ba1f91c80eafe2e29669e5193ecc"}
Sep 11 18:36:06.622220 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:06.622189    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr" event={"ID":"95246b0b-1a60-4e86-b309-34cbf8258bdc","Type":"ContainerStarted","Data":"cc5129141f3f89f67cd0b409ffbfb91d126e3aa5a91ccd78799df50e07bbddfd"}
Sep 11 18:36:06.622220 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:06.622219    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"
Sep 11 18:36:06.643005 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:06.642956    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr" podStartSLOduration=1.6429412810000001 podStartE2EDuration="1.642941281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:06.641906683 +0000 UTC m=+2524.485142123" watchObservedRunningTime="2026-09-11 18:36:06.642941281 +0000 UTC m=+2524.486176722"
Sep 11 18:36:06.729701 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:06.729668    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-79bff6d9c6-fwbmn_fd520e35-84d7-4368-b0c9-96d5c3b10a15/console/0.log"
Sep 11 18:36:06.775934 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:06.775899    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-nhkpk_9222f535-e0de-48b8-9026-ff6b962d0e22/download-server/0.log"
Sep 11 18:36:07.484298 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:07.484263    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-2tttk_4d451b56-86ce-460a-813f-054ec25c4e8b/volume-data-source-validator/0.log"
Sep 11 18:36:08.683995 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:08.683957    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-45lfw_be400dcd-4cff-4bc9-b572-b714103a5c46/dns/0.log"
Sep 11 18:36:08.706563 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:08.706538    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-45lfw_be400dcd-4cff-4bc9-b572-b714103a5c46/kube-rbac-proxy/0.log"
Sep 11 18:36:08.855868 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:08.855836    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pjbh8_8970f95d-bfe6-4d2c-97ea-be9489333243/dns-node-resolver/0.log"
Sep 11 18:36:09.476041 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:09.476013    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fsfql_91dbf02a-fc59-4467-9c9f-aafb5b1594c4/node-ca/0.log"
Sep 11 18:36:11.130514 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:11.130486    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ff57cc6b9-bnldt_cd43172e-8f56-47b0-a780-fa9232ff2d41/kube-auth-proxy/0.log"
Sep 11 18:36:12.249388 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:12.249360    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dfv62_c3ca3ce1-7383-4588-a129-d69fedda85b3/serve-healthcheck-canary/0.log"
Sep 11 18:36:12.637658 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:12.637579    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mt88g/perf-node-gather-daemonset-bs8lr"
Sep 11 18:36:12.841698 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:12.841665    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-26crr_05d87087-b55d-48cd-863f-1b1e6e7e4e72/insights-operator/0.log"
Sep 11 18:36:12.842160 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:12.842144    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-26crr_05d87087-b55d-48cd-863f-1b1e6e7e4e72/insights-operator/1.log"
Sep 11 18:36:13.070496 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:13.070468    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-chg9t_025d8e60-0c15-47a4-bd38-7c29f2e30f9e/kube-rbac-proxy/0.log"
Sep 11 18:36:13.103018 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:13.102993    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-chg9t_025d8e60-0c15-47a4-bd38-7c29f2e30f9e/exporter/0.log"
Sep 11 18:36:13.131153 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:13.131128    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-chg9t_025d8e60-0c15-47a4-bd38-7c29f2e30f9e/extractor/0.log"
Sep 11 18:36:16.148391 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:16.148313    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7_e01fbe9e-fc87-4548-8223-64276fabad17/manager/0.log"
Sep 11 18:36:16.208023 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:16.207992    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bdfbdc9d-b6qp7_e01fbe9e-fc87-4548-8223-64276fabad17/copy-manifests/0.log"
Sep 11 18:36:16.772548 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:16.772485    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ff57cc6b9-bnldt_cd43172e-8f56-47b0-a780-fa9232ff2d41/kube-auth-proxy/0.log"
Sep 11 18:36:17.893204 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:17.893175    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-ffbd766d4-9mjln_6af40138-8dae-46c2-a76c-5163b937a4a8/manager/0.log"
Sep 11 18:36:19.126731 ip-10-0-128-29 kubenswrapper[2560]: I0911 18:36:19.126701    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7ff57cc6b9-bnldt_cd43172e-8f56-47b0-a780-fa9232ff2d41/kube-auth-proxy/0.log"