Sep 11 19:24:21.834944 ip-10-0-138-43 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 19:24:21.834957 ip-10-0-138-43 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 19:24:21.834966 ip-10-0-138-43 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 19:24:21.835174 ip-10-0-138-43 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 19:24:31.854208 ip-10-0-138-43 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 19:24:31.854224 ip-10-0-138-43 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 0f6d9bdb663b41b195c3e9b64c0b5432 --
Sep 11 19:26:23.278982 ip-10-0-138-43 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 19:26:23.624026 ip-10-0-138-43 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 19:26:23.624026 ip-10-0-138-43 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 19:26:23.624026 ip-10-0-138-43 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 19:26:23.624026 ip-10-0-138-43 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 19:26:23.624026 ip-10-0-138-43 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 19:26:23.626310 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.626230    2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 19:26:23.629231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629215    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 19:26:23.629231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629231    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 19:26:23.629294 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629235    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 19:26:23.629294 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629238    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 19:26:23.629294 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629242    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 19:26:23.629294 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629247    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 19:26:23.629294 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629251    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 19:26:23.629294 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629254    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 19:26:23.629294 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629257    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 19:26:23.629294 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629260    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 19:26:23.629294 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629263    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 19:26:23.629294 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629266    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 19:26:23.629294 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629268    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 19:26:23.629294 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629271    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 19:26:23.629294 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629273    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 19:26:23.629294 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629276    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 19:26:23.629294 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629278    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 19:26:23.629294 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629281    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 19:26:23.629294 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629284    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 19:26:23.629294 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629286    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 19:26:23.629294 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629289    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 19:26:23.629763 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629297    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 19:26:23.629763 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629300    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 19:26:23.629763 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629303    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 19:26:23.629763 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629306    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 19:26:23.629763 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629309    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 19:26:23.629763 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629312    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 19:26:23.629763 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629314    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 19:26:23.629763 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629317    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 19:26:23.629763 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629319    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 19:26:23.629763 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629322    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 19:26:23.629763 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629324    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 19:26:23.629763 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629327    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 19:26:23.629763 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629329    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 19:26:23.629763 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629331    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 19:26:23.629763 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629334    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 19:26:23.629763 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629337    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 19:26:23.629763 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629339    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 19:26:23.629763 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629342    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 19:26:23.629763 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629344    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 19:26:23.629763 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629347    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 19:26:23.630280 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629349    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 19:26:23.630280 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629352    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 19:26:23.630280 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629354    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 19:26:23.630280 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629357    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 19:26:23.630280 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629359    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 19:26:23.630280 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629362    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 19:26:23.630280 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629364    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 19:26:23.630280 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629366    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 19:26:23.630280 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629369    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 19:26:23.630280 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629371    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 19:26:23.630280 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629374    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 19:26:23.630280 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629377    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 19:26:23.630280 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629380    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 19:26:23.630280 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629383    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 19:26:23.630280 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629386    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 19:26:23.630280 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629388    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 19:26:23.630280 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629391    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 19:26:23.630280 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629393    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 19:26:23.630280 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629396    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 19:26:23.630280 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629398    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 19:26:23.630770 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629401    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 19:26:23.630770 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629403    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 19:26:23.630770 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629406    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 19:26:23.630770 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629408    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 19:26:23.630770 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629410    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 19:26:23.630770 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629414    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 19:26:23.630770 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629418    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 19:26:23.630770 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629421    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 19:26:23.630770 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629424    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 19:26:23.630770 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629427    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 19:26:23.630770 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629430    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 19:26:23.630770 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629432    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 19:26:23.630770 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629435    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 19:26:23.630770 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629438    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 19:26:23.630770 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629440    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 19:26:23.630770 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629442    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 19:26:23.630770 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629445    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 19:26:23.630770 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629448    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 19:26:23.630770 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629450    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 19:26:23.631231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629453    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 19:26:23.631231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629456    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 19:26:23.631231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629460    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 19:26:23.631231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629463    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 19:26:23.631231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629466    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 19:26:23.631231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629469    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 19:26:23.631231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629473    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 19:26:23.631231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629476    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 19:26:23.631231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629478    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 19:26:23.631231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629481    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 19:26:23.631231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629483    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 19:26:23.631231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629486    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 19:26:23.631231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629489    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 19:26:23.631231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629491    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 19:26:23.631231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629494    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 19:26:23.631231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629496    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 19:26:23.631231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629499    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 19:26:23.631231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629501    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 19:26:23.631231 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.629503    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 19:26:23.631719 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630231    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 19:26:23.631719 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630239    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 19:26:23.631719 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630242    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 19:26:23.631719 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630246    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 19:26:23.631719 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630249    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 19:26:23.631719 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630253    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 19:26:23.631719 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630257    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 19:26:23.631719 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630259    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 19:26:23.631719 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630262    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 19:26:23.631719 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630265    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 19:26:23.631719 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630267    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 19:26:23.631719 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630270    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 19:26:23.631719 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630272    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 19:26:23.631719 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630275    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 19:26:23.631719 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630278    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 19:26:23.631719 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630280    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 19:26:23.631719 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630283    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 19:26:23.631719 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630286    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 19:26:23.631719 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630289    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 19:26:23.632204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630292    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 19:26:23.632204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630295    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 19:26:23.632204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630298    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 19:26:23.632204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630300    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 19:26:23.632204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630303    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 19:26:23.632204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630306    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 19:26:23.632204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630308    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 19:26:23.632204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630311    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 19:26:23.632204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630313    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 19:26:23.632204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630316    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 19:26:23.632204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630318    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 19:26:23.632204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630321    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 19:26:23.632204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630323    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 19:26:23.632204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630326    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 19:26:23.632204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630328    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 19:26:23.632204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630331    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 19:26:23.632204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630333    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 19:26:23.632204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630336    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 19:26:23.632204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630338    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 19:26:23.632204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630341    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 19:26:23.632741 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630343    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 19:26:23.632741 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630346    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 19:26:23.632741 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630348    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 19:26:23.632741 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630351    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 19:26:23.632741 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630353    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 19:26:23.632741 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630356    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 19:26:23.632741 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630358    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 19:26:23.632741 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630361    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 19:26:23.632741 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630363    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 19:26:23.632741 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630366    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 19:26:23.632741 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630368    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 19:26:23.632741 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630372    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 19:26:23.632741 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630376    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 19:26:23.632741 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630380    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 19:26:23.632741 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630383    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 19:26:23.632741 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630386    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 19:26:23.632741 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630388    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 19:26:23.632741 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630391    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 19:26:23.632741 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630394    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 19:26:23.633209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630398    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 19:26:23.633209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630401    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 19:26:23.633209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630404    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 19:26:23.633209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630407    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 19:26:23.633209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630410    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 19:26:23.633209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630413    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 19:26:23.633209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630415    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 19:26:23.633209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630418    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 19:26:23.633209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630420    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 19:26:23.633209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630423    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 19:26:23.633209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630425    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 19:26:23.633209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630427    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 19:26:23.633209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630430    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 19:26:23.633209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630433    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 19:26:23.633209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630435    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 19:26:23.633209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630438    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 19:26:23.633209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630440    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 19:26:23.633209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630443    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 19:26:23.633209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630447    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 19:26:23.633209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630450    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 19:26:23.633693 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630453    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 19:26:23.633693 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630455    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 19:26:23.633693 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630458    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 19:26:23.633693 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630460    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 19:26:23.633693 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630463    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 19:26:23.633693 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630467    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 19:26:23.633693 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630470    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 19:26:23.633693 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630473    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 19:26:23.633693 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630476    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 19:26:23.633693 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630478    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 19:26:23.633693 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630481    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 19:26:23.633693 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630484    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 19:26:23.633693 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630486    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 19:26:23.633693 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630489    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 19:26:23.633693 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630492    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 19:26:23.633693 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630494    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 19:26:23.633693 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630497    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 19:26:23.633693 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630500    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 19:26:23.633693 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630504    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 19:26:23.633693 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630507    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 19:26:23.634192 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630510    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 19:26:23.634192 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630887    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 19:26:23.634192 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630892    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 19:26:23.634192 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630895    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 19:26:23.634192 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630898    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 19:26:23.634192 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630901    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 19:26:23.634192 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630904    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 19:26:23.634192 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630907    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 19:26:23.634192 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630909    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 19:26:23.634192 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630912    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 19:26:23.634192 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630915    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 19:26:23.634192 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630918    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 19:26:23.634192 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630920    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 19:26:23.634192 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630922    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 19:26:23.634192 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630925    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 19:26:23.634192 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630928    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 19:26:23.634192 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630931    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 19:26:23.634192 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630933    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 19:26:23.634192 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630938    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 19:26:23.634192 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630941    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 19:26:23.634668 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630945    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 19:26:23.634668 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630949    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 19:26:23.634668 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630952    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 19:26:23.634668 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630955    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 19:26:23.634668 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630957    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 19:26:23.634668 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630960    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 19:26:23.634668 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630962    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 19:26:23.634668 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630965    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 19:26:23.634668 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630967    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 19:26:23.634668 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630970    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 19:26:23.634668 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630972    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 19:26:23.634668 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630976    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 19:26:23.634668 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630979    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 19:26:23.634668 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630981    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 19:26:23.634668 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630983    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 19:26:23.634668 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630986    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 19:26:23.634668 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630989    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 19:26:23.634668 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630991    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 19:26:23.634668 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630994    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 19:26:23.635222 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.630998    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 19:26:23.635222 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631000    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 19:26:23.635222 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631003    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 19:26:23.635222 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631005    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 19:26:23.635222 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631008    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 19:26:23.635222 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631011    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 19:26:23.635222 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631013    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 19:26:23.635222 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631018    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 19:26:23.635222 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631021    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 19:26:23.635222 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631023    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 19:26:23.635222 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631026    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 19:26:23.635222 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631029    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 19:26:23.635222 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631031    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 19:26:23.635222 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631034    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 19:26:23.635222 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631036    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 19:26:23.635222 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631039    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 19:26:23.635222 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631041    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 19:26:23.635222 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631043    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 19:26:23.635222 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631046    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 19:26:23.635222 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631049    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 19:26:23.635695 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631051    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 19:26:23.635695 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631053    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 19:26:23.635695 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631056    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 19:26:23.635695 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631058    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 19:26:23.635695 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631063    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 19:26:23.635695 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631066    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 19:26:23.635695 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631069    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 19:26:23.635695 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631072    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 19:26:23.635695 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631075    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 19:26:23.635695 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631078    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 19:26:23.635695 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631080    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 19:26:23.635695 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631083    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 19:26:23.635695 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631087    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 19:26:23.635695 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631089    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 19:26:23.635695 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631092    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 19:26:23.635695 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631095    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 19:26:23.635695 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631097    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 19:26:23.635695 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631100    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 19:26:23.635695 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631102    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 19:26:23.636168 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631106    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 19:26:23.636168 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631109    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 19:26:23.636168 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631111    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 19:26:23.636168 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631114    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 19:26:23.636168 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631116    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 19:26:23.636168 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631119    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 19:26:23.636168 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631122    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 19:26:23.636168 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631124    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 19:26:23.636168 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631127    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 19:26:23.636168 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631129    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 19:26:23.636168 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631132    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 19:26:23.636168 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631134    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 19:26:23.636168 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631137    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 19:26:23.636168 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631139    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 19:26:23.636168 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631142    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 19:26:23.636168 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631145    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 19:26:23.636168 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631147    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 19:26:23.636168 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631150    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 19:26:23.636168 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631153    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 19:26:23.636168 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631156    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631158    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631161    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631237    2558 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631270    2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631276    2558 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631280    2558 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631285    2558 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631288    2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631292    2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631296    2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631299    2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631302    2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631306    2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631309    2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631312    2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631315    2558 flags.go:64] FLAG: --cgroup-root=""
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631318    2558 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631321    2558 flags.go:64] FLAG: --client-ca-file=""
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631324    2558 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631326    2558 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631331    2558 flags.go:64] FLAG: --cluster-domain=""
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631333    2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631336    2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 19:26:23.636659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631339    2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631342    2558 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631346    2558 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631349    2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631351    2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631355    2558 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631358    2558 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631361    2558 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631364    2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631367    2558 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631370    2558 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631376    2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631379    2558 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631381    2558 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631384    2558 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631387    2558 flags.go:64] FLAG: --enable-server="true"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631391    2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631398    2558 flags.go:64] FLAG: --event-burst="100"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631401    2558 flags.go:64] FLAG: --event-qps="50"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631403    2558 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631406    2558 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631409    2558 flags.go:64] FLAG: --eviction-hard=""
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631413    2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631416    2558 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631419    2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 19:26:23.637284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631435    2558 flags.go:64] FLAG: --eviction-soft=""
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631440    2558 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631443    2558 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631446    2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631449    2558 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631452    2558 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631455    2558 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631457    2558 flags.go:64] FLAG: --feature-gates=""
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631461    2558 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631464    2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631467    2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631472    2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631475    2558 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631478    2558 flags.go:64] FLAG: --help="false"
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631480    2558 flags.go:64] FLAG: --hostname-override="ip-10-0-138-43.ec2.internal"
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631483    2558 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631486    2558 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631489    2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631492    2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631495    2558 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631498    2558 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631501    2558 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631504    2558 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 19:26:23.637899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631507    2558 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631510    2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631513    2558 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631516    2558 flags.go:64] FLAG: --kube-reserved=""
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631518    2558 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631521    2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631525    2558 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631527    2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631530    2558 flags.go:64] FLAG: --lock-file=""
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631533    2558 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631536    2558 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631539    2558 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631544    2558 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631546    2558 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631549    2558 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631552    2558 flags.go:64] FLAG: --logging-format="text"
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631555    2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631558    2558 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631561    2558 flags.go:64] FLAG: --manifest-url=""
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631563    2558 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631569    2558 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631572    2558 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631576    2558 flags.go:64] FLAG: --max-pods="110"
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631579    2558 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631582    2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 19:26:23.638507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631586    2558 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631589    2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631592    2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631595    2558 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631597    2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631604    2558 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631607    2558 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631610    2558 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631613    2558 flags.go:64] FLAG: --pod-cidr=""
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631616    2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631621    2558 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631624    2558 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631627    2558 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631630    2558 flags.go:64] FLAG: --port="10250"
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631633    2558 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631636    2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b7af0b292449652b"
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631639    2558 flags.go:64] FLAG: --qos-reserved=""
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631642    2558 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631645    2558 flags.go:64] FLAG: --register-node="true"
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631647    2558 flags.go:64] FLAG: --register-with-taints=""
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631651    2558 flags.go:64] FLAG: --registry-burst="10"
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631654    2558 flags.go:64] FLAG: --registry-qps="5"
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631657    2558 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631659    2558 flags.go:64] FLAG: --reserved-memory=""
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631663    2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 19:26:23.639146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631666    2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631669    2558 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631673    2558 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631676    2558 flags.go:64] FLAG: --runonce="false"
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631679    2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631682    2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631685    2558 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631688    2558 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631691    2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631694    2558 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631697    2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631700    2558 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631703    2558 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631718    2558 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631721    2558 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631724    2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631728    2558 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631731    2558 flags.go:64] FLAG: --system-cgroups=""
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631734    2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631741    2558 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631744    2558 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631747    2558 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631752    2558 flags.go:64] FLAG: --tls-min-version=""
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631755    2558 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631758    2558 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 19:26:23.639766 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631760    2558 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631764    2558 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631767    2558 flags.go:64] FLAG: --v="2"
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631771    2558 flags.go:64] FLAG: --version="false"
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631775    2558 flags.go:64] FLAG: --vmodule=""
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631779    2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.631782    2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631897    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631901    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631905    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631908    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631912    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631915    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631917    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631920    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631923    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631925    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631928    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631931    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631935    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631939    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 19:26:23.640364 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631942    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 19:26:23.640888 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631944    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 19:26:23.640888 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631947    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 19:26:23.640888 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631950    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 19:26:23.640888 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631953    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 19:26:23.640888 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631957    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 19:26:23.640888 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631959    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 19:26:23.640888 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631962    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 19:26:23.640888 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631965    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 19:26:23.640888 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631969    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 19:26:23.640888 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631972    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 19:26:23.640888 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631975    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 19:26:23.640888 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631978    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 19:26:23.640888 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631981    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 19:26:23.640888 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631984    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 19:26:23.640888 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631986    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 19:26:23.640888 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631989    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 19:26:23.640888 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631992    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 19:26:23.640888 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631995    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 19:26:23.640888 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.631997    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 19:26:23.640888 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632001    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 19:26:23.641375 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632004    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 19:26:23.641375 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632006    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 19:26:23.641375 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632009    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 19:26:23.641375 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632012    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 19:26:23.641375 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632014    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 19:26:23.641375 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632017    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 19:26:23.641375 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632019    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 19:26:23.641375 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632022    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 19:26:23.641375 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632025    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 19:26:23.641375 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632028    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 19:26:23.641375 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632030    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 19:26:23.641375 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632033    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 19:26:23.641375 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632036    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 19:26:23.641375 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632038    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 19:26:23.641375 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632042    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 19:26:23.641375 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632045    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 19:26:23.641375 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632051    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 19:26:23.641375 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632054    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 19:26:23.641375 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632056    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 19:26:23.641375 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632059    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 19:26:23.641921 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632062    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 19:26:23.641921 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632064    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 19:26:23.641921 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632067    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 19:26:23.641921 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632070    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 19:26:23.641921 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632072    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 19:26:23.641921 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632075    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 19:26:23.641921 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632077    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 19:26:23.641921 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632080    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 19:26:23.641921 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632083    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 19:26:23.641921 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632085    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 19:26:23.641921 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632088    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 19:26:23.641921 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632092    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 19:26:23.641921 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632094    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 19:26:23.641921 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632097    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 19:26:23.641921 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632100    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 19:26:23.641921 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632102    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 19:26:23.641921 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632105    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 19:26:23.641921 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632108    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 19:26:23.641921 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632110    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 19:26:23.641921 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632113    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 19:26:23.642420 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632115    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 19:26:23.642420 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632118    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 19:26:23.642420 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632120    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 19:26:23.642420 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632123    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 19:26:23.642420 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632125    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 19:26:23.642420 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632128    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 19:26:23.642420 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632131    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 19:26:23.642420 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632133    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 19:26:23.642420 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632152    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 19:26:23.642420 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632158    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 19:26:23.642420 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632162    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 19:26:23.642420 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632166    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 19:26:23.642420 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632171    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 19:26:23.642420 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632175    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 19:26:23.642420 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632177    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 19:26:23.642420 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632180    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 19:26:23.642420 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632182    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 19:26:23.642420 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632185    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 19:26:23.642420 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632188    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 19:26:23.642420 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632190    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 19:26:23.642929 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632193    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 19:26:23.642929 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632196    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 19:26:23.642929 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632198    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 19:26:23.642929 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.632202    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 19:26:23.642929 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.632791    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 19:26:23.642929 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.639910    2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 19:26:23.642929 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.639926    2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 19:26:23.642929 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.639944    2558 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 19:26:23.642929 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.639950    2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 19:26:23.642929 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640009    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 19:26:23.642929 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640013    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 19:26:23.642929 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640017    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 19:26:23.642929 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640020    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 19:26:23.642929 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640022    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 19:26:23.642929 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640025    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 19:26:23.643329 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640028    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 19:26:23.643329 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640031    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 19:26:23.643329 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640034    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 19:26:23.643329 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640037    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 19:26:23.643329 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640040    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 19:26:23.643329 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640043    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 19:26:23.643329 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640046    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 19:26:23.643329 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640049    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 19:26:23.643329 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640052    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 19:26:23.643329 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640055    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 19:26:23.643329 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640057    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 19:26:23.643329 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640060    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 19:26:23.643329 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640063    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 19:26:23.643329 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640065    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 19:26:23.643329 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640068    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 19:26:23.643329 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640070    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 19:26:23.643329 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640073    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 19:26:23.643329 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640075    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 19:26:23.643329 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640078    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 19:26:23.643329 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640080    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 19:26:23.643809 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640083    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 19:26:23.643809 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640085    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 19:26:23.643809 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640088    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 19:26:23.643809 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640098    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 19:26:23.643809 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640101    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 19:26:23.643809 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640104    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 19:26:23.643809 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640106    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 19:26:23.643809 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640109    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 19:26:23.643809 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640111    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 19:26:23.643809 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640114    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 19:26:23.643809 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640116    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 19:26:23.643809 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640119    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 19:26:23.643809 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640122    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 19:26:23.643809 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640124    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 19:26:23.643809 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640127    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 19:26:23.643809 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640130    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 19:26:23.643809 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640132    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 19:26:23.643809 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640135    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 19:26:23.643809 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640137    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 19:26:23.643809 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640140    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 19:26:23.644298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640143    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 19:26:23.644298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640145    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 19:26:23.644298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640148    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 19:26:23.644298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640150    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 19:26:23.644298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640153    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 19:26:23.644298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640155    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 19:26:23.644298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640158    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 19:26:23.644298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640160    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 19:26:23.644298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640163    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 19:26:23.644298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640165    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 19:26:23.644298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640168    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 19:26:23.644298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640170    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 19:26:23.644298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640173    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 19:26:23.644298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640175    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 19:26:23.644298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640178    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 19:26:23.644298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640181    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 19:26:23.644298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640189    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 19:26:23.644298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640192    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 19:26:23.644298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640194    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 19:26:23.644298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640197    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 19:26:23.644813 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640199    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 19:26:23.644813 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640202    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 19:26:23.644813 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640204    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 19:26:23.644813 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640207    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 19:26:23.644813 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640209    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 19:26:23.644813 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640212    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 19:26:23.644813 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640214    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 19:26:23.644813 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640217    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 19:26:23.644813 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640219    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 19:26:23.644813 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640222    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 19:26:23.644813 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640224    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 19:26:23.644813 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640227    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 19:26:23.644813 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640229    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 19:26:23.644813 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640232    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 19:26:23.644813 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640234    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 19:26:23.644813 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640236    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 19:26:23.644813 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640239    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 19:26:23.644813 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640241    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 19:26:23.644813 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640245    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 19:26:23.644813 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640249    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 19:26:23.645322 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640252    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 19:26:23.645322 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640255    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 19:26:23.645322 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640258    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 19:26:23.645322 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640260    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 19:26:23.645322 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640263    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 19:26:23.645322 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640267    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 19:26:23.645322 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640270    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 19:26:23.645322 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640273    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 19:26:23.645322 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640276    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 19:26:23.645322 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640285    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 19:26:23.645322 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640288    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 19:26:23.645322 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640291    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 19:26:23.645322 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640294    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 19:26:23.645322 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.640300    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 19:26:23.645675 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640445    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 19:26:23.645675 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640453    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 19:26:23.645675 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640456    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 19:26:23.645675 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640459    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 19:26:23.645675 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640462    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 19:26:23.645675 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640464    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 19:26:23.645675 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640467    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 19:26:23.645675 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640470    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 19:26:23.645675 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640472    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 19:26:23.645675 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640475    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 19:26:23.645675 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640477    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 19:26:23.645675 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640480    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 19:26:23.645675 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640482    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 19:26:23.645675 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640485    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 19:26:23.645675 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640488    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 19:26:23.645675 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640490    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 19:26:23.645675 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640493    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 19:26:23.645675 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640495    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 19:26:23.645675 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640498    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 19:26:23.645675 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640500    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 19:26:23.646209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640503    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 19:26:23.646209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640505    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 19:26:23.646209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640508    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 19:26:23.646209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640510    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 19:26:23.646209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640512    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 19:26:23.646209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640515    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 19:26:23.646209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640518    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 19:26:23.646209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640521    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 19:26:23.646209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640529    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 19:26:23.646209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640532    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 19:26:23.646209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640534    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 19:26:23.646209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640537    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 19:26:23.646209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640540    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 19:26:23.646209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640542    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 19:26:23.646209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640545    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 19:26:23.646209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640548    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 19:26:23.646209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640550    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 19:26:23.646209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640553    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 19:26:23.646209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640555    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 19:26:23.646209 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640558    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 19:26:23.646687 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640560    2558 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 19:26:23.646687 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640563    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 19:26:23.646687 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640565    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 19:26:23.646687 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640568    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 19:26:23.646687 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640570    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 19:26:23.646687 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640573    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 19:26:23.646687 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640575    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 19:26:23.646687 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640578    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 19:26:23.646687 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640580    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 19:26:23.646687 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640583    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 19:26:23.646687 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640585    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 19:26:23.646687 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640588    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 19:26:23.646687 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640590    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 19:26:23.646687 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640593    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 19:26:23.646687 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640595    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 19:26:23.646687 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640599    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 19:26:23.646687 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640602    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 19:26:23.646687 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640606    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 19:26:23.646687 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640609    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 19:26:23.647156 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640612    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 19:26:23.647156 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640614    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 19:26:23.647156 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640627    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 19:26:23.647156 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640631    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 19:26:23.647156 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640633    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 19:26:23.647156 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640636    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 19:26:23.647156 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640638    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 19:26:23.647156 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640641    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 19:26:23.647156 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640644    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 19:26:23.647156 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640656    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 19:26:23.647156 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640660    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 19:26:23.647156 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640663    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 19:26:23.647156 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640666    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 19:26:23.647156 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640669    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 19:26:23.647156 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640671    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 19:26:23.647156 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640674    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 19:26:23.647156 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640678    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 19:26:23.647156 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640680    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 19:26:23.647156 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640683    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 19:26:23.647685 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640686    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 19:26:23.647685 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640689    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 19:26:23.647685 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640691    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 19:26:23.647685 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640694    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 19:26:23.647685 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640696    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 19:26:23.647685 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640699    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 19:26:23.647685 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640701    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 19:26:23.647685 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640704    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 19:26:23.647685 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640721    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 19:26:23.647685 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640724    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 19:26:23.647685 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640727    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 19:26:23.647685 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640729    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 19:26:23.647685 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640732    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 19:26:23.647685 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640734    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 19:26:23.647685 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640737    2558 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 19:26:23.647685 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640740    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 19:26:23.647685 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640750    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 19:26:23.647685 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640752    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 19:26:23.647685 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640755    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 19:26:23.647685 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640757    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 19:26:23.648276 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:23.640760    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 19:26:23.648276 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.640765    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 19:26:23.648276 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.640881    2558 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 19:26:23.648276 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.643084    2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 19:26:23.648276 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.643869    2558 server.go:1019] "Starting client certificate rotation"
Sep 11 19:26:23.648276 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.643968    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 19:26:23.648276 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.644000    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 19:26:23.648276 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.646524    2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 19:26:23.652881 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.652858    2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 19:26:23.665600 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.665583    2558 log.go:25] "Validated CRI v1 runtime API"
Sep 11 19:26:23.668247 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.668232    2558 log.go:25] "Validated CRI v1 image API"
Sep 11 19:26:23.669061 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.669043    2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 19:26:23.672101 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.672083    2558 fs.go:135] Filesystem UUIDs: map[025aa06e-3414-4f6a-a52d-28a8b099fa43:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fd3348c8-3ce6-4a7e-ae86-6b9915d00743:/dev/nvme0n1p4]
Sep 11 19:26:23.672174 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.672100    2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 19:26:23.675415 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.675400    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 19:26:23.678290 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.678184    2558 manager.go:216] Machine: {Timestamp:2026-09-11 19:26:23.676949308 +0000 UTC m=+0.294118790 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100393 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f88c01b014a069367ce8500cb37bb SystemUUID:ec2f88c0-1b01-4a06-9367-ce8500cb37bb BootID:0f6d9bdb-663b-41b1-95c3-e9b64c0b5432 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ac:a1:4a:c1:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ac:a1:4a:c1:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:3c:99:06:2c:65 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 19:26:23.678290 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.678289    2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 19:26:23.678397 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.678361    2558 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 19:26:23.681360 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.681340    2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 19:26:23.681491 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.681362    2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-43.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 19:26:23.682088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.682077    2558 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 19:26:23.682127 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.682091    2558 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 19:26:23.682127 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.682098    2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 19:26:23.682967 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.682957    2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 19:26:23.684034 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.684024    2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 19:26:23.688740 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.688727    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 19:26:23.688857 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.688847    2558 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 19:26:23.688895 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.688878    2558 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 19:26:23.688895 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.688888    2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 19:26:23.688964 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.688901    2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 19:26:23.688964 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.688911    2558 kubelet.go:398] "Adding apiserver pod source"
Sep 11 19:26:23.688964 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.688919    2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 19:26:23.689286 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.689272    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 19:26:23.689335 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.689299    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 19:26:23.691283 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.691256    2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 19:26:23.693533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.693520    2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 19:26:23.693595 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.693550    2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 19:26:23.695630 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.695616    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 19:26:23.695697 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.695637    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 19:26:23.695697 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.695643    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 19:26:23.695697 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.695648    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 19:26:23.695697 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.695656    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 19:26:23.695697 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.695661    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 19:26:23.695697 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.695674    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 19:26:23.695697 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.695679    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 19:26:23.695697 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.695686    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 19:26:23.695697 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.695691    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 19:26:23.695697 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.695699    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 19:26:23.696014 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.695718    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 19:26:23.696014 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.695741    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 19:26:23.696014 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.695749    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 19:26:23.699060 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.699046    2558 server.go:1268] "Started kubelet"
Sep 11 19:26:23.699236 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.699174    2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 19:26:23.699302 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.699256    2558 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 19:26:23.703263 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.703230    2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 19:26:23.703516 ip-10-0-138-43 systemd[1]: Started Kubernetes Kubelet.
Sep 11 19:26:23.704636 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.703697    2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-43.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 19:26:23.704636 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:23.704080    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 19:26:23.704636 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.703244    2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 19:26:23.704636 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:23.704401    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 19:26:23.705020 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:23.704989    2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 19:26:23.705904 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.705878    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8b848"
Sep 11 19:26:23.705904 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.705895    2558 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 19:26:23.709647 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:23.708285    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-43.ec2.internal.18d45acbe2922e88  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-43.ec2.internal,UID:ip-10-0-138-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-43.ec2.internal,},FirstTimestamp:2026-09-11 19:26:23.699021448 +0000 UTC m=+0.316190930,LastTimestamp:2026-09-11 19:26:23.699021448 +0000 UTC m=+0.316190930,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-43.ec2.internal,}"
Sep 11 19:26:23.711929 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.711909    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 19:26:23.712541 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.712410    2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 19:26:23.712668 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:23.712649    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-43.ec2.internal\" not found"
Sep 11 19:26:23.713286 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.713268    2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 19:26:23.713286 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.713274    2558 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 19:26:23.713393 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.713303    2558 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 19:26:23.713393 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.713352    2558 factory.go:55] Registering systemd factory
Sep 11 19:26:23.713393 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.713366    2558 factory.go:223] Registration of the systemd container factory successfully
Sep 11 19:26:23.713393 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.713378    2558 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 19:26:23.713393 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.713389    2558 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 19:26:23.713604 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.713566    2558 factory.go:153] Registering CRI-O factory
Sep 11 19:26:23.713604 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.713580    2558 factory.go:223] Registration of the crio container factory successfully
Sep 11 19:26:23.713688 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.713626    2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 19:26:23.713688 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.713643    2558 factory.go:103] Registering Raw factory
Sep 11 19:26:23.713688 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.713656    2558 manager.go:1196] Started watching for new ooms in manager
Sep 11 19:26:23.713995 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.713981    2558 manager.go:349] Starting recovery of all containers
Sep 11 19:26:23.719183 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.719155    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8b848"
Sep 11 19:26:23.719699 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:23.719677    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 19:26:23.719796 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:23.719762    2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-43.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 19:26:23.724211 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.724193    2558 manager.go:354] Recovery completed
Sep 11 19:26:23.728345 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.728333    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 19:26:23.730120 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.730104    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 19:26:23.730205 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.730133    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 19:26:23.730205 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.730148    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasSufficientPID"
Sep 11 19:26:23.730860 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.730844    2558 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 19:26:23.730860 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.730856    2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 19:26:23.730970 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.730868    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 19:26:23.730970 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.730875    2558 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 19:26:23.732719 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.732688    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 19:26:23.732780 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.732727    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 19:26:23.732780 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.732741    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasSufficientPID"
Sep 11 19:26:23.732854 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:23.732743    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-43.ec2.internal.18d45acbe46caf13  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-43.ec2.internal,UID:ip-10-0-138-43.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-43.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-43.ec2.internal,},FirstTimestamp:2026-09-11 19:26:23.730118419 +0000 UTC m=+0.347287900,LastTimestamp:2026-09-11 19:26:23.730118419 +0000 UTC m=+0.347287900,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-43.ec2.internal,}"
Sep 11 19:26:23.732954 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.732939    2558 policy_none.go:49] "None policy: Start"
Sep 11 19:26:23.732990 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.732961    2558 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 19:26:23.732990 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.732974    2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 19:26:23.734952 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.734937    2558 policy_none.go:47] "Start"
Sep 11 19:26:23.761307 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.761289    2558 manager.go:337] "Starting Device Plugin manager"
Sep 11 19:26:23.785781 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:23.761331    2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 19:26:23.785781 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.761341    2558 server.go:87] "Starting device plugin registration server"
Sep 11 19:26:23.785781 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.761598    2558 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 19:26:23.785781 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.761611    2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 19:26:23.785781 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.761758    2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 19:26:23.785781 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.761837    2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 19:26:23.785781 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.761844    2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 19:26:23.785781 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:23.762347    2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 19:26:23.785781 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:23.762396    2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-43.ec2.internal\" not found"
Sep 11 19:26:23.805975 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.805947    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 19:26:23.807095 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.807077    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 19:26:23.807167 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.807119    2558 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 19:26:23.807167 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.807137    2558 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 19:26:23.807246 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:23.807172    2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 19:26:23.809694 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.809675    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 19:26:23.861714 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.861685    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 19:26:23.862619 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.862600    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 19:26:23.862698 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.862625    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 19:26:23.862698 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.862640    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasSufficientPID"
Sep 11 19:26:23.862698 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.862662    2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-43.ec2.internal"
Sep 11 19:26:23.871143 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.871129    2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-43.ec2.internal"
Sep 11 19:26:23.871191 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:23.871151    2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-43.ec2.internal\": node \"ip-10-0-138-43.ec2.internal\" not found"
Sep 11 19:26:23.890092 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:23.890034    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-43.ec2.internal\" not found"
Sep 11 19:26:23.907847 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.907829    2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-43.ec2.internal"]
Sep 11 19:26:23.907911 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.907900    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 19:26:23.909142 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.909124    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 19:26:23.909224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.909150    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 19:26:23.909224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.909165    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasSufficientPID"
Sep 11 19:26:23.911330 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.911318    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 19:26:23.911457 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.911445    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal"
Sep 11 19:26:23.911498 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.911472    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 19:26:23.912066 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.912048    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 19:26:23.912157 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.912071    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 19:26:23.912157 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.912083    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasSufficientPID"
Sep 11 19:26:23.912157 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.912100    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 19:26:23.912157 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.912122    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 19:26:23.912157 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.912133    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasSufficientPID"
Sep 11 19:26:23.914187 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.914169    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31104fc89b43315a443aee28694a8f6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal\" (UID: \"f31104fc89b43315a443aee28694a8f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal"
Sep 11 19:26:23.914273 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.914192    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af89a8d622e91b5bb6c931160e64948c-config\") pod \"kube-apiserver-proxy-ip-10-0-138-43.ec2.internal\" (UID: \"af89a8d622e91b5bb6c931160e64948c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-43.ec2.internal"
Sep 11 19:26:23.914273 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.914208    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f31104fc89b43315a443aee28694a8f6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal\" (UID: \"f31104fc89b43315a443aee28694a8f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal"
Sep 11 19:26:23.914382 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.914298    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-43.ec2.internal"
Sep 11 19:26:23.914382 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.914329    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 19:26:23.915211 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.915193    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 19:26:23.915311 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.915223    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 19:26:23.915311 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:23.915237    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeHasSufficientPID"
Sep 11 19:26:23.940219 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:23.940197    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-43.ec2.internal\" not found" node="ip-10-0-138-43.ec2.internal"
Sep 11 19:26:23.944397 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:23.944382    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-43.ec2.internal\" not found" node="ip-10-0-138-43.ec2.internal"
Sep 11 19:26:23.990110 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:23.990094    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-43.ec2.internal\" not found"
Sep 11 19:26:24.015248 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:24.015229    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af89a8d622e91b5bb6c931160e64948c-config\") pod \"kube-apiserver-proxy-ip-10-0-138-43.ec2.internal\" (UID: \"af89a8d622e91b5bb6c931160e64948c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-43.ec2.internal"
Sep 11 19:26:24.015301 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:24.015254    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f31104fc89b43315a443aee28694a8f6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal\" (UID: \"f31104fc89b43315a443aee28694a8f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal"
Sep 11 19:26:24.015301 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:24.015275    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31104fc89b43315a443aee28694a8f6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal\" (UID: \"f31104fc89b43315a443aee28694a8f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal"
Sep 11 19:26:24.015363 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:24.015347    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f31104fc89b43315a443aee28694a8f6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal\" (UID: \"f31104fc89b43315a443aee28694a8f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal"
Sep 11 19:26:24.015418 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:24.015403    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/af89a8d622e91b5bb6c931160e64948c-config\") pod \"kube-apiserver-proxy-ip-10-0-138-43.ec2.internal\" (UID: \"af89a8d622e91b5bb6c931160e64948c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-43.ec2.internal"
Sep 11 19:26:24.015475 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:24.015463    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f31104fc89b43315a443aee28694a8f6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal\" (UID: \"f31104fc89b43315a443aee28694a8f6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal"
Sep 11 19:26:24.090587 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:24.090566    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-43.ec2.internal\" not found"
Sep 11 19:26:24.191249 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:24.191218    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-43.ec2.internal\" not found"
Sep 11 19:26:24.243683 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:24.243646    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal"
Sep 11 19:26:24.248671 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:24.248652    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-43.ec2.internal"
Sep 11 19:26:24.291744 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:24.291702    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-43.ec2.internal\" not found"
Sep 11 19:26:24.392246 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:24.392209    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-43.ec2.internal\" not found"
Sep 11 19:26:24.492753 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:24.492686    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-43.ec2.internal\" not found"
Sep 11 19:26:24.593229 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:24.593192    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-43.ec2.internal\" not found"
Sep 11 19:26:24.644924 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:24.644901    2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 19:26:24.645333 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:24.645047    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 19:26:24.682177 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:24.682154    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 19:26:24.693341 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:24.693312    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-43.ec2.internal\" not found"
Sep 11 19:26:24.730402 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:24.730348    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 19:21:24 +0000 UTC" deadline="2028-04-21 12:57:12.478838171 +0000 UTC"
Sep 11 19:26:24.730402 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:24.730401    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14105h30m47.74844238s"
Sep 11 19:26:24.794491 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:24.794459    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-43.ec2.internal\" not found"
Sep 11 19:26:24.894859 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:24.894831    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-43.ec2.internal\" not found"
Sep 11 19:26:24.902579 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:24.902546    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf89a8d622e91b5bb6c931160e64948c.slice/crio-5b48f2c6d8964fa15dd3a4ff60f71f5ea39a137d754abf1b4f838a6b1530e224 WatchSource:0}: Error finding container 5b48f2c6d8964fa15dd3a4ff60f71f5ea39a137d754abf1b4f838a6b1530e224: Status 404 returned error can't find the container with id 5b48f2c6d8964fa15dd3a4ff60f71f5ea39a137d754abf1b4f838a6b1530e224
Sep 11 19:26:24.902800 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:24.902780    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31104fc89b43315a443aee28694a8f6.slice/crio-533f2fb0f286f80c9b0323e00214591147dae4ba7d4b3587c73eb8165ec1b52b WatchSource:0}: Error finding container 533f2fb0f286f80c9b0323e00214591147dae4ba7d4b3587c73eb8165ec1b52b: Status 404 returned error can't find the container with id 533f2fb0f286f80c9b0323e00214591147dae4ba7d4b3587c73eb8165ec1b52b
Sep 11 19:26:24.909375 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:24.909357    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 19:26:24.919602 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:24.919581    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 19:26:25.013764 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.013675    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-43.ec2.internal"
Sep 11 19:26:25.024063 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.024043    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 19:26:25.025610 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.025597    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal"
Sep 11 19:26:25.035834 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.035815    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 19:26:25.104291 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.104263    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 19:26:25.505879 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.505651    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 19:26:25.691084 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.691051    2558 apiserver.go:52] "Watching apiserver"
Sep 11 19:26:25.700438 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.699878    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 19:26:25.702426 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.702046    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-43.ec2.internal","openshift-dns/node-resolver-z8xdl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal","openshift-multus/multus-additional-cni-plugins-whp4x","openshift-multus/multus-b4f5h","openshift-multus/network-metrics-daemon-zzscf","openshift-network-diagnostics/network-check-target-mdl79","openshift-network-operator/iptables-alerter-7rglf","kube-system/global-pull-secret-syncer-pfnzl","kube-system/konnectivity-agent-l6ktp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp","openshift-cluster-node-tuning-operator/tuned-2wkv2","openshift-image-registry/node-ca-d5jh8","openshift-ovn-kubernetes/ovnkube-node-q9jc9"]
Sep 11 19:26:25.704939 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.704916    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:25.705055 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:25.704996    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdl79" podUID="3402721a-0f03-4681-bc7b-facbf92d31b2"
Sep 11 19:26:25.707098 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.707069    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d5jh8"
Sep 11 19:26:25.709376 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.709255    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pfbwm\""
Sep 11 19:26:25.709804 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.709538    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 19:26:25.709804 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.709761    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 19:26:25.710111 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.710084    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 19:26:25.710503 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.710478    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.712424 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.712402    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 19:26:25.713120 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.712942    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.713219 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.713147    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 19:26:25.713588 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.713333    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 19:26:25.713758 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.713736    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 19:26:25.713889 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.713869    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 19:26:25.713978 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.713960    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 19:26:25.714224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.714207    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sc9kv\""
Sep 11 19:26:25.714411 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.714391    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 19:26:25.714964 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.714811    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 19:26:25.714964 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.714873    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 19:26:25.715205 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.715168    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 19:26:25.715350 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.715331    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 19:26:25.715530 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.715509    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 19:26:25.715668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.715647    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mtlf7\""
Sep 11 19:26:25.715915 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.715884    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.718094 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.718075    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 19:26:25.718222 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.718199    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vfkst\""
Sep 11 19:26:25.718679 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.718662    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:25.718827 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:25.718731    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzscf" podUID="e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef"
Sep 11 19:26:25.721684 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.721583    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:25.721684 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:25.721663    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfnzl" podUID="58780500-db7c-40c3-b4ab-b484237c4ef2"
Sep 11 19:26:25.722865 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.722839    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9dd5f51f-878d-4022-8cec-df8ddf05eba9-ovn-node-metrics-cert\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.722985 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.722886    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b137eb7-4e92-44fe-882d-0ca19704ee2c-host\") pod \"node-ca-d5jh8\" (UID: \"0b137eb7-4e92-44fe-882d-0ca19704ee2c\") " pod="openshift-image-registry/node-ca-d5jh8"
Sep 11 19:26:25.722985 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.722914    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-run-netns\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.722985 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.722941    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-log-socket\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.722985 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.722968    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-system-cni-dir\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.723291 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.722995    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-kubelet\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.723291 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723021    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-slash\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.723291 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723049    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-var-lib-openvswitch\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.723291 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723077    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-etc-openvswitch\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.723291 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723117    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb1870fa-237b-4a9e-91b1-9741250e32d0-system-cni-dir\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.723291 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723149    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb1870fa-237b-4a9e-91b1-9741250e32d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.723291 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723176    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/952e8fe9-61fe-447f-8b98-62fa962b1d7a-cni-binary-copy\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.723291 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723204    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b137eb7-4e92-44fe-882d-0ca19704ee2c-serviceca\") pod \"node-ca-d5jh8\" (UID: \"0b137eb7-4e92-44fe-882d-0ca19704ee2c\") " pod="openshift-image-registry/node-ca-d5jh8"
Sep 11 19:26:25.723291 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723229    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb1870fa-237b-4a9e-91b1-9741250e32d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.723291 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723258    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-host-var-lib-cni-bin\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.723925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723321    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r65nc\" (UniqueName: \"kubernetes.io/projected/952e8fe9-61fe-447f-8b98-62fa962b1d7a-kube-api-access-r65nc\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.723925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723366    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-run-ovn\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.723925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723436    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9dd5f51f-878d-4022-8cec-df8ddf05eba9-env-overrides\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.723925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723471    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9dd5f51f-878d-4022-8cec-df8ddf05eba9-ovnkube-script-lib\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.723925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723499    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb1870fa-237b-4a9e-91b1-9741250e32d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.723925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723547    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvnw9\" (UniqueName: \"kubernetes.io/projected/9dd5f51f-878d-4022-8cec-df8ddf05eba9-kube-api-access-cvnw9\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.723925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723584    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-node-log\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.723925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723622    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-cni-bin\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.723925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723649    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-cni-netd\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.723925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723675    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.723925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723702    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw825\" (UniqueName: \"kubernetes.io/projected/cb1870fa-237b-4a9e-91b1-9741250e32d0-kube-api-access-jw825\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.723925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723727    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-multus-cni-dir\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.723925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723767    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-cnibin\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.723925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723820    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bswn\" (UniqueName: \"kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn\") pod \"network-check-target-mdl79\" (UID: \"3402721a-0f03-4681-bc7b-facbf92d31b2\") " pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:25.723925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723849    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-systemd-units\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.723925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723897    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-run-systemd\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.725093 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723935    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-run-openvswitch\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.725093 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723964    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb1870fa-237b-4a9e-91b1-9741250e32d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.725093 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.723994    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-multus-socket-dir-parent\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.725093 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.724022    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-host-var-lib-cni-multus\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.725093 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.724049    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-host-var-lib-kubelet\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.725093 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.724095    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm7ng\" (UniqueName: \"kubernetes.io/projected/0b137eb7-4e92-44fe-882d-0ca19704ee2c-kube-api-access-mm7ng\") pod \"node-ca-d5jh8\" (UID: \"0b137eb7-4e92-44fe-882d-0ca19704ee2c\") " pod="openshift-image-registry/node-ca-d5jh8"
Sep 11 19:26:25.725093 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.724134    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-run-ovn-kubernetes\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.725093 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.724165    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-host-run-k8s-cni-cncf-io\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.725093 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.724204    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-host-run-netns\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.725093 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.724240    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-hostroot\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.725093 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.724266    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-etc-kubernetes\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.725093 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.724292    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb1870fa-237b-4a9e-91b1-9741250e32d0-cnibin\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.725093 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.724336    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-multus-conf-dir\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.725093 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.724370    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/952e8fe9-61fe-447f-8b98-62fa962b1d7a-multus-daemon-config\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.725093 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.724403    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-host-run-multus-certs\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.725093 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.724427    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9dd5f51f-878d-4022-8cec-df8ddf05eba9-ovnkube-config\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.725093 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.724770    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7rglf"
Sep 11 19:26:25.726203 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.725770    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 19:26:25.727508 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.727346    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rm699\""
Sep 11 19:26:25.727508 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.727434    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 19:26:25.727703 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.727364    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 19:26:25.728242 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.727965    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z8xdl"
Sep 11 19:26:25.728242 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.728068    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 19:26:25.730373 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.730352    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5cl97\""
Sep 11 19:26:25.730476 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.730358    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 19:26:25.730543 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.730497    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 19:26:25.731521 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.731503    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l6ktp"
Sep 11 19:26:25.733730 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.733575    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 19:26:25.734261 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.734014    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 19:26:25.734261 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.734082    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9kmr9\""
Sep 11 19:26:25.735542 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.735282    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.737310 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.737292    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 19:26:25.737522 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.737504    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 19:26:25.737581 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.737371    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8q9lb\""
Sep 11 19:26:25.737648 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.737293    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 19:26:25.738486 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.738351    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.741281 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.741200    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 19:26:25.741281 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.741253    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7kkw4\""
Sep 11 19:26:25.741607 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.741582    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 19:26:25.745915 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.745885    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mbt2h"
Sep 11 19:26:25.754095 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.754071    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mbt2h"
Sep 11 19:26:25.812264 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.812185    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal" event={"ID":"f31104fc89b43315a443aee28694a8f6","Type":"ContainerStarted","Data":"533f2fb0f286f80c9b0323e00214591147dae4ba7d4b3587c73eb8165ec1b52b"}
Sep 11 19:26:25.813726 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.813664    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-43.ec2.internal" event={"ID":"af89a8d622e91b5bb6c931160e64948c","Type":"ContainerStarted","Data":"5b48f2c6d8964fa15dd3a4ff60f71f5ea39a137d754abf1b4f838a6b1530e224"}
Sep 11 19:26:25.814125 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.814105    2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 19:26:25.825187 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825158    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-hostroot\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.825303 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825199    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-etc-kubernetes\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.825303 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825255    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-socket-dir\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.825303 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825295    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-hostroot\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.825487 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825314    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-device-dir\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.825487 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825342    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmzz8\" (UniqueName: \"kubernetes.io/projected/865d5eb0-ea5c-4665-ba99-f5bbb171bcc9-kube-api-access-lmzz8\") pod \"iptables-alerter-7rglf\" (UID: \"865d5eb0-ea5c-4665-ba99-f5bbb171bcc9\") " pod="openshift-network-operator/iptables-alerter-7rglf"
Sep 11 19:26:25.825487 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825346    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-etc-kubernetes\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.825487 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825369    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d4ca207f-ccf3-4f01-bd0d-cd4cc9148997-konnectivity-ca\") pod \"konnectivity-agent-l6ktp\" (UID: \"d4ca207f-ccf3-4f01-bd0d-cd4cc9148997\") " pod="kube-system/konnectivity-agent-l6ktp"
Sep 11 19:26:25.825487 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825418    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/952e8fe9-61fe-447f-8b98-62fa962b1d7a-multus-daemon-config\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.825487 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825447    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7027ce0-9c8b-41f9-98c8-47905e740a83-hosts-file\") pod \"node-resolver-z8xdl\" (UID: \"f7027ce0-9c8b-41f9-98c8-47905e740a83\") " pod="openshift-dns/node-resolver-z8xdl"
Sep 11 19:26:25.825487 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825472    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-etc-sysctl-conf\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.825871 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825495    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-etc-systemd\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.825871 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825521    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9dd5f51f-878d-4022-8cec-df8ddf05eba9-ovn-node-metrics-cert\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.825871 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825545    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b137eb7-4e92-44fe-882d-0ca19704ee2c-host\") pod \"node-ca-d5jh8\" (UID: \"0b137eb7-4e92-44fe-882d-0ca19704ee2c\") " pod="openshift-image-registry/node-ca-d5jh8"
Sep 11 19:26:25.825871 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825585    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-run-netns\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.825871 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825611    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-system-cni-dir\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.825871 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825636    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/865d5eb0-ea5c-4665-ba99-f5bbb171bcc9-host-slash\") pod \"iptables-alerter-7rglf\" (UID: \"865d5eb0-ea5c-4665-ba99-f5bbb171bcc9\") " pod="openshift-network-operator/iptables-alerter-7rglf"
Sep 11 19:26:25.825871 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825660    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-host\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.825871 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825685    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-slash\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.825871 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825711    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-var-lib-openvswitch\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.825871 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825739    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-etc-openvswitch\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.825871 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825782    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb1870fa-237b-4a9e-91b1-9741250e32d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.825871 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825808    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/865d5eb0-ea5c-4665-ba99-f5bbb171bcc9-iptables-alerter-script\") pod \"iptables-alerter-7rglf\" (UID: \"865d5eb0-ea5c-4665-ba99-f5bbb171bcc9\") " pod="openshift-network-operator/iptables-alerter-7rglf"
Sep 11 19:26:25.825871 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825833    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/58780500-db7c-40c3-b4ab-b484237c4ef2-kubelet-config\") pod \"global-pull-secret-syncer-pfnzl\" (UID: \"58780500-db7c-40c3-b4ab-b484237c4ef2\") " pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:25.826533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825873    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-lib-modules\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.826533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825912    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9dd5f51f-878d-4022-8cec-df8ddf05eba9-env-overrides\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.826533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825937    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9dd5f51f-878d-4022-8cec-df8ddf05eba9-ovnkube-script-lib\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.826533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825962    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.826533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.825987    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/58780500-db7c-40c3-b4ab-b484237c4ef2-dbus\") pod \"global-pull-secret-syncer-pfnzl\" (UID: \"58780500-db7c-40c3-b4ab-b484237c4ef2\") " pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:25.826533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826012    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkdt4\" (UniqueName: \"kubernetes.io/projected/0946be7a-584b-455d-853e-d66a4c93fc80-kube-api-access-qkdt4\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.826533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826013    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/952e8fe9-61fe-447f-8b98-62fa962b1d7a-multus-daemon-config\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.826533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826037    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-node-log\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.826533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826074    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-cni-netd\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.826533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826080    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-var-lib-openvswitch\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.826533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826105    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw825\" (UniqueName: \"kubernetes.io/projected/cb1870fa-237b-4a9e-91b1-9741250e32d0-kube-api-access-jw825\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.826533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826148    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-multus-cni-dir\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.826533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826233    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-slash\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.826533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826259    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-etc-openvswitch\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.826533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826363    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb1870fa-237b-4a9e-91b1-9741250e32d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.826533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826399    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-cnibin\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.826533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826425    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-host-var-lib-cni-multus\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.827391 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826451    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-multus-conf-dir\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.827391 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826470    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slzkm\" (UniqueName: \"kubernetes.io/projected/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-kube-api-access-slzkm\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.827391 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826489    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bswn\" (UniqueName: \"kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn\") pod \"network-check-target-mdl79\" (UID: \"3402721a-0f03-4681-bc7b-facbf92d31b2\") " pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:25.827391 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826505    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-systemd-units\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.827391 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826503    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-multus-cni-dir\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.827391 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826520    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-multus-socket-dir-parent\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.827391 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826536    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-run\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.827391 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826552    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-var-lib-kubelet\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.827391 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826568    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0946be7a-584b-455d-853e-d66a4c93fc80-tmp\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.827391 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826583    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-run-ovn-kubernetes\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.827391 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826600    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-host-run-netns\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.827391 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826657    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7027ce0-9c8b-41f9-98c8-47905e740a83-tmp-dir\") pod \"node-resolver-z8xdl\" (UID: \"f7027ce0-9c8b-41f9-98c8-47905e740a83\") " pod="openshift-dns/node-resolver-z8xdl"
Sep 11 19:26:25.827391 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826672    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0946be7a-584b-455d-853e-d66a4c93fc80-etc-tuned\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.827391 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826692    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb1870fa-237b-4a9e-91b1-9741250e32d0-cnibin\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.827391 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826708    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-host-run-multus-certs\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.827391 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826763    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b137eb7-4e92-44fe-882d-0ca19704ee2c-host\") pod \"node-ca-d5jh8\" (UID: \"0b137eb7-4e92-44fe-882d-0ca19704ee2c\") " pod="openshift-image-registry/node-ca-d5jh8"
Sep 11 19:26:25.827391 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826786    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-run-ovn-kubernetes\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.828340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826857    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb1870fa-237b-4a9e-91b1-9741250e32d0-cnibin\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.828340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826860    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-run-netns\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.828340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826913    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-registration-dir\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.828340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826949    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-cni-netd\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.828340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826948    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf599\" (UniqueName: \"kubernetes.io/projected/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-kube-api-access-lf599\") pod \"network-metrics-daemon-zzscf\" (UID: \"e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef\") " pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:25.828340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826966    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9dd5f51f-878d-4022-8cec-df8ddf05eba9-env-overrides\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.828340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826973    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-system-cni-dir\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.828340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.826987    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-etc-modprobe-d\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.828340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827025    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-systemd-units\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.828340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827053    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9dd5f51f-878d-4022-8cec-df8ddf05eba9-ovnkube-config\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.828340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827064    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-host-var-lib-cni-multus\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.828340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827096    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-log-socket\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.828340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827099    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-multus-conf-dir\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.828340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827117    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-node-log\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.828340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827132    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-host-run-multus-certs\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.828340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827177    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-host-run-netns\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.828340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827198    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-multus-socket-dir-parent\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.828340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827173    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-log-socket\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.829250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827229    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-cnibin\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.829250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827231    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-etc-selinux\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.829250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827261    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l74p\" (UniqueName: \"kubernetes.io/projected/f7027ce0-9c8b-41f9-98c8-47905e740a83-kube-api-access-9l74p\") pod \"node-resolver-z8xdl\" (UID: \"f7027ce0-9c8b-41f9-98c8-47905e740a83\") " pod="openshift-dns/node-resolver-z8xdl"
Sep 11 19:26:25.829250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827295    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-etc-sysctl-d\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.829250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827323    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-sys\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.829250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827359    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-kubelet\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.829250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827387    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb1870fa-237b-4a9e-91b1-9741250e32d0-system-cni-dir\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.829250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827428    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-kubelet\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.829250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827463    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/952e8fe9-61fe-447f-8b98-62fa962b1d7a-cni-binary-copy\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.829250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827470    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb1870fa-237b-4a9e-91b1-9741250e32d0-system-cni-dir\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.829250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827515    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-host-var-lib-cni-bin\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.829250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827569    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r65nc\" (UniqueName: \"kubernetes.io/projected/952e8fe9-61fe-447f-8b98-62fa962b1d7a-kube-api-access-r65nc\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.829250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827599    2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 19:26:25.829250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827626    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b137eb7-4e92-44fe-882d-0ca19704ee2c-serviceca\") pod \"node-ca-d5jh8\" (UID: \"0b137eb7-4e92-44fe-882d-0ca19704ee2c\") " pod="openshift-image-registry/node-ca-d5jh8"
Sep 11 19:26:25.829250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827711    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb1870fa-237b-4a9e-91b1-9741250e32d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.829250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827751    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs\") pod \"network-metrics-daemon-zzscf\" (UID: \"e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef\") " pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:25.829250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827779    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-run-ovn\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.829250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827998    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9dd5f51f-878d-4022-8cec-df8ddf05eba9-ovnkube-script-lib\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.830224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827714    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9dd5f51f-878d-4022-8cec-df8ddf05eba9-ovnkube-config\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.830224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828026    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/952e8fe9-61fe-447f-8b98-62fa962b1d7a-cni-binary-copy\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.830224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828053    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb1870fa-237b-4a9e-91b1-9741250e32d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.830224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828090    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-run-ovn\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.830224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828111    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0b137eb7-4e92-44fe-882d-0ca19704ee2c-serviceca\") pod \"node-ca-d5jh8\" (UID: \"0b137eb7-4e92-44fe-882d-0ca19704ee2c\") " pod="openshift-image-registry/node-ca-d5jh8"
Sep 11 19:26:25.830224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828118    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret\") pod \"global-pull-secret-syncer-pfnzl\" (UID: \"58780500-db7c-40c3-b4ab-b484237c4ef2\") " pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:25.830224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828149    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-etc-kubernetes\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.830224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828175    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvnw9\" (UniqueName: \"kubernetes.io/projected/9dd5f51f-878d-4022-8cec-df8ddf05eba9-kube-api-access-cvnw9\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.830224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828191    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-cni-bin\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.830224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828214    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.830224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.827606    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-host-var-lib-cni-bin\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.830224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828242    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d4ca207f-ccf3-4f01-bd0d-cd4cc9148997-agent-certs\") pod \"konnectivity-agent-l6ktp\" (UID: \"d4ca207f-ccf3-4f01-bd0d-cd4cc9148997\") " pod="kube-system/konnectivity-agent-l6ktp"
Sep 11 19:26:25.830224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828272    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-etc-sysconfig\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.830224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828300    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-sys-fs\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.830224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828306    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.830224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828272    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-host-cni-bin\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.830224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828368    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-run-systemd\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.831108 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828382    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb1870fa-237b-4a9e-91b1-9741250e32d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.831108 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828329    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-run-systemd\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.831108 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828448    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-run-openvswitch\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.831108 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828478    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb1870fa-237b-4a9e-91b1-9741250e32d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.831108 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828500    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-host-var-lib-kubelet\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.831108 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828500    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9dd5f51f-878d-4022-8cec-df8ddf05eba9-run-openvswitch\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.831108 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828523    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm7ng\" (UniqueName: \"kubernetes.io/projected/0b137eb7-4e92-44fe-882d-0ca19704ee2c-kube-api-access-mm7ng\") pod \"node-ca-d5jh8\" (UID: \"0b137eb7-4e92-44fe-882d-0ca19704ee2c\") " pod="openshift-image-registry/node-ca-d5jh8"
Sep 11 19:26:25.831108 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828545    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-host-run-k8s-cni-cncf-io\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.831108 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828550    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-host-var-lib-kubelet\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.831108 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828649    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/952e8fe9-61fe-447f-8b98-62fa962b1d7a-host-run-k8s-cni-cncf-io\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.831108 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828685    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cb1870fa-237b-4a9e-91b1-9741250e32d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.831108 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.828896    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cb1870fa-237b-4a9e-91b1-9741250e32d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.832996 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.832851    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9dd5f51f-878d-4022-8cec-df8ddf05eba9-ovn-node-metrics-cert\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.834402 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:25.834379    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 19:26:25.834494 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:25.834411    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 19:26:25.834494 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:25.834426    2558 projected.go:196] Error preparing data for projected volume kube-api-access-9bswn for pod openshift-network-diagnostics/network-check-target-mdl79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:26:25.834494 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:25.834491    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn podName:3402721a-0f03-4681-bc7b-facbf92d31b2 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:26.334470746 +0000 UTC m=+2.951640237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9bswn" (UniqueName: "kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn") pod "network-check-target-mdl79" (UID: "3402721a-0f03-4681-bc7b-facbf92d31b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:26:25.837310 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.837287    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r65nc\" (UniqueName: \"kubernetes.io/projected/952e8fe9-61fe-447f-8b98-62fa962b1d7a-kube-api-access-r65nc\") pod \"multus-b4f5h\" (UID: \"952e8fe9-61fe-447f-8b98-62fa962b1d7a\") " pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:25.837440 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.837414    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm7ng\" (UniqueName: \"kubernetes.io/projected/0b137eb7-4e92-44fe-882d-0ca19704ee2c-kube-api-access-mm7ng\") pod \"node-ca-d5jh8\" (UID: \"0b137eb7-4e92-44fe-882d-0ca19704ee2c\") " pod="openshift-image-registry/node-ca-d5jh8"
Sep 11 19:26:25.837769 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.837747    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvnw9\" (UniqueName: \"kubernetes.io/projected/9dd5f51f-878d-4022-8cec-df8ddf05eba9-kube-api-access-cvnw9\") pod \"ovnkube-node-q9jc9\" (UID: \"9dd5f51f-878d-4022-8cec-df8ddf05eba9\") " pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:25.837902 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.837872    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw825\" (UniqueName: \"kubernetes.io/projected/cb1870fa-237b-4a9e-91b1-9741250e32d0-kube-api-access-jw825\") pod \"multus-additional-cni-plugins-whp4x\" (UID: \"cb1870fa-237b-4a9e-91b1-9741250e32d0\") " pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:25.929483 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929403    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-socket-dir\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.929483 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929454    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-device-dir\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.929483 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929484    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmzz8\" (UniqueName: \"kubernetes.io/projected/865d5eb0-ea5c-4665-ba99-f5bbb171bcc9-kube-api-access-lmzz8\") pod \"iptables-alerter-7rglf\" (UID: \"865d5eb0-ea5c-4665-ba99-f5bbb171bcc9\") " pod="openshift-network-operator/iptables-alerter-7rglf"
Sep 11 19:26:25.929775 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929513    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d4ca207f-ccf3-4f01-bd0d-cd4cc9148997-konnectivity-ca\") pod \"konnectivity-agent-l6ktp\" (UID: \"d4ca207f-ccf3-4f01-bd0d-cd4cc9148997\") " pod="kube-system/konnectivity-agent-l6ktp"
Sep 11 19:26:25.929775 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929547    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7027ce0-9c8b-41f9-98c8-47905e740a83-hosts-file\") pod \"node-resolver-z8xdl\" (UID: \"f7027ce0-9c8b-41f9-98c8-47905e740a83\") " pod="openshift-dns/node-resolver-z8xdl"
Sep 11 19:26:25.929775 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929553    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-device-dir\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.929775 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929574    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-etc-sysctl-conf\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.929775 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929602    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-etc-systemd\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.929775 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929633    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/865d5eb0-ea5c-4665-ba99-f5bbb171bcc9-host-slash\") pod \"iptables-alerter-7rglf\" (UID: \"865d5eb0-ea5c-4665-ba99-f5bbb171bcc9\") " pod="openshift-network-operator/iptables-alerter-7rglf"
Sep 11 19:26:25.929775 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929657    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-host\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.929775 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929690    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/865d5eb0-ea5c-4665-ba99-f5bbb171bcc9-iptables-alerter-script\") pod \"iptables-alerter-7rglf\" (UID: \"865d5eb0-ea5c-4665-ba99-f5bbb171bcc9\") " pod="openshift-network-operator/iptables-alerter-7rglf"
Sep 11 19:26:25.929775 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929693    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-socket-dir\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.929775 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929719    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/58780500-db7c-40c3-b4ab-b484237c4ef2-kubelet-config\") pod \"global-pull-secret-syncer-pfnzl\" (UID: \"58780500-db7c-40c3-b4ab-b484237c4ef2\") " pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:25.929775 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929770    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-lib-modules\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.930327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929804    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.930327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929821    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-etc-sysctl-conf\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.930327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929836    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/58780500-db7c-40c3-b4ab-b484237c4ef2-dbus\") pod \"global-pull-secret-syncer-pfnzl\" (UID: \"58780500-db7c-40c3-b4ab-b484237c4ef2\") " pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:25.930327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929883    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkdt4\" (UniqueName: \"kubernetes.io/projected/0946be7a-584b-455d-853e-d66a4c93fc80-kube-api-access-qkdt4\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.930327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929925    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slzkm\" (UniqueName: \"kubernetes.io/projected/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-kube-api-access-slzkm\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.930327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929967    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-run\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.930327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.929993    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/58780500-db7c-40c3-b4ab-b484237c4ef2-dbus\") pod \"global-pull-secret-syncer-pfnzl\" (UID: \"58780500-db7c-40c3-b4ab-b484237c4ef2\") " pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:25.930327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930003    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-var-lib-kubelet\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.930327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930040    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0946be7a-584b-455d-853e-d66a4c93fc80-tmp\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.930327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930063    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-var-lib-kubelet\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.930327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930089    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7027ce0-9c8b-41f9-98c8-47905e740a83-tmp-dir\") pod \"node-resolver-z8xdl\" (UID: \"f7027ce0-9c8b-41f9-98c8-47905e740a83\") " pod="openshift-dns/node-resolver-z8xdl"
Sep 11 19:26:25.930327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930118    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0946be7a-584b-455d-853e-d66a4c93fc80-etc-tuned\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.930327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930130    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d4ca207f-ccf3-4f01-bd0d-cd4cc9148997-konnectivity-ca\") pod \"konnectivity-agent-l6ktp\" (UID: \"d4ca207f-ccf3-4f01-bd0d-cd4cc9148997\") " pod="kube-system/konnectivity-agent-l6ktp"
Sep 11 19:26:25.930327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930147    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-registration-dir\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.930327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930178    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf599\" (UniqueName: \"kubernetes.io/projected/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-kube-api-access-lf599\") pod \"network-metrics-daemon-zzscf\" (UID: \"e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef\") " pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:25.930327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930193    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/58780500-db7c-40c3-b4ab-b484237c4ef2-kubelet-config\") pod \"global-pull-secret-syncer-pfnzl\" (UID: \"58780500-db7c-40c3-b4ab-b484237c4ef2\") " pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:25.930327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930206    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-etc-modprobe-d\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.931272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930238    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7027ce0-9c8b-41f9-98c8-47905e740a83-hosts-file\") pod \"node-resolver-z8xdl\" (UID: \"f7027ce0-9c8b-41f9-98c8-47905e740a83\") " pod="openshift-dns/node-resolver-z8xdl"
Sep 11 19:26:25.931272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930243    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-etc-systemd\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.931272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930250    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-etc-selinux\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.931272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930270    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/865d5eb0-ea5c-4665-ba99-f5bbb171bcc9-host-slash\") pod \"iptables-alerter-7rglf\" (UID: \"865d5eb0-ea5c-4665-ba99-f5bbb171bcc9\") " pod="openshift-network-operator/iptables-alerter-7rglf"
Sep 11 19:26:25.931272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930297    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l74p\" (UniqueName: \"kubernetes.io/projected/f7027ce0-9c8b-41f9-98c8-47905e740a83-kube-api-access-9l74p\") pod \"node-resolver-z8xdl\" (UID: \"f7027ce0-9c8b-41f9-98c8-47905e740a83\") " pod="openshift-dns/node-resolver-z8xdl"
Sep 11 19:26:25.931272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930320    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-host\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.931272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930324    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-etc-sysctl-d\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.931272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930350    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-etc-selinux\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.931272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930355    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-run\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.931272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930351    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-sys\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.931272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930391    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-sys\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.931272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930416    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs\") pod \"network-metrics-daemon-zzscf\" (UID: \"e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef\") " pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:25.931272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930447    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret\") pod \"global-pull-secret-syncer-pfnzl\" (UID: \"58780500-db7c-40c3-b4ab-b484237c4ef2\") " pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:25.931272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930463    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-lib-modules\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.931272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930473    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-etc-kubernetes\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.931272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930504    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d4ca207f-ccf3-4f01-bd0d-cd4cc9148997-agent-certs\") pod \"konnectivity-agent-l6ktp\" (UID: \"d4ca207f-ccf3-4f01-bd0d-cd4cc9148997\") " pod="kube-system/konnectivity-agent-l6ktp"
Sep 11 19:26:25.931272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930531    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-etc-sysconfig\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.931272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930564    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-sys-fs\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.932297 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930622    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-etc-sysctl-d\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.932297 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930659    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-sys-fs\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.932297 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930683    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-etc-kubernetes\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.932297 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930711    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.932297 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930729    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/865d5eb0-ea5c-4665-ba99-f5bbb171bcc9-iptables-alerter-script\") pod \"iptables-alerter-7rglf\" (UID: \"865d5eb0-ea5c-4665-ba99-f5bbb171bcc9\") " pod="openshift-network-operator/iptables-alerter-7rglf"
Sep 11 19:26:25.932297 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:25.930738    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:26:25.932297 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930799    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-etc-sysconfig\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.932297 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:25.930808    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs podName:e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef nodeName:}" failed. No retries permitted until 2026-09-11 19:26:26.430793927 +0000 UTC m=+3.047963398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs") pod "network-metrics-daemon-zzscf" (UID: "e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:26:25.932297 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.930845    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-registration-dir\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:25.932297 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:25.931011    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 19:26:25.932297 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.931052    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7027ce0-9c8b-41f9-98c8-47905e740a83-tmp-dir\") pod \"node-resolver-z8xdl\" (UID: \"f7027ce0-9c8b-41f9-98c8-47905e740a83\") " pod="openshift-dns/node-resolver-z8xdl"
Sep 11 19:26:25.932297 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:25.931062    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret podName:58780500-db7c-40c3-b4ab-b484237c4ef2 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:26.431044722 +0000 UTC m=+3.048214215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret") pod "global-pull-secret-syncer-pfnzl" (UID: "58780500-db7c-40c3-b4ab-b484237c4ef2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 19:26:25.932297 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.931170    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0946be7a-584b-455d-853e-d66a4c93fc80-etc-modprobe-d\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.933561 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.933525    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0946be7a-584b-455d-853e-d66a4c93fc80-tmp\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.933991 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.933965    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d4ca207f-ccf3-4f01-bd0d-cd4cc9148997-agent-certs\") pod \"konnectivity-agent-l6ktp\" (UID: \"d4ca207f-ccf3-4f01-bd0d-cd4cc9148997\") " pod="kube-system/konnectivity-agent-l6ktp"
Sep 11 19:26:25.934890 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.934865    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0946be7a-584b-455d-853e-d66a4c93fc80-etc-tuned\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.940070 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.940042    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf599\" (UniqueName: \"kubernetes.io/projected/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-kube-api-access-lf599\") pod \"network-metrics-daemon-zzscf\" (UID: \"e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef\") " pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:25.940524 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.940500    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkdt4\" (UniqueName: \"kubernetes.io/projected/0946be7a-584b-455d-853e-d66a4c93fc80-kube-api-access-qkdt4\") pod \"tuned-2wkv2\" (UID: \"0946be7a-584b-455d-853e-d66a4c93fc80\") " pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:25.940604 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.940591    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmzz8\" (UniqueName: \"kubernetes.io/projected/865d5eb0-ea5c-4665-ba99-f5bbb171bcc9-kube-api-access-lmzz8\") pod \"iptables-alerter-7rglf\" (UID: \"865d5eb0-ea5c-4665-ba99-f5bbb171bcc9\") " pod="openshift-network-operator/iptables-alerter-7rglf"
Sep 11 19:26:25.941040 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.941019    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l74p\" (UniqueName: \"kubernetes.io/projected/f7027ce0-9c8b-41f9-98c8-47905e740a83-kube-api-access-9l74p\") pod \"node-resolver-z8xdl\" (UID: \"f7027ce0-9c8b-41f9-98c8-47905e740a83\") " pod="openshift-dns/node-resolver-z8xdl"
Sep 11 19:26:25.941394 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:25.941366    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slzkm\" (UniqueName: \"kubernetes.io/projected/856ee7ca-bf2a-41bf-8577-13e6c1d2c33c-kube-api-access-slzkm\") pod \"aws-ebs-csi-driver-node-pvfmp\" (UID: \"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:26.022085 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:26.021999    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d5jh8"
Sep 11 19:26:26.031225 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:26.031200    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:26.044069 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:26.044044    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-whp4x"
Sep 11 19:26:26.050184 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:26.050160    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b4f5h"
Sep 11 19:26:26.060076 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:26.060058    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7rglf"
Sep 11 19:26:26.067783 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:26.067766    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z8xdl"
Sep 11 19:26:26.076740 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:26.076707    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l6ktp"
Sep 11 19:26:26.084714 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:26.084693    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp"
Sep 11 19:26:26.091662 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:26.091644    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2wkv2"
Sep 11 19:26:26.433808 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:26.433772    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bswn\" (UniqueName: \"kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn\") pod \"network-check-target-mdl79\" (UID: \"3402721a-0f03-4681-bc7b-facbf92d31b2\") " pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:26.434011 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:26.433829    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs\") pod \"network-metrics-daemon-zzscf\" (UID: \"e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef\") " pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:26.434011 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:26.433851    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret\") pod \"global-pull-secret-syncer-pfnzl\" (UID: \"58780500-db7c-40c3-b4ab-b484237c4ef2\") " pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:26.434011 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:26.433943    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 19:26:26.434011 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:26.433960    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 19:26:26.434011 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:26.433969    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 19:26:26.434011 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:26.433982    2558 projected.go:196] Error preparing data for projected volume kube-api-access-9bswn for pod openshift-network-diagnostics/network-check-target-mdl79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:26:26.434011 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:26.434001    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:26:26.434364 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:26.434020    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret podName:58780500-db7c-40c3-b4ab-b484237c4ef2 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:27.434001929 +0000 UTC m=+4.051171404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret") pod "global-pull-secret-syncer-pfnzl" (UID: "58780500-db7c-40c3-b4ab-b484237c4ef2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 19:26:26.434364 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:26.434042    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn podName:3402721a-0f03-4681-bc7b-facbf92d31b2 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:27.434031766 +0000 UTC m=+4.051201238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bswn" (UniqueName: "kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn") pod "network-check-target-mdl79" (UID: "3402721a-0f03-4681-bc7b-facbf92d31b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:26:26.434364 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:26.434063    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs podName:e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef nodeName:}" failed. No retries permitted until 2026-09-11 19:26:27.434053887 +0000 UTC m=+4.051223389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs") pod "network-metrics-daemon-zzscf" (UID: "e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:26:26.757856 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:26.757762    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 19:21:26 +0000 UTC" deadline="2028-03-14 15:26:40.456416506 +0000 UTC"
Sep 11 19:26:26.757856 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:26.757797    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13196h0m13.698624986s"
Sep 11 19:26:27.043395 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:27.043356    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7027ce0_9c8b_41f9_98c8_47905e740a83.slice/crio-83fe7ea4ca61c1a9a75f616c26df0caaa2a1928c3d854177779ce334376d837e WatchSource:0}: Error finding container 83fe7ea4ca61c1a9a75f616c26df0caaa2a1928c3d854177779ce334376d837e: Status 404 returned error can't find the container with id 83fe7ea4ca61c1a9a75f616c26df0caaa2a1928c3d854177779ce334376d837e
Sep 11 19:26:27.045308 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:27.045281    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0946be7a_584b_455d_853e_d66a4c93fc80.slice/crio-d5bbc4186faf40b4448581ecb3c4b762751845765a33961649582027e8d01f17 WatchSource:0}: Error finding container d5bbc4186faf40b4448581ecb3c4b762751845765a33961649582027e8d01f17: Status 404 returned error can't find the container with id d5bbc4186faf40b4448581ecb3c4b762751845765a33961649582027e8d01f17
Sep 11 19:26:27.048332 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:27.047662    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ca207f_ccf3_4f01_bd0d_cd4cc9148997.slice/crio-f38570fc0df47f069133b869348b17215208f268dc90756f3bab3a8b020d9634 WatchSource:0}: Error finding container f38570fc0df47f069133b869348b17215208f268dc90756f3bab3a8b020d9634: Status 404 returned error can't find the container with id f38570fc0df47f069133b869348b17215208f268dc90756f3bab3a8b020d9634
Sep 11 19:26:27.051446 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:27.051419    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1870fa_237b_4a9e_91b1_9741250e32d0.slice/crio-88449e54d7fd76c2cb4d796fdfd8bde60f4d7d92a4385dfbd2048e40dc22877b WatchSource:0}: Error finding container 88449e54d7fd76c2cb4d796fdfd8bde60f4d7d92a4385dfbd2048e40dc22877b: Status 404 returned error can't find the container with id 88449e54d7fd76c2cb4d796fdfd8bde60f4d7d92a4385dfbd2048e40dc22877b
Sep 11 19:26:27.052132 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:27.051910    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952e8fe9_61fe_447f_8b98_62fa962b1d7a.slice/crio-f7d56b3affca0a84528ad43ac572633084ea4cde74283823b036a23bb5cb63f9 WatchSource:0}: Error finding container f7d56b3affca0a84528ad43ac572633084ea4cde74283823b036a23bb5cb63f9: Status 404 returned error can't find the container with id f7d56b3affca0a84528ad43ac572633084ea4cde74283823b036a23bb5cb63f9
Sep 11 19:26:27.052912 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:27.052847    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b137eb7_4e92_44fe_882d_0ca19704ee2c.slice/crio-b79c03e27596681d8ca5b74e26dcf124efa4a3e6982de5d5f76900f7aaa72381 WatchSource:0}: Error finding container b79c03e27596681d8ca5b74e26dcf124efa4a3e6982de5d5f76900f7aaa72381: Status 404 returned error can't find the container with id b79c03e27596681d8ca5b74e26dcf124efa4a3e6982de5d5f76900f7aaa72381
Sep 11 19:26:27.055512 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:27.055394    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865d5eb0_ea5c_4665_ba99_f5bbb171bcc9.slice/crio-faf94e6f58da033af01f53abc0431a69c61bc66f6cba80f1d434435e1c7a1c60 WatchSource:0}: Error finding container faf94e6f58da033af01f53abc0431a69c61bc66f6cba80f1d434435e1c7a1c60: Status 404 returned error can't find the container with id faf94e6f58da033af01f53abc0431a69c61bc66f6cba80f1d434435e1c7a1c60
Sep 11 19:26:27.056204 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:27.056126    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856ee7ca_bf2a_41bf_8577_13e6c1d2c33c.slice/crio-6b0a5b351031fc09f7b548a133c48b1fc5064a8570239e7bc24b3780a64812b8 WatchSource:0}: Error finding container 6b0a5b351031fc09f7b548a133c48b1fc5064a8570239e7bc24b3780a64812b8: Status 404 returned error can't find the container with id 6b0a5b351031fc09f7b548a133c48b1fc5064a8570239e7bc24b3780a64812b8
Sep 11 19:26:27.441641 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.441610    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bswn\" (UniqueName: \"kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn\") pod \"network-check-target-mdl79\" (UID: \"3402721a-0f03-4681-bc7b-facbf92d31b2\") " pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:27.441779 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.441657    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs\") pod \"network-metrics-daemon-zzscf\" (UID: \"e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef\") " pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:27.441779 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.441679    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret\") pod \"global-pull-secret-syncer-pfnzl\" (UID: \"58780500-db7c-40c3-b4ab-b484237c4ef2\") " pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:27.441874 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:27.441804    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 19:26:27.441874 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:27.441836    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 19:26:27.441874 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:27.441847    2558 projected.go:196] Error preparing data for projected volume kube-api-access-9bswn for pod openshift-network-diagnostics/network-check-target-mdl79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:26:27.441874 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:27.441861    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 19:26:27.442009 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:27.441806    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:26:27.442009 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:27.441899    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn podName:3402721a-0f03-4681-bc7b-facbf92d31b2 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:29.441880213 +0000 UTC m=+6.059049704 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bswn" (UniqueName: "kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn") pod "network-check-target-mdl79" (UID: "3402721a-0f03-4681-bc7b-facbf92d31b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:26:27.442009 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:27.441920    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs podName:e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef nodeName:}" failed. No retries permitted until 2026-09-11 19:26:29.441908227 +0000 UTC m=+6.059077698 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs") pod "network-metrics-daemon-zzscf" (UID: "e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:26:27.442009 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:27.441932    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret podName:58780500-db7c-40c3-b4ab-b484237c4ef2 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:29.441926175 +0000 UTC m=+6.059095646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret") pod "global-pull-secret-syncer-pfnzl" (UID: "58780500-db7c-40c3-b4ab-b484237c4ef2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 19:26:27.759174 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.758970    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 19:21:26 +0000 UTC" deadline="2028-04-06 19:16:42.075511165 +0000 UTC"
Sep 11 19:26:27.759174 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.759008    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13751h50m14.316508666s"
Sep 11 19:26:27.810395 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.810364    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:27.810552 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:27.810515    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzscf" podUID="e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef"
Sep 11 19:26:27.811012 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.810988    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:27.811129 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:27.811102    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfnzl" podUID="58780500-db7c-40c3-b4ab-b484237c4ef2"
Sep 11 19:26:27.812048 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.812024    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:27.812415 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:27.812387    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdl79" podUID="3402721a-0f03-4681-bc7b-facbf92d31b2"
Sep 11 19:26:27.855176 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.855142    2558 generic.go:356] "Generic (PLEG): container finished" podID="f31104fc89b43315a443aee28694a8f6" containerID="703cb7bb83fba98c2d8b360b8877258b87d19ea22f50c4f3bbd6471c97a66e5d" exitCode=0
Sep 11 19:26:27.855321 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.855247    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal" event={"ID":"f31104fc89b43315a443aee28694a8f6","Type":"ContainerDied","Data":"703cb7bb83fba98c2d8b360b8877258b87d19ea22f50c4f3bbd6471c97a66e5d"}
Sep 11 19:26:27.858861 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.858738    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9" event={"ID":"9dd5f51f-878d-4022-8cec-df8ddf05eba9","Type":"ContainerStarted","Data":"91c8a221d5f27dbe13257cb7bb81d79d64e0c564b2d567df4a9a380eebe7ff4a"}
Sep 11 19:26:27.867479 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.867450    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7rglf" event={"ID":"865d5eb0-ea5c-4665-ba99-f5bbb171bcc9","Type":"ContainerStarted","Data":"faf94e6f58da033af01f53abc0431a69c61bc66f6cba80f1d434435e1c7a1c60"}
Sep 11 19:26:27.881308 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.881240    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l6ktp" event={"ID":"d4ca207f-ccf3-4f01-bd0d-cd4cc9148997","Type":"ContainerStarted","Data":"f38570fc0df47f069133b869348b17215208f268dc90756f3bab3a8b020d9634"}
Sep 11 19:26:27.888345 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.888295    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2wkv2" event={"ID":"0946be7a-584b-455d-853e-d66a4c93fc80","Type":"ContainerStarted","Data":"d5bbc4186faf40b4448581ecb3c4b762751845765a33961649582027e8d01f17"}
Sep 11 19:26:27.899625 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.899492    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z8xdl" event={"ID":"f7027ce0-9c8b-41f9-98c8-47905e740a83","Type":"ContainerStarted","Data":"83fe7ea4ca61c1a9a75f616c26df0caaa2a1928c3d854177779ce334376d837e"}
Sep 11 19:26:27.905895 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.905836    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-43.ec2.internal" event={"ID":"af89a8d622e91b5bb6c931160e64948c","Type":"ContainerStarted","Data":"d6b295c89a266ad64c3b735b2342fecf65069b2aac6cfb7704edd768d8cc64f8"}
Sep 11 19:26:27.920937 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.920700    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-43.ec2.internal" podStartSLOduration=2.920682282 podStartE2EDuration="2.920682282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:27.920628963 +0000 UTC m=+4.537798565" watchObservedRunningTime="2026-09-11 19:26:27.920682282 +0000 UTC m=+4.537851783"
Sep 11 19:26:27.924202 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.924125    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp" event={"ID":"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c","Type":"ContainerStarted","Data":"6b0a5b351031fc09f7b548a133c48b1fc5064a8570239e7bc24b3780a64812b8"}
Sep 11 19:26:27.935674 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.935609    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d5jh8" event={"ID":"0b137eb7-4e92-44fe-882d-0ca19704ee2c","Type":"ContainerStarted","Data":"b79c03e27596681d8ca5b74e26dcf124efa4a3e6982de5d5f76900f7aaa72381"}
Sep 11 19:26:27.944650 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.944614    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b4f5h" event={"ID":"952e8fe9-61fe-447f-8b98-62fa962b1d7a","Type":"ContainerStarted","Data":"f7d56b3affca0a84528ad43ac572633084ea4cde74283823b036a23bb5cb63f9"}
Sep 11 19:26:27.964109 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:27.964048    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whp4x" event={"ID":"cb1870fa-237b-4a9e-91b1-9741250e32d0","Type":"ContainerStarted","Data":"88449e54d7fd76c2cb4d796fdfd8bde60f4d7d92a4385dfbd2048e40dc22877b"}
Sep 11 19:26:28.995603 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:28.995569    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal" event={"ID":"f31104fc89b43315a443aee28694a8f6","Type":"ContainerStarted","Data":"13ca6eb81db4963072f7ddb82127156fd093ce539666e7a44ba8de1f11248eb0"}
Sep 11 19:26:29.458913 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:29.458618    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs\") pod \"network-metrics-daemon-zzscf\" (UID: \"e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef\") " pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:29.458913 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:29.458912    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:26:29.459186 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:29.458940    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret\") pod \"global-pull-secret-syncer-pfnzl\" (UID: \"58780500-db7c-40c3-b4ab-b484237c4ef2\") " pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:29.459186 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:29.458991    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs podName:e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef nodeName:}" failed. No retries permitted until 2026-09-11 19:26:33.458969702 +0000 UTC m=+10.076139192 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs") pod "network-metrics-daemon-zzscf" (UID: "e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:26:29.459186 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:29.459056    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 19:26:29.459186 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:29.459057    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bswn\" (UniqueName: \"kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn\") pod \"network-check-target-mdl79\" (UID: \"3402721a-0f03-4681-bc7b-facbf92d31b2\") " pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:29.459186 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:29.459111    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret podName:58780500-db7c-40c3-b4ab-b484237c4ef2 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:33.4590938 +0000 UTC m=+10.076263299 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret") pod "global-pull-secret-syncer-pfnzl" (UID: "58780500-db7c-40c3-b4ab-b484237c4ef2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 19:26:29.459186 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:29.459168    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 19:26:29.459186 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:29.459190    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 19:26:29.459574 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:29.459236    2558 projected.go:196] Error preparing data for projected volume kube-api-access-9bswn for pod openshift-network-diagnostics/network-check-target-mdl79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:26:29.459574 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:29.459283    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn podName:3402721a-0f03-4681-bc7b-facbf92d31b2 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:33.459268975 +0000 UTC m=+10.076438449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bswn" (UniqueName: "kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn") pod "network-check-target-mdl79" (UID: "3402721a-0f03-4681-bc7b-facbf92d31b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:26:29.810832 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:29.810732    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:29.811009 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:29.810907    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzscf" podUID="e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef"
Sep 11 19:26:29.811350 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:29.811320    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:29.811475 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:29.811427    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfnzl" podUID="58780500-db7c-40c3-b4ab-b484237c4ef2"
Sep 11 19:26:29.811543 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:29.811510    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:29.811595 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:29.811562    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdl79" podUID="3402721a-0f03-4681-bc7b-facbf92d31b2"
Sep 11 19:26:31.807680 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:31.807640    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:31.808173 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:31.807821    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzscf" podUID="e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef"
Sep 11 19:26:31.808173 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:31.807904    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:31.808173 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:31.808015    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfnzl" podUID="58780500-db7c-40c3-b4ab-b484237c4ef2"
Sep 11 19:26:31.808173 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:31.808070    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:31.808173 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:31.808138    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdl79" podUID="3402721a-0f03-4681-bc7b-facbf92d31b2"
Sep 11 19:26:33.492013 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:33.491975    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs\") pod \"network-metrics-daemon-zzscf\" (UID: \"e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef\") " pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:33.492463 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:33.492028    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret\") pod \"global-pull-secret-syncer-pfnzl\" (UID: \"58780500-db7c-40c3-b4ab-b484237c4ef2\") " pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:33.492463 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:33.492168    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 19:26:33.492463 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:33.492175    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:26:33.492463 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:33.492206    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bswn\" (UniqueName: \"kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn\") pod \"network-check-target-mdl79\" (UID: \"3402721a-0f03-4681-bc7b-facbf92d31b2\") " pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:33.492463 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:33.492232    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret podName:58780500-db7c-40c3-b4ab-b484237c4ef2 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:41.492213888 +0000 UTC m=+18.109383365 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret") pod "global-pull-secret-syncer-pfnzl" (UID: "58780500-db7c-40c3-b4ab-b484237c4ef2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 19:26:33.492463 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:33.492291    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 19:26:33.492463 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:33.492301    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs podName:e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef nodeName:}" failed. No retries permitted until 2026-09-11 19:26:41.492271882 +0000 UTC m=+18.109441367 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs") pod "network-metrics-daemon-zzscf" (UID: "e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:26:33.492463 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:33.492305    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 19:26:33.492463 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:33.492319    2558 projected.go:196] Error preparing data for projected volume kube-api-access-9bswn for pod openshift-network-diagnostics/network-check-target-mdl79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:26:33.492463 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:33.492358    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn podName:3402721a-0f03-4681-bc7b-facbf92d31b2 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:41.49234556 +0000 UTC m=+18.109515035 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bswn" (UniqueName: "kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn") pod "network-check-target-mdl79" (UID: "3402721a-0f03-4681-bc7b-facbf92d31b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:26:33.808404 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:33.808244    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:33.808404 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:33.808245    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:33.809666 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:33.809407    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzscf" podUID="e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef"
Sep 11 19:26:33.809666 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:33.809469    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdl79" podUID="3402721a-0f03-4681-bc7b-facbf92d31b2"
Sep 11 19:26:33.809666 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:33.809524    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:33.809666 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:33.809599    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfnzl" podUID="58780500-db7c-40c3-b4ab-b484237c4ef2"
Sep 11 19:26:35.808355 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:35.808320    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:35.808770 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:35.808465    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzscf" podUID="e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef"
Sep 11 19:26:35.808877 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:35.808856    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:35.808980 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:35.808960    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfnzl" podUID="58780500-db7c-40c3-b4ab-b484237c4ef2"
Sep 11 19:26:35.809112 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:35.809089    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:35.810701 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:35.810670    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdl79" podUID="3402721a-0f03-4681-bc7b-facbf92d31b2"
Sep 11 19:26:37.808220 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:37.808172    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:37.808220 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:37.808199    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:37.808751 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:37.808322    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdl79" podUID="3402721a-0f03-4681-bc7b-facbf92d31b2"
Sep 11 19:26:37.808751 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:37.808379    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:37.808751 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:37.808491    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzscf" podUID="e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef"
Sep 11 19:26:37.808751 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:37.808549    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfnzl" podUID="58780500-db7c-40c3-b4ab-b484237c4ef2"
Sep 11 19:26:39.807735 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:39.807688    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:39.808171 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:39.807699    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:39.808171 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:39.807801    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdl79" podUID="3402721a-0f03-4681-bc7b-facbf92d31b2"
Sep 11 19:26:39.808171 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:39.807821    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:39.808171 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:39.807909    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzscf" podUID="e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef"
Sep 11 19:26:39.808171 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:39.807980    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfnzl" podUID="58780500-db7c-40c3-b4ab-b484237c4ef2"
Sep 11 19:26:41.556324 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:41.556282    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bswn\" (UniqueName: \"kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn\") pod \"network-check-target-mdl79\" (UID: \"3402721a-0f03-4681-bc7b-facbf92d31b2\") " pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:41.556827 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:41.556347    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs\") pod \"network-metrics-daemon-zzscf\" (UID: \"e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef\") " pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:41.556827 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:41.556442    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:26:41.556827 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:41.556463    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 19:26:41.556827 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:41.556487    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 19:26:41.556827 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:41.556501    2558 projected.go:196] Error preparing data for projected volume kube-api-access-9bswn for pod openshift-network-diagnostics/network-check-target-mdl79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:26:41.556827 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:41.556506    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs podName:e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef nodeName:}" failed. No retries permitted until 2026-09-11 19:26:57.556486866 +0000 UTC m=+34.173656362 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs") pod "network-metrics-daemon-zzscf" (UID: "e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:26:41.556827 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:41.556540    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret\") pod \"global-pull-secret-syncer-pfnzl\" (UID: \"58780500-db7c-40c3-b4ab-b484237c4ef2\") " pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:41.556827 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:41.556551    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn podName:3402721a-0f03-4681-bc7b-facbf92d31b2 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:57.55653477 +0000 UTC m=+34.173704253 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bswn" (UniqueName: "kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn") pod "network-check-target-mdl79" (UID: "3402721a-0f03-4681-bc7b-facbf92d31b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:26:41.556827 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:41.556666    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 19:26:41.556827 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:41.556733    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret podName:58780500-db7c-40c3-b4ab-b484237c4ef2 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:57.556718853 +0000 UTC m=+34.173888337 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret") pod "global-pull-secret-syncer-pfnzl" (UID: "58780500-db7c-40c3-b4ab-b484237c4ef2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 19:26:41.808452 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:41.808415    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:41.808452 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:41.808434    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:41.808696 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:41.808415    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:41.808696 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:41.808549    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfnzl" podUID="58780500-db7c-40c3-b4ab-b484237c4ef2"
Sep 11 19:26:41.808805 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:41.808721    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzscf" podUID="e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef"
Sep 11 19:26:41.808866 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:41.808836    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdl79" podUID="3402721a-0f03-4681-bc7b-facbf92d31b2"
Sep 11 19:26:43.808911 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:43.808877    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:43.809344 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:43.808971    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzscf" podUID="e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef"
Sep 11 19:26:43.809344 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:43.809062    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:43.809523 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:43.809493    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdl79" podUID="3402721a-0f03-4681-bc7b-facbf92d31b2"
Sep 11 19:26:43.812226 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:43.809233    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:43.812226 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:43.809871    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfnzl" podUID="58780500-db7c-40c3-b4ab-b484237c4ef2"
Sep 11 19:26:45.027813 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:45.027115    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l6ktp" event={"ID":"d4ca207f-ccf3-4f01-bd0d-cd4cc9148997","Type":"ContainerStarted","Data":"11f3da91fbdda9bb5680c8df48ba969e4bbc68c5206df33c513814a838ba3fa2"}
Sep 11 19:26:45.030221 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:45.030126    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp" event={"ID":"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c","Type":"ContainerStarted","Data":"608d5922230d6fac6e6abb568699c4fbf81a2a4e657b73f4c97fbb0d7f68892a"}
Sep 11 19:26:45.033573 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:45.032907    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whp4x" event={"ID":"cb1870fa-237b-4a9e-91b1-9741250e32d0","Type":"ContainerStarted","Data":"bf8724328f905208d100baa44ca0f46b741f4e7e85686b6d15b855f7d3d8d636"}
Sep 11 19:26:45.043111 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:45.043066    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l6ktp" podStartSLOduration=6.896220185 podStartE2EDuration="21.043049301s" totalImagesPullingTime="14.146829116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:45.042442776 +0000 UTC m=+21.659612266" watchObservedRunningTime="2026-09-11 19:26:45.043049301 +0000 UTC m=+21.660218794"
Sep 11 19:26:45.043686 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:45.043472    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-43.ec2.internal" podStartSLOduration=20.043463853 podStartE2EDuration="20.043463853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:29.011712237 +0000 UTC m=+5.628881735" watchObservedRunningTime="2026-09-11 19:26:45.043463853 +0000 UTC m=+21.660633347"
Sep 11 19:26:45.571195 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:45.570985    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l6ktp_d4ca207f-ccf3-4f01-bd0d-cd4cc9148997/konnectivity-agent/0.log" limit=1024
Sep 11 19:26:45.690095 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:45.690069    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l6ktp_d4ca207f-ccf3-4f01-bd0d-cd4cc9148997/konnectivity-agent/0.log" limit=1024
Sep 11 19:26:45.791057 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:45.791026    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l6ktp_d4ca207f-ccf3-4f01-bd0d-cd4cc9148997/konnectivity-agent/0.log" limit=1024
Sep 11 19:26:45.807959 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:45.807931    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:45.808053 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:45.808019    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfnzl" podUID="58780500-db7c-40c3-b4ab-b484237c4ef2"
Sep 11 19:26:45.808111 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:45.807931    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:45.808111 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:45.808074    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:45.808223 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:45.808146    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzscf" podUID="e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef"
Sep 11 19:26:45.808223 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:45.808195    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdl79" podUID="3402721a-0f03-4681-bc7b-facbf92d31b2"
Sep 11 19:26:46.037658 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.037625    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9" event={"ID":"9dd5f51f-878d-4022-8cec-df8ddf05eba9","Type":"ContainerStarted","Data":"65cb236d8b78940de1d102def983fcc27e29ca091f1baa3579aba4e1fd1d2603"}
Sep 11 19:26:46.038361 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.037700    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9" event={"ID":"9dd5f51f-878d-4022-8cec-df8ddf05eba9","Type":"ContainerStarted","Data":"f0cd7a18438e9c397eda7f5b2d29969a2605daa4c9995f898e64d870d19b6b49"}
Sep 11 19:26:46.038361 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.037753    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9" event={"ID":"9dd5f51f-878d-4022-8cec-df8ddf05eba9","Type":"ContainerStarted","Data":"5f483a8ecb036fb8aa68cfe953ac98b22f4316115aa8d835f9f9623ab4c05c6c"}
Sep 11 19:26:46.038361 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.037782    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9" event={"ID":"9dd5f51f-878d-4022-8cec-df8ddf05eba9","Type":"ContainerStarted","Data":"f61da84d93472248d90b214b063cfcc2f00496ac5e5ff510fd85267afe34888f"}
Sep 11 19:26:46.038361 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.037822    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9" event={"ID":"9dd5f51f-878d-4022-8cec-df8ddf05eba9","Type":"ContainerStarted","Data":"d97d0d8f3df67cd9539f08270142ddcac030e507890fdf93984a415e5da39ece"}
Sep 11 19:26:46.038361 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.037851    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9" event={"ID":"9dd5f51f-878d-4022-8cec-df8ddf05eba9","Type":"ContainerStarted","Data":"c85a74f92cdf87565dc909b771e799c82ffc7298bd75b8c4ac671c864e22860d"}
Sep 11 19:26:46.039068 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.039039    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2wkv2" event={"ID":"0946be7a-584b-455d-853e-d66a4c93fc80","Type":"ContainerStarted","Data":"6aaa8aec4eba641568d2eb6ae8f3af7cc4c8052fc3ba9253fa78017c2ed68b3a"}
Sep 11 19:26:46.040315 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.040291    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z8xdl" event={"ID":"f7027ce0-9c8b-41f9-98c8-47905e740a83","Type":"ContainerStarted","Data":"a47ca322ae0816a2342db47479053bf293873f38f68b0b13c335617501f10223"}
Sep 11 19:26:46.041671 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.041635    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d5jh8" event={"ID":"0b137eb7-4e92-44fe-882d-0ca19704ee2c","Type":"ContainerStarted","Data":"c3058ae567377810d006f1ece6ef46b91a39af75400b9621e8a56f73902d47e3"}
Sep 11 19:26:46.043069 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.043045    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b4f5h" event={"ID":"952e8fe9-61fe-447f-8b98-62fa962b1d7a","Type":"ContainerStarted","Data":"8686267ac528513d3909a4e0ef2932cd3ed7ceddf40272c56ca3cec63410db83"}
Sep 11 19:26:46.044435 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.044414    2558 generic.go:356] "Generic (PLEG): container finished" podID="cb1870fa-237b-4a9e-91b1-9741250e32d0" containerID="bf8724328f905208d100baa44ca0f46b741f4e7e85686b6d15b855f7d3d8d636" exitCode=0
Sep 11 19:26:46.044539 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.044477    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whp4x" event={"ID":"cb1870fa-237b-4a9e-91b1-9741250e32d0","Type":"ContainerDied","Data":"bf8724328f905208d100baa44ca0f46b741f4e7e85686b6d15b855f7d3d8d636"}
Sep 11 19:26:46.071078 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.071040    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2wkv2" podStartSLOduration=4.389030051 podStartE2EDuration="22.071028532s" totalImagesPullingTime="17.681998481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:46.054465776 +0000 UTC m=+22.671635280" watchObservedRunningTime="2026-09-11 19:26:46.071028532 +0000 UTC m=+22.688198023"
Sep 11 19:26:46.071415 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.071391    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-b4f5h" podStartSLOduration=3.9181635630000002 podStartE2EDuration="22.071383492s" totalImagesPullingTime="18.153219929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:46.070932169 +0000 UTC m=+22.688101659" watchObservedRunningTime="2026-09-11 19:26:46.071383492 +0000 UTC m=+22.688552982"
Sep 11 19:26:46.084801 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.084762    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-z8xdl" podStartSLOduration=4.404924323 podStartE2EDuration="22.084749971s" totalImagesPullingTime="17.679825648s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:46.084531101 +0000 UTC m=+22.701700616" watchObservedRunningTime="2026-09-11 19:26:46.084749971 +0000 UTC m=+22.701919463"
Sep 11 19:26:46.098923 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.098887    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d5jh8" podStartSLOduration=4.428900681 podStartE2EDuration="22.098876534s" totalImagesPullingTime="17.669975853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:46.098773847 +0000 UTC m=+22.715943339" watchObservedRunningTime="2026-09-11 19:26:46.098876534 +0000 UTC m=+22.716046025"
Sep 11 19:26:46.199787 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.199765    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l6ktp_d4ca207f-ccf3-4f01-bd0d-cd4cc9148997/konnectivity-agent/0.log" limit=1024
Sep 11 19:26:46.300596 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.300545    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l6ktp_d4ca207f-ccf3-4f01-bd0d-cd4cc9148997/konnectivity-agent/0.log" limit=1024
Sep 11 19:26:46.372229 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.372205    2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 19:26:46.515227 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.515195    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l6ktp_d4ca207f-ccf3-4f01-bd0d-cd4cc9148997/konnectivity-agent/0.log" limit=1024
Sep 11 19:26:46.571122 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.571032    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l6ktp_d4ca207f-ccf3-4f01-bd0d-cd4cc9148997/konnectivity-agent/0.log" limit=1024
Sep 11 19:26:46.687224 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.687185    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l6ktp_d4ca207f-ccf3-4f01-bd0d-cd4cc9148997/konnectivity-agent/0.log" limit=1024
Sep 11 19:26:46.777285 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.777196    2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T19:26:46.372225616Z","UUID":"2eef680f-71f0-4f9f-8893-da091940fe0c","Handler":null,"Name":"","Endpoint":""}
Sep 11 19:26:46.778392 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.778375    2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 19:26:46.778392 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.778405    2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 19:26:46.786498 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.786478    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l6ktp_d4ca207f-ccf3-4f01-bd0d-cd4cc9148997/konnectivity-agent/0.log" limit=1024
Sep 11 19:26:46.912903 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.912875    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l6ktp_d4ca207f-ccf3-4f01-bd0d-cd4cc9148997/konnectivity-agent/0.log" limit=1024
Sep 11 19:26:46.961404 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:46.961374    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l6ktp_d4ca207f-ccf3-4f01-bd0d-cd4cc9148997/konnectivity-agent/0.log" limit=1024
Sep 11 19:26:47.049322 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:47.049287    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7rglf" event={"ID":"865d5eb0-ea5c-4665-ba99-f5bbb171bcc9","Type":"ContainerStarted","Data":"6633f4bdbc737e482e35f8dbd3758725961eb117edde02d68465dbfe380e8632"}
Sep 11 19:26:47.052083 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:47.052056    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp" event={"ID":"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c","Type":"ContainerStarted","Data":"2d8448628f1638c170a5016dcaab373ff199e9ee529dfe2cd28998c44706389f"}
Sep 11 19:26:47.065309 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:47.065261    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7rglf" podStartSLOduration=5.398705924 podStartE2EDuration="23.065241436s" totalImagesPullingTime="17.666535512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:47.06510121 +0000 UTC m=+23.682270703" watchObservedRunningTime="2026-09-11 19:26:47.065241436 +0000 UTC m=+23.682410927"
Sep 11 19:26:47.496884 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:47.496857    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l6ktp"
Sep 11 19:26:47.497496 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:47.497470    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l6ktp"
Sep 11 19:26:47.722901 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:47.722870    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l6ktp_d4ca207f-ccf3-4f01-bd0d-cd4cc9148997/konnectivity-agent/0.log" limit=1024
Sep 11 19:26:47.808130 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:47.808098    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:47.808317 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:47.808098    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:47.808317 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:47.808244    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzscf" podUID="e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef"
Sep 11 19:26:47.808317 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:47.808310    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfnzl" podUID="58780500-db7c-40c3-b4ab-b484237c4ef2"
Sep 11 19:26:47.808484 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:47.808098    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:47.808484 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:47.808385    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdl79" podUID="3402721a-0f03-4681-bc7b-facbf92d31b2"
Sep 11 19:26:47.827184 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:47.827157    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l6ktp_d4ca207f-ccf3-4f01-bd0d-cd4cc9148997/konnectivity-agent/0.log" limit=1024
Sep 11 19:26:48.058260 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:48.058164    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9" event={"ID":"9dd5f51f-878d-4022-8cec-df8ddf05eba9","Type":"ContainerStarted","Data":"236b7f92b6794302f1a13231fc2570f2c4e92af25991cc5209cbdd91ee07ff92"}
Sep 11 19:26:48.060737 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:48.060691    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp" event={"ID":"856ee7ca-bf2a-41bf-8577-13e6c1d2c33c","Type":"ContainerStarted","Data":"673e219f8eea2d47cc96b19b20706b18eaa52941f1a6391671d097370e2619c6"}
Sep 11 19:26:48.061255 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:48.061236    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l6ktp"
Sep 11 19:26:48.061721 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:48.061691    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l6ktp"
Sep 11 19:26:48.079384 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:48.079342    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvfmp" podStartSLOduration=7.697604823 podStartE2EDuration="24.07932688s" totalImagesPullingTime="16.381722057s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:48.078927192 +0000 UTC m=+24.696096683" watchObservedRunningTime="2026-09-11 19:26:48.07932688 +0000 UTC m=+24.696496371"
Sep 11 19:26:48.126576 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:48.126543    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l6ktp_d4ca207f-ccf3-4f01-bd0d-cd4cc9148997/konnectivity-agent/0.log" limit=1024
Sep 11 19:26:48.291864 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:48.291833    2558 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l6ktp_d4ca207f-ccf3-4f01-bd0d-cd4cc9148997/konnectivity-agent/0.log" limit=1024
Sep 11 19:26:49.808239 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:49.808204    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:49.808239 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:49.808222    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:49.808755 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:49.808209    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:49.808755 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:49.808339    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzscf" podUID="e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef"
Sep 11 19:26:49.808755 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:49.808515    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfnzl" podUID="58780500-db7c-40c3-b4ab-b484237c4ef2"
Sep 11 19:26:49.808755 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:49.808600    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdl79" podUID="3402721a-0f03-4681-bc7b-facbf92d31b2"
Sep 11 19:26:51.072095 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:51.072014    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9" event={"ID":"9dd5f51f-878d-4022-8cec-df8ddf05eba9","Type":"ContainerStarted","Data":"5536f5bd55fa613d5dbd6f42c421b7ad17999df9f6cf566392516b4e38926b6a"}
Sep 11 19:26:51.072863 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:51.072302    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:51.087471 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:51.087452    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:51.097746 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:51.097686    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9" podStartSLOduration=9.112428685 podStartE2EDuration="27.097672444s" totalImagesPullingTime="17.985243759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:51.096750066 +0000 UTC m=+27.713919557" watchObservedRunningTime="2026-09-11 19:26:51.097672444 +0000 UTC m=+27.714841934"
Sep 11 19:26:51.807752 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:51.807696    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:51.807918 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:51.807696    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:51.807918 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:51.807839    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:51.807918 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:51.807823    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdl79" podUID="3402721a-0f03-4681-bc7b-facbf92d31b2"
Sep 11 19:26:51.808093 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:51.807927    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfnzl" podUID="58780500-db7c-40c3-b4ab-b484237c4ef2"
Sep 11 19:26:51.808093 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:51.808006    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzscf" podUID="e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef"
Sep 11 19:26:52.076960 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:52.076865    2558 generic.go:356] "Generic (PLEG): container finished" podID="cb1870fa-237b-4a9e-91b1-9741250e32d0" containerID="9616fd315a47e813e097dd1f52e24640b58dce2540bc7be395ccc59c2adabc68" exitCode=0
Sep 11 19:26:52.077626 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:52.076950    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whp4x" event={"ID":"cb1870fa-237b-4a9e-91b1-9741250e32d0","Type":"ContainerDied","Data":"9616fd315a47e813e097dd1f52e24640b58dce2540bc7be395ccc59c2adabc68"}
Sep 11 19:26:52.077626 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:52.077080    2558 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 11 19:26:52.077626 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:52.077554    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:52.096853 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:52.096818    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:52.824605 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:52.824388    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:26:52.837559 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:52.837524    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pfnzl"]
Sep 11 19:26:52.837702 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:52.837626    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:52.837764 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:52.837727    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfnzl" podUID="58780500-db7c-40c3-b4ab-b484237c4ef2"
Sep 11 19:26:52.840012 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:52.839988    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zzscf"]
Sep 11 19:26:52.840247 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:52.840086    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:52.840247 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:52.840193    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzscf" podUID="e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef"
Sep 11 19:26:52.840594 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:52.840571    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mdl79"]
Sep 11 19:26:52.840698 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:52.840643    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:52.840758 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:52.840724    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdl79" podUID="3402721a-0f03-4681-bc7b-facbf92d31b2"
Sep 11 19:26:54.083308 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:54.083273    2558 generic.go:356] "Generic (PLEG): container finished" podID="cb1870fa-237b-4a9e-91b1-9741250e32d0" containerID="4deae52068d7c2e12f280a7b9ac14ceb91a465401b1d639f6737ebb1438b9896" exitCode=0
Sep 11 19:26:54.083743 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:54.083355    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whp4x" event={"ID":"cb1870fa-237b-4a9e-91b1-9741250e32d0","Type":"ContainerDied","Data":"4deae52068d7c2e12f280a7b9ac14ceb91a465401b1d639f6737ebb1438b9896"}
Sep 11 19:26:54.808146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:54.808072    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:54.808289 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:54.808175    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:54.808289 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:54.808173    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdl79" podUID="3402721a-0f03-4681-bc7b-facbf92d31b2"
Sep 11 19:26:54.808289 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:54.808181    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:54.808398 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:54.808323    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzscf" podUID="e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef"
Sep 11 19:26:54.808398 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:54.808357    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfnzl" podUID="58780500-db7c-40c3-b4ab-b484237c4ef2"
Sep 11 19:26:56.089567 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:56.089533    2558 generic.go:356] "Generic (PLEG): container finished" podID="cb1870fa-237b-4a9e-91b1-9741250e32d0" containerID="e13c67499ed54de406c743d58d6327738597a11a272a3e68043d76f9d14fa626" exitCode=0
Sep 11 19:26:56.089959 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:56.089575    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whp4x" event={"ID":"cb1870fa-237b-4a9e-91b1-9741250e32d0","Type":"ContainerDied","Data":"e13c67499ed54de406c743d58d6327738597a11a272a3e68043d76f9d14fa626"}
Sep 11 19:26:56.808588 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:56.808331    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:56.808778 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:56.808338    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:56.808778 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:56.808684    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mdl79" podUID="3402721a-0f03-4681-bc7b-facbf92d31b2"
Sep 11 19:26:56.808778 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:56.808370    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:56.808778 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:56.808748    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zzscf" podUID="e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef"
Sep 11 19:26:56.809003 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:56.808820    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pfnzl" podUID="58780500-db7c-40c3-b4ab-b484237c4ef2"
Sep 11 19:26:57.576274 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:57.576241    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bswn\" (UniqueName: \"kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn\") pod \"network-check-target-mdl79\" (UID: \"3402721a-0f03-4681-bc7b-facbf92d31b2\") " pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:57.576631 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:57.576290    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs\") pod \"network-metrics-daemon-zzscf\" (UID: \"e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef\") " pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:57.576631 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:57.576307    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret\") pod \"global-pull-secret-syncer-pfnzl\" (UID: \"58780500-db7c-40c3-b4ab-b484237c4ef2\") " pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:57.576631 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:57.576394    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 19:26:57.576631 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:57.576394    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:26:57.576631 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:57.576406    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 19:26:57.576631 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:57.576424    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 19:26:57.576631 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:57.576436    2558 projected.go:196] Error preparing data for projected volume kube-api-access-9bswn for pod openshift-network-diagnostics/network-check-target-mdl79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:26:57.576631 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:57.576448    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs podName:e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef nodeName:}" failed. No retries permitted until 2026-09-11 19:27:29.576432489 +0000 UTC m=+66.193601970 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs") pod "network-metrics-daemon-zzscf" (UID: "e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 19:26:57.576631 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:57.576466    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret podName:58780500-db7c-40c3-b4ab-b484237c4ef2 nodeName:}" failed. No retries permitted until 2026-09-11 19:27:29.576459526 +0000 UTC m=+66.193628995 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret") pod "global-pull-secret-syncer-pfnzl" (UID: "58780500-db7c-40c3-b4ab-b484237c4ef2") : object "kube-system"/"original-pull-secret" not registered
Sep 11 19:26:57.576631 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:57.576487    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn podName:3402721a-0f03-4681-bc7b-facbf92d31b2 nodeName:}" failed. No retries permitted until 2026-09-11 19:27:29.576472038 +0000 UTC m=+66.193641522 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9bswn" (UniqueName: "kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn") pod "network-check-target-mdl79" (UID: "3402721a-0f03-4681-bc7b-facbf92d31b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 19:26:58.688075 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.688043    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-43.ec2.internal" event="NodeReady"
Sep 11 19:26:58.688524 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.688192    2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 19:26:58.733158 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.733127    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qq9hm"]
Sep 11 19:26:58.736214 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.736190    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nflsh"]
Sep 11 19:26:58.736338 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.736324    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:26:58.738480 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.738460    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 19:26:58.738585 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.738461    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x76hz\""
Sep 11 19:26:58.738585 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.738461    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 19:26:58.739775 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.739761    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nflsh"
Sep 11 19:26:58.741985 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.741965    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 19:26:58.742078 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.741990    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 19:26:58.742078 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.742031    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 19:26:58.742315 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.742300    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kxtl7\""
Sep 11 19:26:58.745049 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.745031    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qq9hm"]
Sep 11 19:26:58.749004 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.748983    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nflsh"]
Sep 11 19:26:58.807476 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.807451    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:26:58.807611 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.807545    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:26:58.807611 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.807570    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:26:58.809783 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.809766    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 19:26:58.809927 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.809907    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 19:26:58.810028 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.809930    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 19:26:58.810028 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.809975    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2f99f\""
Sep 11 19:26:58.810129 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.810036    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k4fsc\""
Sep 11 19:26:58.810129 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.810096    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 19:26:58.884777 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.884675    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22079f6a-8b4b-46a7-a9c9-5656255a8144-tmp-dir\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:26:58.884777 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.884739    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txfvs\" (UniqueName: \"kubernetes.io/projected/22079f6a-8b4b-46a7-a9c9-5656255a8144-kube-api-access-txfvs\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:26:58.884960 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.884803    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/22079f6a-8b4b-46a7-a9c9-5656255a8144-config-volume\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:26:58.884960 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.884825    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b32f7b7b-b959-4213-b66f-e1840988b22b-cert\") pod \"ingress-canary-nflsh\" (UID: \"b32f7b7b-b959-4213-b66f-e1840988b22b\") " pod="openshift-ingress-canary/ingress-canary-nflsh"
Sep 11 19:26:58.884960 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.884882    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:26:58.884960 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.884906    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clbhq\" (UniqueName: \"kubernetes.io/projected/b32f7b7b-b959-4213-b66f-e1840988b22b-kube-api-access-clbhq\") pod \"ingress-canary-nflsh\" (UID: \"b32f7b7b-b959-4213-b66f-e1840988b22b\") " pod="openshift-ingress-canary/ingress-canary-nflsh"
Sep 11 19:26:58.985974 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.985935    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:26:58.986168 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.985982    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clbhq\" (UniqueName: \"kubernetes.io/projected/b32f7b7b-b959-4213-b66f-e1840988b22b-kube-api-access-clbhq\") pod \"ingress-canary-nflsh\" (UID: \"b32f7b7b-b959-4213-b66f-e1840988b22b\") " pod="openshift-ingress-canary/ingress-canary-nflsh"
Sep 11 19:26:58.986168 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.986017    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22079f6a-8b4b-46a7-a9c9-5656255a8144-tmp-dir\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:26:58.986168 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.986045    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txfvs\" (UniqueName: \"kubernetes.io/projected/22079f6a-8b4b-46a7-a9c9-5656255a8144-kube-api-access-txfvs\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:26:58.986168 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:58.986089    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 19:26:58.986168 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.986140    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/22079f6a-8b4b-46a7-a9c9-5656255a8144-config-volume\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:26:58.986168 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:58.986163    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls podName:22079f6a-8b4b-46a7-a9c9-5656255a8144 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:59.486142362 +0000 UTC m=+36.103311840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls") pod "dns-default-qq9hm" (UID: "22079f6a-8b4b-46a7-a9c9-5656255a8144") : secret "dns-default-metrics-tls" not found
Sep 11 19:26:58.986451 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.986189    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b32f7b7b-b959-4213-b66f-e1840988b22b-cert\") pod \"ingress-canary-nflsh\" (UID: \"b32f7b7b-b959-4213-b66f-e1840988b22b\") " pod="openshift-ingress-canary/ingress-canary-nflsh"
Sep 11 19:26:58.986493 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.986452    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22079f6a-8b4b-46a7-a9c9-5656255a8144-tmp-dir\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:26:58.986938 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.986913    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/22079f6a-8b4b-46a7-a9c9-5656255a8144-config-volume\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:26:58.990362 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:58.990338    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b32f7b7b-b959-4213-b66f-e1840988b22b-cert\") pod \"ingress-canary-nflsh\" (UID: \"b32f7b7b-b959-4213-b66f-e1840988b22b\") " pod="openshift-ingress-canary/ingress-canary-nflsh"
Sep 11 19:26:59.006895 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:59.006870    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txfvs\" (UniqueName: \"kubernetes.io/projected/22079f6a-8b4b-46a7-a9c9-5656255a8144-kube-api-access-txfvs\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:26:59.008298 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:59.008275    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clbhq\" (UniqueName: \"kubernetes.io/projected/b32f7b7b-b959-4213-b66f-e1840988b22b-kube-api-access-clbhq\") pod \"ingress-canary-nflsh\" (UID: \"b32f7b7b-b959-4213-b66f-e1840988b22b\") " pod="openshift-ingress-canary/ingress-canary-nflsh"
Sep 11 19:26:59.054405 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:59.054378    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nflsh"
Sep 11 19:26:59.334413 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:59.334384    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nflsh"]
Sep 11 19:26:59.339030 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:26:59.338988    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32f7b7b_b959_4213_b66f_e1840988b22b.slice/crio-c3efde9ca1feb8038c785b0d5f5aae80ce09f213d4d7a637e4b1a97396f3c19b WatchSource:0}: Error finding container c3efde9ca1feb8038c785b0d5f5aae80ce09f213d4d7a637e4b1a97396f3c19b: Status 404 returned error can't find the container with id c3efde9ca1feb8038c785b0d5f5aae80ce09f213d4d7a637e4b1a97396f3c19b
Sep 11 19:26:59.490315 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:26:59.490283    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:26:59.490487 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:59.490389    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 19:26:59.490487 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:26:59.490444    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls podName:22079f6a-8b4b-46a7-a9c9-5656255a8144 nodeName:}" failed. No retries permitted until 2026-09-11 19:27:00.490429616 +0000 UTC m=+37.107599086 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls") pod "dns-default-qq9hm" (UID: "22079f6a-8b4b-46a7-a9c9-5656255a8144") : secret "dns-default-metrics-tls" not found
Sep 11 19:27:00.101188 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.101151    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nflsh" event={"ID":"b32f7b7b-b959-4213-b66f-e1840988b22b","Type":"ContainerStarted","Data":"c3efde9ca1feb8038c785b0d5f5aae80ce09f213d4d7a637e4b1a97396f3c19b"}
Sep 11 19:27:00.379805 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.379735    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9"]
Sep 11 19:27:00.384198 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.384171    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9"
Sep 11 19:27:00.388074 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.388048    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 19:27:00.388396 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.388376    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 19:27:00.388650 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.388635    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 19:27:00.388807 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.388785    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-7xw7k\""
Sep 11 19:27:00.388898 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.388868    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 19:27:00.390805 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.390779    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"]
Sep 11 19:27:00.395324 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.395301    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"
Sep 11 19:27:00.397625 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.397607    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 19:27:00.408777 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.408753    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9"]
Sep 11 19:27:00.420421 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.420382    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"]
Sep 11 19:27:00.498349 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.498313    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:27:00.498544 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.498366    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6039f54e-4387-4ba3-979a-21ade90fee05-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6689948644-7q7k9\" (UID: \"6039f54e-4387-4ba3-979a-21ade90fee05\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9"
Sep 11 19:27:00.498544 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.498403    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4122f5ec-feaf-4a04-be80-d8231f2013aa-tmp\") pod \"klusterlet-addon-workmgr-6d8b86cd98-dcwtj\" (UID: \"4122f5ec-feaf-4a04-be80-d8231f2013aa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"
Sep 11 19:27:00.498544 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.498428    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7gsx\" (UniqueName: \"kubernetes.io/projected/4122f5ec-feaf-4a04-be80-d8231f2013aa-kube-api-access-k7gsx\") pod \"klusterlet-addon-workmgr-6d8b86cd98-dcwtj\" (UID: \"4122f5ec-feaf-4a04-be80-d8231f2013aa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"
Sep 11 19:27:00.498743 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:00.498588    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 19:27:00.498743 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.498627    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4122f5ec-feaf-4a04-be80-d8231f2013aa-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d8b86cd98-dcwtj\" (UID: \"4122f5ec-feaf-4a04-be80-d8231f2013aa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"
Sep 11 19:27:00.498743 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:00.498646    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls podName:22079f6a-8b4b-46a7-a9c9-5656255a8144 nodeName:}" failed. No retries permitted until 2026-09-11 19:27:02.498627141 +0000 UTC m=+39.115796621 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls") pod "dns-default-qq9hm" (UID: "22079f6a-8b4b-46a7-a9c9-5656255a8144") : secret "dns-default-metrics-tls" not found
Sep 11 19:27:00.498743 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.498683    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fb5c\" (UniqueName: \"kubernetes.io/projected/6039f54e-4387-4ba3-979a-21ade90fee05-kube-api-access-5fb5c\") pod \"managed-serviceaccount-addon-agent-6689948644-7q7k9\" (UID: \"6039f54e-4387-4ba3-979a-21ade90fee05\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9"
Sep 11 19:27:00.503750 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.503726    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79cfc578bf-bccp4"]
Sep 11 19:27:00.507028 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.507006    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.509058 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.508992    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 19:27:00.509563 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.509290    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 19:27:00.509563 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.509357    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p67zb\""
Sep 11 19:27:00.509563 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.509394    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 19:27:00.519134 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.519099    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 19:27:00.526109 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.526088    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79cfc578bf-bccp4"]
Sep 11 19:27:00.599342 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.599077    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4122f5ec-feaf-4a04-be80-d8231f2013aa-tmp\") pod \"klusterlet-addon-workmgr-6d8b86cd98-dcwtj\" (UID: \"4122f5ec-feaf-4a04-be80-d8231f2013aa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"
Sep 11 19:27:00.599342 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.599132    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7gsx\" (UniqueName: \"kubernetes.io/projected/4122f5ec-feaf-4a04-be80-d8231f2013aa-kube-api-access-k7gsx\") pod \"klusterlet-addon-workmgr-6d8b86cd98-dcwtj\" (UID: \"4122f5ec-feaf-4a04-be80-d8231f2013aa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"
Sep 11 19:27:00.599342 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.599164    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3a98b87-6d7d-483b-b456-f9f80e1f2407-trusted-ca\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.599342 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.599258    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-bound-sa-token\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.599342 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.599333    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.599732 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.599359    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4tk4\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-kube-api-access-q4tk4\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.599732 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.599381    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3a98b87-6d7d-483b-b456-f9f80e1f2407-ca-trust-extracted\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.599732 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.599398    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-certificates\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.599732 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.599478    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3a98b87-6d7d-483b-b456-f9f80e1f2407-installation-pull-secrets\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.599732 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.599559    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4122f5ec-feaf-4a04-be80-d8231f2013aa-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d8b86cd98-dcwtj\" (UID: \"4122f5ec-feaf-4a04-be80-d8231f2013aa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"
Sep 11 19:27:00.599732 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.599557    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4122f5ec-feaf-4a04-be80-d8231f2013aa-tmp\") pod \"klusterlet-addon-workmgr-6d8b86cd98-dcwtj\" (UID: \"4122f5ec-feaf-4a04-be80-d8231f2013aa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"
Sep 11 19:27:00.599732 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.599623    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fb5c\" (UniqueName: \"kubernetes.io/projected/6039f54e-4387-4ba3-979a-21ade90fee05-kube-api-access-5fb5c\") pod \"managed-serviceaccount-addon-agent-6689948644-7q7k9\" (UID: \"6039f54e-4387-4ba3-979a-21ade90fee05\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9"
Sep 11 19:27:00.599732 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.599684    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3a98b87-6d7d-483b-b456-f9f80e1f2407-image-registry-private-configuration\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.600028 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.599742    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6039f54e-4387-4ba3-979a-21ade90fee05-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6689948644-7q7k9\" (UID: \"6039f54e-4387-4ba3-979a-21ade90fee05\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9"
Sep 11 19:27:00.603161 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.602975    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6039f54e-4387-4ba3-979a-21ade90fee05-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6689948644-7q7k9\" (UID: \"6039f54e-4387-4ba3-979a-21ade90fee05\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9"
Sep 11 19:27:00.603885 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.603837    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4122f5ec-feaf-4a04-be80-d8231f2013aa-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d8b86cd98-dcwtj\" (UID: \"4122f5ec-feaf-4a04-be80-d8231f2013aa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"
Sep 11 19:27:00.608775 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.608748    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fb5c\" (UniqueName: \"kubernetes.io/projected/6039f54e-4387-4ba3-979a-21ade90fee05-kube-api-access-5fb5c\") pod \"managed-serviceaccount-addon-agent-6689948644-7q7k9\" (UID: \"6039f54e-4387-4ba3-979a-21ade90fee05\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9"
Sep 11 19:27:00.608885 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.608870    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7gsx\" (UniqueName: \"kubernetes.io/projected/4122f5ec-feaf-4a04-be80-d8231f2013aa-kube-api-access-k7gsx\") pod \"klusterlet-addon-workmgr-6d8b86cd98-dcwtj\" (UID: \"4122f5ec-feaf-4a04-be80-d8231f2013aa\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"
Sep 11 19:27:00.700975 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.700514    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3a98b87-6d7d-483b-b456-f9f80e1f2407-trusted-ca\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.700975 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.700564    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-bound-sa-token\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.700975 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.700620    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.700975 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.700645    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4tk4\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-kube-api-access-q4tk4\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.700975 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.700676    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3a98b87-6d7d-483b-b456-f9f80e1f2407-ca-trust-extracted\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.700975 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.700700    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-certificates\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.700975 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.700759    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3a98b87-6d7d-483b-b456-f9f80e1f2407-installation-pull-secrets\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.700975 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.700826    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3a98b87-6d7d-483b-b456-f9f80e1f2407-image-registry-private-configuration\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.701471 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.701208    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3a98b87-6d7d-483b-b456-f9f80e1f2407-ca-trust-extracted\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.701471 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:00.701316    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 19:27:00.701471 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:00.701334    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79cfc578bf-bccp4: secret "image-registry-tls" not found
Sep 11 19:27:00.701471 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:00.701402    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls podName:b3a98b87-6d7d-483b-b456-f9f80e1f2407 nodeName:}" failed. No retries permitted until 2026-09-11 19:27:01.201380567 +0000 UTC m=+37.818550036 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls") pod "image-registry-79cfc578bf-bccp4" (UID: "b3a98b87-6d7d-483b-b456-f9f80e1f2407") : secret "image-registry-tls" not found
Sep 11 19:27:00.701702 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.701651    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3a98b87-6d7d-483b-b456-f9f80e1f2407-trusted-ca\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.702085 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.702035    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-certificates\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.704382 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.704332    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3a98b87-6d7d-483b-b456-f9f80e1f2407-image-registry-private-configuration\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.704650 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.704629    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3a98b87-6d7d-483b-b456-f9f80e1f2407-installation-pull-secrets\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.706697 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.706666    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9"
Sep 11 19:27:00.710042 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.709813    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4tk4\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-kube-api-access-q4tk4\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.710042 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.709832    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-bound-sa-token\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:00.717827 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:00.717807    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"
Sep 11 19:27:01.204219 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:01.204178    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:01.204653 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:01.204341    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 19:27:01.204653 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:01.204358    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79cfc578bf-bccp4: secret "image-registry-tls" not found
Sep 11 19:27:01.204653 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:01.204438    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls podName:b3a98b87-6d7d-483b-b456-f9f80e1f2407 nodeName:}" failed. No retries permitted until 2026-09-11 19:27:02.204417648 +0000 UTC m=+38.821587117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls") pod "image-registry-79cfc578bf-bccp4" (UID: "b3a98b87-6d7d-483b-b456-f9f80e1f2407") : secret "image-registry-tls" not found
Sep 11 19:27:02.212855 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:02.212820    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:02.213296 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:02.212998    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 19:27:02.213296 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:02.213024    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79cfc578bf-bccp4: secret "image-registry-tls" not found
Sep 11 19:27:02.213296 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:02.213099    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls podName:b3a98b87-6d7d-483b-b456-f9f80e1f2407 nodeName:}" failed. No retries permitted until 2026-09-11 19:27:04.213077085 +0000 UTC m=+40.830246578 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls") pod "image-registry-79cfc578bf-bccp4" (UID: "b3a98b87-6d7d-483b-b456-f9f80e1f2407") : secret "image-registry-tls" not found
Sep 11 19:27:02.516231 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:02.516131    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:27:02.516398 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:02.516309    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 19:27:02.516398 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:02.516390    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls podName:22079f6a-8b4b-46a7-a9c9-5656255a8144 nodeName:}" failed. No retries permitted until 2026-09-11 19:27:06.516370427 +0000 UTC m=+43.133539901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls") pod "dns-default-qq9hm" (UID: "22079f6a-8b4b-46a7-a9c9-5656255a8144") : secret "dns-default-metrics-tls" not found
Sep 11 19:27:03.431010 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:03.430976    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"]
Sep 11 19:27:03.433975 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:03.433951    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9"]
Sep 11 19:27:03.444020 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:27:03.443976    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6039f54e_4387_4ba3_979a_21ade90fee05.slice/crio-b9c6638dd226c57703dcad6f05e8dfee4875e58542b59464fd42646acecbba6a WatchSource:0}: Error finding container b9c6638dd226c57703dcad6f05e8dfee4875e58542b59464fd42646acecbba6a: Status 404 returned error can't find the container with id b9c6638dd226c57703dcad6f05e8dfee4875e58542b59464fd42646acecbba6a
Sep 11 19:27:03.444255 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:27:03.444232    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4122f5ec_feaf_4a04_be80_d8231f2013aa.slice/crio-7c5b2ca9a1ddd98c1b92d3b7273bac3c1e1103277dc43667d667d00d269b59d3 WatchSource:0}: Error finding container 7c5b2ca9a1ddd98c1b92d3b7273bac3c1e1103277dc43667d667d00d269b59d3: Status 404 returned error can't find the container with id 7c5b2ca9a1ddd98c1b92d3b7273bac3c1e1103277dc43667d667d00d269b59d3
Sep 11 19:27:04.113562 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:04.113527    2558 generic.go:356] "Generic (PLEG): container finished" podID="cb1870fa-237b-4a9e-91b1-9741250e32d0" containerID="ab84a03a0e8aee50839792e18a7b5f50b92ee68f859a7cd3c5b8d6a843180bd8" exitCode=0
Sep 11 19:27:04.113737 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:04.113595    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whp4x" event={"ID":"cb1870fa-237b-4a9e-91b1-9741250e32d0","Type":"ContainerDied","Data":"ab84a03a0e8aee50839792e18a7b5f50b92ee68f859a7cd3c5b8d6a843180bd8"}
Sep 11 19:27:04.115181 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:04.115154    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nflsh" event={"ID":"b32f7b7b-b959-4213-b66f-e1840988b22b","Type":"ContainerStarted","Data":"1d7aee8c1b97272deb2a1f3734f0f0d79ed6f1f0168778d24a52a4f95864c9c2"}
Sep 11 19:27:04.116472 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:04.116449    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9" event={"ID":"6039f54e-4387-4ba3-979a-21ade90fee05","Type":"ContainerStarted","Data":"b9c6638dd226c57703dcad6f05e8dfee4875e58542b59464fd42646acecbba6a"}
Sep 11 19:27:04.117611 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:04.117591    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj" event={"ID":"4122f5ec-feaf-4a04-be80-d8231f2013aa","Type":"ContainerStarted","Data":"7c5b2ca9a1ddd98c1b92d3b7273bac3c1e1103277dc43667d667d00d269b59d3"}
Sep 11 19:27:04.144138 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:04.144090    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nflsh" podStartSLOduration=2.31572593 podStartE2EDuration="6.144076578s" totalImagesPullingTime="3.828350648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:27:04.143407484 +0000 UTC m=+40.760577000" watchObservedRunningTime="2026-09-11 19:27:04.144076578 +0000 UTC m=+40.761246090"
Sep 11 19:27:04.230634 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:04.230603    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:04.230766 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:04.230730    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 19:27:04.230766 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:04.230745    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79cfc578bf-bccp4: secret "image-registry-tls" not found
Sep 11 19:27:04.230844 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:04.230799    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls podName:b3a98b87-6d7d-483b-b456-f9f80e1f2407 nodeName:}" failed. No retries permitted until 2026-09-11 19:27:08.230780782 +0000 UTC m=+44.847950265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls") pod "image-registry-79cfc578bf-bccp4" (UID: "b3a98b87-6d7d-483b-b456-f9f80e1f2407") : secret "image-registry-tls" not found
Sep 11 19:27:05.124986 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:05.124940    2558 generic.go:356] "Generic (PLEG): container finished" podID="cb1870fa-237b-4a9e-91b1-9741250e32d0" containerID="5bb9fd6a421177af5a16506ea7be93e0925f41ba7b0be30e147e0524b3b88119" exitCode=0
Sep 11 19:27:05.125425 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:05.125106    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whp4x" event={"ID":"cb1870fa-237b-4a9e-91b1-9741250e32d0","Type":"ContainerDied","Data":"5bb9fd6a421177af5a16506ea7be93e0925f41ba7b0be30e147e0524b3b88119"}
Sep 11 19:27:06.132877 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:06.132831    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-whp4x" event={"ID":"cb1870fa-237b-4a9e-91b1-9741250e32d0","Type":"ContainerStarted","Data":"c15d55e3f138f752ff040ed77e91b9db82dcb2b604474249f5c435dcb75d81d3"}
Sep 11 19:27:06.153832 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:06.153410    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-whp4x" podStartSLOduration=9.919229534 podStartE2EDuration="42.153388006s" totalImagesPullingTime="32.234158472s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:27:06.151838915 +0000 UTC m=+42.769008433" watchObservedRunningTime="2026-09-11 19:27:06.153388006 +0000 UTC m=+42.770557495"
Sep 11 19:27:06.549362 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:06.549326    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:27:06.549603 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:06.549433    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 19:27:06.549603 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:06.549492    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls podName:22079f6a-8b4b-46a7-a9c9-5656255a8144 nodeName:}" failed. No retries permitted until 2026-09-11 19:27:14.549478758 +0000 UTC m=+51.166648226 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls") pod "dns-default-qq9hm" (UID: "22079f6a-8b4b-46a7-a9c9-5656255a8144") : secret "dns-default-metrics-tls" not found
Sep 11 19:27:08.264328 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:08.264286    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:08.264738 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:08.264443    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 19:27:08.264738 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:08.264463    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79cfc578bf-bccp4: secret "image-registry-tls" not found
Sep 11 19:27:08.264738 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:08.264521    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls podName:b3a98b87-6d7d-483b-b456-f9f80e1f2407 nodeName:}" failed. No retries permitted until 2026-09-11 19:27:16.264505956 +0000 UTC m=+52.881675429 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls") pod "image-registry-79cfc578bf-bccp4" (UID: "b3a98b87-6d7d-483b-b456-f9f80e1f2407") : secret "image-registry-tls" not found
Sep 11 19:27:09.142802 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:09.142750    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj" event={"ID":"4122f5ec-feaf-4a04-be80-d8231f2013aa","Type":"ContainerStarted","Data":"b621192b66979af6474e06c32769a9e11206cb52ae917754df5d630cb5190383"}
Sep 11 19:27:09.143004 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:09.142980    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"
Sep 11 19:27:09.145002 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:09.144979    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"
Sep 11 19:27:09.159802 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:09.159759    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj" podStartSLOduration=4.132342079 podStartE2EDuration="9.159745656s" totalImagesPullingTime="5.027403577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:27:09.159007084 +0000 UTC m=+45.776176576" watchObservedRunningTime="2026-09-11 19:27:09.159745656 +0000 UTC m=+45.776915147"
Sep 11 19:27:11.148477 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:11.148394    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9" event={"ID":"6039f54e-4387-4ba3-979a-21ade90fee05","Type":"ContainerStarted","Data":"a8342ff154442ff765108774a562f5aeccf2c52b161eb2903c1cafe7360678dc"}
Sep 11 19:27:11.162269 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:11.162228    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9" podStartSLOduration=3.883803421 podStartE2EDuration="11.162212652s" totalImagesPullingTime="7.278409231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:27:11.16176521 +0000 UTC m=+47.778934701" watchObservedRunningTime="2026-09-11 19:27:11.162212652 +0000 UTC m=+47.779382142"
Sep 11 19:27:14.608298 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:14.608253    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:27:14.608680 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:14.608411    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 19:27:14.608680 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:14.608473    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls podName:22079f6a-8b4b-46a7-a9c9-5656255a8144 nodeName:}" failed. No retries permitted until 2026-09-11 19:27:30.60845471 +0000 UTC m=+67.225624193 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls") pod "dns-default-qq9hm" (UID: "22079f6a-8b4b-46a7-a9c9-5656255a8144") : secret "dns-default-metrics-tls" not found
Sep 11 19:27:16.325785 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:16.325746    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:16.326186 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:16.325860    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 19:27:16.326186 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:16.325871    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79cfc578bf-bccp4: secret "image-registry-tls" not found
Sep 11 19:27:16.326186 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:16.325934    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls podName:b3a98b87-6d7d-483b-b456-f9f80e1f2407 nodeName:}" failed. No retries permitted until 2026-09-11 19:27:32.325919635 +0000 UTC m=+68.943089104 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls") pod "image-registry-79cfc578bf-bccp4" (UID: "b3a98b87-6d7d-483b-b456-f9f80e1f2407") : secret "image-registry-tls" not found
Sep 11 19:27:24.093344 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:24.093318    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-q9jc9"
Sep 11 19:27:29.621686 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:29.621639    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bswn\" (UniqueName: \"kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn\") pod \"network-check-target-mdl79\" (UID: \"3402721a-0f03-4681-bc7b-facbf92d31b2\") " pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:27:29.621686 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:29.621720    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs\") pod \"network-metrics-daemon-zzscf\" (UID: \"e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef\") " pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:27:29.622148 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:29.621741    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret\") pod \"global-pull-secret-syncer-pfnzl\" (UID: \"58780500-db7c-40c3-b4ab-b484237c4ef2\") " pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:27:29.624097 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:29.624071    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 19:27:29.624160 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:29.624100    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 19:27:29.624160 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:29.624076    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 19:27:29.632884 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:29.632866    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 19:27:29.632936 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:29.632930    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs podName:e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef nodeName:}" failed. No retries permitted until 2026-09-11 19:28:33.632911974 +0000 UTC m=+130.250081458 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs") pod "network-metrics-daemon-zzscf" (UID: "e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef") : secret "metrics-daemon-secret" not found
Sep 11 19:27:29.635023 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:29.635008    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 19:27:29.635530 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:29.635511    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58780500-db7c-40c3-b4ab-b484237c4ef2-original-pull-secret\") pod \"global-pull-secret-syncer-pfnzl\" (UID: \"58780500-db7c-40c3-b4ab-b484237c4ef2\") " pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:27:29.645457 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:29.645429    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bswn\" (UniqueName: \"kubernetes.io/projected/3402721a-0f03-4681-bc7b-facbf92d31b2-kube-api-access-9bswn\") pod \"network-check-target-mdl79\" (UID: \"3402721a-0f03-4681-bc7b-facbf92d31b2\") " pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:27:29.718615 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:29.718580    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2f99f\""
Sep 11 19:27:29.726293 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:29.726267    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pfnzl"
Sep 11 19:27:29.727341 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:29.727324    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:27:29.996289 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:29.996255    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pfnzl"]
Sep 11 19:27:29.996984 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:29.996962    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mdl79"]
Sep 11 19:27:29.999346 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:27:29.999318    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3402721a_0f03_4681_bc7b_facbf92d31b2.slice/crio-50e9fb80dc3d2c637dba0bb8cd0db6b00d443a614c9d9279affa9eeb3b9e9fea WatchSource:0}: Error finding container 50e9fb80dc3d2c637dba0bb8cd0db6b00d443a614c9d9279affa9eeb3b9e9fea: Status 404 returned error can't find the container with id 50e9fb80dc3d2c637dba0bb8cd0db6b00d443a614c9d9279affa9eeb3b9e9fea
Sep 11 19:27:29.999817 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:27:29.999796    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58780500_db7c_40c3_b4ab_b484237c4ef2.slice/crio-ba63eec0b725fd058f0010345f8a2bc1a0b0919748ec653ceb9513cd6280556a WatchSource:0}: Error finding container ba63eec0b725fd058f0010345f8a2bc1a0b0919748ec653ceb9513cd6280556a: Status 404 returned error can't find the container with id ba63eec0b725fd058f0010345f8a2bc1a0b0919748ec653ceb9513cd6280556a
Sep 11 19:27:30.192951 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:30.192915    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mdl79" event={"ID":"3402721a-0f03-4681-bc7b-facbf92d31b2","Type":"ContainerStarted","Data":"50e9fb80dc3d2c637dba0bb8cd0db6b00d443a614c9d9279affa9eeb3b9e9fea"}
Sep 11 19:27:30.194041 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:30.194011    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pfnzl" event={"ID":"58780500-db7c-40c3-b4ab-b484237c4ef2","Type":"ContainerStarted","Data":"ba63eec0b725fd058f0010345f8a2bc1a0b0919748ec653ceb9513cd6280556a"}
Sep 11 19:27:30.629591 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:30.629536    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:27:30.630169 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:30.629724    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 19:27:30.630169 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:30.629812    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls podName:22079f6a-8b4b-46a7-a9c9-5656255a8144 nodeName:}" failed. No retries permitted until 2026-09-11 19:28:02.629790699 +0000 UTC m=+99.246960168 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls") pod "dns-default-qq9hm" (UID: "22079f6a-8b4b-46a7-a9c9-5656255a8144") : secret "dns-default-metrics-tls" not found
Sep 11 19:27:32.346250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:32.346214    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:27:32.346686 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:32.346368    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 19:27:32.346686 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:32.346387    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79cfc578bf-bccp4: secret "image-registry-tls" not found
Sep 11 19:27:32.346686 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:27:32.346455    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls podName:b3a98b87-6d7d-483b-b456-f9f80e1f2407 nodeName:}" failed. No retries permitted until 2026-09-11 19:28:04.346440193 +0000 UTC m=+100.963609667 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls") pod "image-registry-79cfc578bf-bccp4" (UID: "b3a98b87-6d7d-483b-b456-f9f80e1f2407") : secret "image-registry-tls" not found
Sep 11 19:27:35.210481 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:35.210445    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mdl79" event={"ID":"3402721a-0f03-4681-bc7b-facbf92d31b2","Type":"ContainerStarted","Data":"c6177267440ec865f38a684ad29ec22fb1c9e9715a99af6dbb48e8926855885f"}
Sep 11 19:27:35.210950 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:35.210574    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:27:35.212040 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:35.212018    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pfnzl" event={"ID":"58780500-db7c-40c3-b4ab-b484237c4ef2","Type":"ContainerStarted","Data":"c59ca13b91748c46ebf7de4bfe3fdcaa796ff0f9823f5bfb5780979d2b89d546"}
Sep 11 19:27:35.228163 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:35.228115    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mdl79" podStartSLOduration=66.666200194 podStartE2EDuration="1m11.228100099s" totalImagesPullingTime="4.561899905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:27:35.227743845 +0000 UTC m=+71.844913338" watchObservedRunningTime="2026-09-11 19:27:35.228100099 +0000 UTC m=+71.845269584"
Sep 11 19:27:35.241023 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:27:35.240960    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pfnzl" podStartSLOduration=66.673193933 podStartE2EDuration="1m11.240941481s" totalImagesPullingTime="4.567747548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:27:35.2401514 +0000 UTC m=+71.857320892" watchObservedRunningTime="2026-09-11 19:27:35.240941481 +0000 UTC m=+71.858110973"
Sep 11 19:28:02.657371 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:28:02.657281    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:28:02.657821 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:28:02.657441    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 19:28:02.657821 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:28:02.657527    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls podName:22079f6a-8b4b-46a7-a9c9-5656255a8144 nodeName:}" failed. No retries permitted until 2026-09-11 19:29:06.657506461 +0000 UTC m=+163.274675950 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls") pod "dns-default-qq9hm" (UID: "22079f6a-8b4b-46a7-a9c9-5656255a8144") : secret "dns-default-metrics-tls" not found
Sep 11 19:28:04.371292 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:28:04.371248    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:28:04.371671 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:28:04.371411    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 19:28:04.371671 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:28:04.371431    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79cfc578bf-bccp4: secret "image-registry-tls" not found
Sep 11 19:28:04.371671 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:28:04.371488    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls podName:b3a98b87-6d7d-483b-b456-f9f80e1f2407 nodeName:}" failed. No retries permitted until 2026-09-11 19:29:08.371472345 +0000 UTC m=+164.988641813 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls") pod "image-registry-79cfc578bf-bccp4" (UID: "b3a98b87-6d7d-483b-b456-f9f80e1f2407") : secret "image-registry-tls" not found
Sep 11 19:28:06.220561 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:28:06.220527    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mdl79"
Sep 11 19:28:33.668277 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:28:33.668242    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs\") pod \"network-metrics-daemon-zzscf\" (UID: \"e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef\") " pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:28:33.668746 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:28:33.668364    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 19:28:33.668746 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:28:33.668419    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs podName:e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef nodeName:}" failed. No retries permitted until 2026-09-11 19:30:35.668404949 +0000 UTC m=+252.285574419 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs") pod "network-metrics-daemon-zzscf" (UID: "e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef") : secret "metrics-daemon-secret" not found
Sep 11 19:29:01.747368 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:29:01.747311    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-qq9hm" podUID="22079f6a-8b4b-46a7-a9c9-5656255a8144"
Sep 11 19:29:01.821794 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:29:01.821766    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-zzscf" podUID="e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef"
Sep 11 19:29:02.434954 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:29:02.434925    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:29:03.520900 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:29:03.520831    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-79cfc578bf-bccp4" podUID="b3a98b87-6d7d-483b-b456-f9f80e1f2407"
Sep 11 19:29:04.439510 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:29:04.439482    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:29:06.691219 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:29:06.691182    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:29:06.691602 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:29:06.691342    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 19:29:06.691602 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:29:06.691419    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls podName:22079f6a-8b4b-46a7-a9c9-5656255a8144 nodeName:}" failed. No retries permitted until 2026-09-11 19:31:08.691399226 +0000 UTC m=+285.308568716 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls") pod "dns-default-qq9hm" (UID: "22079f6a-8b4b-46a7-a9c9-5656255a8144") : secret "dns-default-metrics-tls" not found
Sep 11 19:29:08.402103 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:29:08.402065    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls\") pod \"image-registry-79cfc578bf-bccp4\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") " pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:29:08.402552 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:29:08.402179    2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 19:29:08.402552 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:29:08.402191    2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-79cfc578bf-bccp4: secret "image-registry-tls" not found
Sep 11 19:29:08.402552 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:29:08.402247    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls podName:b3a98b87-6d7d-483b-b456-f9f80e1f2407 nodeName:}" failed. No retries permitted until 2026-09-11 19:31:10.402232995 +0000 UTC m=+287.019402464 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls") pod "image-registry-79cfc578bf-bccp4" (UID: "b3a98b87-6d7d-483b-b456-f9f80e1f2407") : secret "image-registry-tls" not found
Sep 11 19:29:09.144011 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:29:09.143958    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj" podUID="4122f5ec-feaf-4a04-be80-d8231f2013aa" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.9:8000/readyz\": dial tcp 10.133.0.9:8000: connect: connection refused"
Sep 11 19:29:09.453438 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:29:09.453401    2558 generic.go:356] "Generic (PLEG): container finished" podID="4122f5ec-feaf-4a04-be80-d8231f2013aa" containerID="b621192b66979af6474e06c32769a9e11206cb52ae917754df5d630cb5190383" exitCode=1
Sep 11 19:29:09.453809 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:29:09.453474    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj" event={"ID":"4122f5ec-feaf-4a04-be80-d8231f2013aa","Type":"ContainerDied","Data":"b621192b66979af6474e06c32769a9e11206cb52ae917754df5d630cb5190383"}
Sep 11 19:29:09.453851 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:29:09.453824    2558 scope.go:117] "RemoveContainer" containerID="b621192b66979af6474e06c32769a9e11206cb52ae917754df5d630cb5190383"
Sep 11 19:29:10.458166 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:29:10.458130    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj" event={"ID":"4122f5ec-feaf-4a04-be80-d8231f2013aa","Type":"ContainerStarted","Data":"17c584abe1bd0211293c5965697b69fadb9213e1f4e134a94e5d0a6d5c760688"}
Sep 11 19:29:10.458542 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:29:10.458415    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"
Sep 11 19:29:10.459046 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:29:10.459026    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"
Sep 11 19:29:11.462693 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:29:11.462659    2558 generic.go:356] "Generic (PLEG): container finished" podID="6039f54e-4387-4ba3-979a-21ade90fee05" containerID="a8342ff154442ff765108774a562f5aeccf2c52b161eb2903c1cafe7360678dc" exitCode=255
Sep 11 19:29:11.463070 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:29:11.462732    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9" event={"ID":"6039f54e-4387-4ba3-979a-21ade90fee05","Type":"ContainerDied","Data":"a8342ff154442ff765108774a562f5aeccf2c52b161eb2903c1cafe7360678dc"}
Sep 11 19:29:11.468196 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:29:11.468181    2558 scope.go:117] "RemoveContainer" containerID="a8342ff154442ff765108774a562f5aeccf2c52b161eb2903c1cafe7360678dc"
Sep 11 19:29:12.468509 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:29:12.468474    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9" event={"ID":"6039f54e-4387-4ba3-979a-21ade90fee05","Type":"ContainerStarted","Data":"5c3f4f0547176dbcdc41ce89615a1a35d0ccba8b13f59049386ae684f540602c"}
Sep 11 19:29:14.807496 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:29:14.807445    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:30:28.703272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:28.703238    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-55czf"]
Sep 11 19:30:28.705901 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:28.705884    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-55czf"
Sep 11 19:30:28.707886 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:28.707864    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 11 19:30:28.708570 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:28.708554    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kkvg6\""
Sep 11 19:30:28.708643 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:28.708562    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 11 19:30:28.717733 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:28.717691    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-55czf"]
Sep 11 19:30:28.834274 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:28.834235    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gqt8\" (UniqueName: \"kubernetes.io/projected/88643ea7-21db-4b13-a376-ac84810bcf61-kube-api-access-7gqt8\") pod \"volume-data-source-validator-86695b7cc7-55czf\" (UID: \"88643ea7-21db-4b13-a376-ac84810bcf61\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-55czf"
Sep 11 19:30:28.935615 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:28.935580    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gqt8\" (UniqueName: \"kubernetes.io/projected/88643ea7-21db-4b13-a376-ac84810bcf61-kube-api-access-7gqt8\") pod \"volume-data-source-validator-86695b7cc7-55czf\" (UID: \"88643ea7-21db-4b13-a376-ac84810bcf61\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-55czf"
Sep 11 19:30:28.943631 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:28.943609    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gqt8\" (UniqueName: \"kubernetes.io/projected/88643ea7-21db-4b13-a376-ac84810bcf61-kube-api-access-7gqt8\") pod \"volume-data-source-validator-86695b7cc7-55czf\" (UID: \"88643ea7-21db-4b13-a376-ac84810bcf61\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-55czf"
Sep 11 19:30:29.014118 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:29.014027    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-55czf"
Sep 11 19:30:29.280822 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:29.280743    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-55czf"]
Sep 11 19:30:29.284181 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:30:29.284144    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88643ea7_21db_4b13_a376_ac84810bcf61.slice/crio-431cb38e664dfcc672b9995f12f6104624775d763e3b0df7f777567a731b7b28 WatchSource:0}: Error finding container 431cb38e664dfcc672b9995f12f6104624775d763e3b0df7f777567a731b7b28: Status 404 returned error can't find the container with id 431cb38e664dfcc672b9995f12f6104624775d763e3b0df7f777567a731b7b28
Sep 11 19:30:29.684930 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:29.684892    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-55czf" event={"ID":"88643ea7-21db-4b13-a376-ac84810bcf61","Type":"ContainerStarted","Data":"431cb38e664dfcc672b9995f12f6104624775d763e3b0df7f777567a731b7b28"}
Sep 11 19:30:31.694145 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:31.694109    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-55czf" event={"ID":"88643ea7-21db-4b13-a376-ac84810bcf61","Type":"ContainerStarted","Data":"9f8ec42e92635867c0bb16ce02b5ec2ac422332ee0861d840bc7a184cb04db9c"}
Sep 11 19:30:31.710202 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:31.710154    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-55czf" podStartSLOduration=2.23718652 podStartE2EDuration="3.710138118s" totalImagesPullingTime="1.472951598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:30:31.709247468 +0000 UTC m=+248.326416960" watchObservedRunningTime="2026-09-11 19:30:31.710138118 +0000 UTC m=+248.327307645"
Sep 11 19:30:34.060780 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.060750    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xmrpx"]
Sep 11 19:30:34.063800 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.063779    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:34.065774 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.065749    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 19:30:34.065897 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.065837    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 19:30:34.066422 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.066392    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 19:30:34.066575 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.066560    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 19:30:34.066652 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.066622    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s6r98\""
Sep 11 19:30:34.077150 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.077130    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xmrpx"]
Sep 11 19:30:34.176196 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.176164    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-data-volume\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:34.176310 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.176242    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:34.176310 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.176299    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:34.176387 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.176322    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqchz\" (UniqueName: \"kubernetes.io/projected/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-kube-api-access-lqchz\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:34.176420 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.176388    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-crio-socket\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:34.276647 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.276614    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-crio-socket\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:34.276896 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.276668    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-data-volume\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:34.276896 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.276696    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:34.276896 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.276744    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-crio-socket\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:34.276896 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.276747    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:34.276896 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.276791    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqchz\" (UniqueName: \"kubernetes.io/projected/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-kube-api-access-lqchz\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:34.276896 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:34.276813    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 19:30:34.276896 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:34.276876    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-kube-rbac-proxy-cm podName:55d96fb4-b74a-4fa1-9e45-36ac15f57e67 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:34.77685727 +0000 UTC m=+251.394026761 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmrpx" (UID: "55d96fb4-b74a-4fa1-9e45-36ac15f57e67") : configmap "kube-rbac-proxy" not found
Sep 11 19:30:34.277217 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:34.276912    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 19:30:34.277217 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:34.276984    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-insights-runtime-extractor-tls podName:55d96fb4-b74a-4fa1-9e45-36ac15f57e67 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:34.776965103 +0000 UTC m=+251.394134577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmrpx" (UID: "55d96fb4-b74a-4fa1-9e45-36ac15f57e67") : secret "insights-runtime-extractor-tls" not found
Sep 11 19:30:34.277217 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.277010    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-data-volume\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:34.284505 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.284485    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqchz\" (UniqueName: \"kubernetes.io/projected/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-kube-api-access-lqchz\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:34.780034 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.779983    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:34.780220 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:34.780084    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:34.780220 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:34.780130    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 19:30:34.780220 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:34.780158    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 19:30:34.780220 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:34.780207    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-insights-runtime-extractor-tls podName:55d96fb4-b74a-4fa1-9e45-36ac15f57e67 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:35.780191242 +0000 UTC m=+252.397360711 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmrpx" (UID: "55d96fb4-b74a-4fa1-9e45-36ac15f57e67") : secret "insights-runtime-extractor-tls" not found
Sep 11 19:30:34.780360 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:34.780223    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-kube-rbac-proxy-cm podName:55d96fb4-b74a-4fa1-9e45-36ac15f57e67 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:35.780217321 +0000 UTC m=+252.397386790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmrpx" (UID: "55d96fb4-b74a-4fa1-9e45-36ac15f57e67") : configmap "kube-rbac-proxy" not found
Sep 11 19:30:35.687945 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:35.687905    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs\") pod \"network-metrics-daemon-zzscf\" (UID: \"e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef\") " pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:30:35.688352 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:35.688025    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 19:30:35.688352 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:35.688075    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs podName:e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef nodeName:}" failed. No retries permitted until 2026-09-11 19:32:37.68806102 +0000 UTC m=+374.305230489 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs") pod "network-metrics-daemon-zzscf" (UID: "e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef") : secret "metrics-daemon-secret" not found
Sep 11 19:30:35.788252 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:35.788192    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:35.788433 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:35.788299    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:35.788433 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:35.788362    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 19:30:35.788433 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:35.788380    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 19:30:35.788530 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:35.788439    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-kube-rbac-proxy-cm podName:55d96fb4-b74a-4fa1-9e45-36ac15f57e67 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:37.788425159 +0000 UTC m=+254.405594634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmrpx" (UID: "55d96fb4-b74a-4fa1-9e45-36ac15f57e67") : configmap "kube-rbac-proxy" not found
Sep 11 19:30:35.788530 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:35.788454    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-insights-runtime-extractor-tls podName:55d96fb4-b74a-4fa1-9e45-36ac15f57e67 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:37.788447785 +0000 UTC m=+254.405617254 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmrpx" (UID: "55d96fb4-b74a-4fa1-9e45-36ac15f57e67") : secret "insights-runtime-extractor-tls" not found
Sep 11 19:30:36.209532 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:36.209510    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z8xdl_f7027ce0-9c8b-41f9-98c8-47905e740a83/dns-node-resolver/0.log"
Sep 11 19:30:36.809235 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:36.809210    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d5jh8_0b137eb7-4e92-44fe-882d-0ca19704ee2c/node-ca/0.log"
Sep 11 19:30:37.803593 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:37.803554    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:37.803818 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:37.803604    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:37.803818 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:37.803685    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 19:30:37.803818 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:37.803692    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 19:30:37.803818 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:37.803776    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-kube-rbac-proxy-cm podName:55d96fb4-b74a-4fa1-9e45-36ac15f57e67 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:41.803755839 +0000 UTC m=+258.420925313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xmrpx" (UID: "55d96fb4-b74a-4fa1-9e45-36ac15f57e67") : configmap "kube-rbac-proxy" not found
Sep 11 19:30:37.803818 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:37.803808    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-insights-runtime-extractor-tls podName:55d96fb4-b74a-4fa1-9e45-36ac15f57e67 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:41.803795304 +0000 UTC m=+258.420964773 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmrpx" (UID: "55d96fb4-b74a-4fa1-9e45-36ac15f57e67") : secret "insights-runtime-extractor-tls" not found
Sep 11 19:30:37.809943 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:37.809921    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nflsh_b32f7b7b-b959-4213-b66f-e1840988b22b/serve-healthcheck-canary/0.log"
Sep 11 19:30:41.833929 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:41.833893    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:41.833929 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:41.833937    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:41.834329 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:41.834026    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 19:30:41.834329 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:41.834072    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-insights-runtime-extractor-tls podName:55d96fb4-b74a-4fa1-9e45-36ac15f57e67 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:49.8340599 +0000 UTC m=+266.451229369 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xmrpx" (UID: "55d96fb4-b74a-4fa1-9e45-36ac15f57e67") : secret "insights-runtime-extractor-tls" not found
Sep 11 19:30:41.834447 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:41.834428    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:49.894725 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:49.894622    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:49.896805 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:49.896782    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/55d96fb4-b74a-4fa1-9e45-36ac15f57e67-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xmrpx\" (UID: \"55d96fb4-b74a-4fa1-9e45-36ac15f57e67\") " pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:49.972486 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:49.972460    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xmrpx"
Sep 11 19:30:50.241276 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:50.241247    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xmrpx"]
Sep 11 19:30:50.244162 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:30:50.244134    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d96fb4_b74a_4fa1_9e45_36ac15f57e67.slice/crio-bc32059a0a9dfd5dbd7197dd3a4456cd07d486a91acd552e51ffc2df18ed4f65 WatchSource:0}: Error finding container bc32059a0a9dfd5dbd7197dd3a4456cd07d486a91acd552e51ffc2df18ed4f65: Status 404 returned error can't find the container with id bc32059a0a9dfd5dbd7197dd3a4456cd07d486a91acd552e51ffc2df18ed4f65
Sep 11 19:30:50.751885 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:50.751854    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmrpx" event={"ID":"55d96fb4-b74a-4fa1-9e45-36ac15f57e67","Type":"ContainerStarted","Data":"a913d4c5cc07a242c5ad1c0aad9f71e171883bfd19d9786fb1f8badb8b65eee0"}
Sep 11 19:30:50.752041 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:50.751908    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmrpx" event={"ID":"55d96fb4-b74a-4fa1-9e45-36ac15f57e67","Type":"ContainerStarted","Data":"bc32059a0a9dfd5dbd7197dd3a4456cd07d486a91acd552e51ffc2df18ed4f65"}
Sep 11 19:30:51.757199 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:51.757155    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmrpx" event={"ID":"55d96fb4-b74a-4fa1-9e45-36ac15f57e67","Type":"ContainerStarted","Data":"7c8d1edb63c060a1bd6c715938aaa7b09b735a7bd636abb487ffee6941d63f5a"}
Sep 11 19:30:52.762479 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:52.762384    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xmrpx" event={"ID":"55d96fb4-b74a-4fa1-9e45-36ac15f57e67","Type":"ContainerStarted","Data":"a201512dbc72bc778bc2be5f9974554e97b19db5e6db0d91e12692b0f817aef8"}
Sep 11 19:30:52.778445 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:52.778393    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xmrpx" podStartSLOduration=16.622106213 podStartE2EDuration="18.778381497s" totalImagesPullingTime="2.156275284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:30:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:30:52.777507232 +0000 UTC m=+269.394676734" watchObservedRunningTime="2026-09-11 19:30:52.778381497 +0000 UTC m=+269.395550987"
Sep 11 19:30:54.941354 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:54.941320    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79cfc578bf-bccp4"]
Sep 11 19:30:54.941808 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:30:54.941562    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-79cfc578bf-bccp4" podUID="b3a98b87-6d7d-483b-b456-f9f80e1f2407"
Sep 11 19:30:54.977523 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:54.977479    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8dc4fc6b6-2l26s"]
Sep 11 19:30:54.979337 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:54.979321    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:54.994572 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:54.994549    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8dc4fc6b6-2l26s"]
Sep 11 19:30:55.133325 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.133282    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-installation-pull-secrets\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.133325 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.133324    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4xzg\" (UniqueName: \"kubernetes.io/projected/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-kube-api-access-q4xzg\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.133539 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.133349    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-registry-tls\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.133539 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.133375    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-trusted-ca\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.133539 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.133438    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-bound-sa-token\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.133539 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.133509    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-ca-trust-extracted\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.133699 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.133549    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-registry-certificates\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.133699 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.133566    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-image-registry-private-configuration\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.234502 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.234418    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-ca-trust-extracted\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.234502 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.234479    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-registry-certificates\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.234502 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.234498    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-image-registry-private-configuration\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.234782 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.234515    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-installation-pull-secrets\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.234782 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.234560    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4xzg\" (UniqueName: \"kubernetes.io/projected/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-kube-api-access-q4xzg\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.234782 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.234577    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-registry-tls\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.234782 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.234596    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-trusted-ca\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.234782 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.234617    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-bound-sa-token\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.234945 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.234921    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-ca-trust-extracted\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.235315 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.235290    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-registry-certificates\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.235492 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.235464    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-trusted-ca\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.237185 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.237156    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-installation-pull-secrets\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.237295 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.237224    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-registry-tls\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.237406 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.237385    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-image-registry-private-configuration\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.242443 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.242418    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-bound-sa-token\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.242694 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.242675    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4xzg\" (UniqueName: \"kubernetes.io/projected/ac79548d-9052-4c47-b8b0-5a9917c7e8e0-kube-api-access-q4xzg\") pod \"image-registry-8dc4fc6b6-2l26s\" (UID: \"ac79548d-9052-4c47-b8b0-5a9917c7e8e0\") " pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.289778 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.289753    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p67zb\""
Sep 11 19:30:55.298429 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.298408    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.568209 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.568131    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8dc4fc6b6-2l26s"]
Sep 11 19:30:55.572269 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:30:55.572240    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac79548d_9052_4c47_b8b0_5a9917c7e8e0.slice/crio-50098e4ed0d3f08543f4f2943adb255c9771af25332e0c4980b6d0f29d039feb WatchSource:0}: Error finding container 50098e4ed0d3f08543f4f2943adb255c9771af25332e0c4980b6d0f29d039feb: Status 404 returned error can't find the container with id 50098e4ed0d3f08543f4f2943adb255c9771af25332e0c4980b6d0f29d039feb
Sep 11 19:30:55.773296 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.773258    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s" event={"ID":"ac79548d-9052-4c47-b8b0-5a9917c7e8e0","Type":"ContainerStarted","Data":"cfa26ee8213e131b14180ce3498ecc3101bb959a0627418843df1ecfd82eb47a"}
Sep 11 19:30:55.773438 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.773340    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s" event={"ID":"ac79548d-9052-4c47-b8b0-5a9917c7e8e0","Type":"ContainerStarted","Data":"50098e4ed0d3f08543f4f2943adb255c9771af25332e0c4980b6d0f29d039feb"}
Sep 11 19:30:55.773438 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.773275    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:30:55.773438 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.773374    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:30:55.779982 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.779963    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:30:55.794019 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.793977    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s" podStartSLOduration=1.793964217 podStartE2EDuration="1.793964217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:30:55.793045654 +0000 UTC m=+272.410215137" watchObservedRunningTime="2026-09-11 19:30:55.793964217 +0000 UTC m=+272.411133726"
Sep 11 19:30:55.940409 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.940382    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-certificates\") pod \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") "
Sep 11 19:30:55.940570 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.940425    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3a98b87-6d7d-483b-b456-f9f80e1f2407-image-registry-private-configuration\") pod \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") "
Sep 11 19:30:55.940570 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.940463    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3a98b87-6d7d-483b-b456-f9f80e1f2407-trusted-ca\") pod \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") "
Sep 11 19:30:55.940742 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.940662    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3a98b87-6d7d-483b-b456-f9f80e1f2407-ca-trust-extracted\") pod \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") "
Sep 11 19:30:55.940742 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.940689    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b3a98b87-6d7d-483b-b456-f9f80e1f2407" (UID: "b3a98b87-6d7d-483b-b456-f9f80e1f2407"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:30:55.940742 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.940733    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4tk4\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-kube-api-access-q4tk4\") pod \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") "
Sep 11 19:30:55.940900 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.940821    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a98b87-6d7d-483b-b456-f9f80e1f2407-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b3a98b87-6d7d-483b-b456-f9f80e1f2407" (UID: "b3a98b87-6d7d-483b-b456-f9f80e1f2407"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:30:55.940900 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.940810    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-bound-sa-token\") pod \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") "
Sep 11 19:30:55.940900 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.940864    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3a98b87-6d7d-483b-b456-f9f80e1f2407-installation-pull-secrets\") pod \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\" (UID: \"b3a98b87-6d7d-483b-b456-f9f80e1f2407\") "
Sep 11 19:30:55.941057 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.940937    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3a98b87-6d7d-483b-b456-f9f80e1f2407-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b3a98b87-6d7d-483b-b456-f9f80e1f2407" (UID: "b3a98b87-6d7d-483b-b456-f9f80e1f2407"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:30:55.942194 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.942150    2558 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-certificates\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:30:55.942609 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.942197    2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b3a98b87-6d7d-483b-b456-f9f80e1f2407-trusted-ca\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:30:55.942609 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.942214    2558 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b3a98b87-6d7d-483b-b456-f9f80e1f2407-ca-trust-extracted\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:30:55.943064 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.943035    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3a98b87-6d7d-483b-b456-f9f80e1f2407-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b3a98b87-6d7d-483b-b456-f9f80e1f2407" (UID: "b3a98b87-6d7d-483b-b456-f9f80e1f2407"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:30:55.943166 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.943145    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b3a98b87-6d7d-483b-b456-f9f80e1f2407" (UID: "b3a98b87-6d7d-483b-b456-f9f80e1f2407"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:30:55.943204 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.943193    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-kube-api-access-q4tk4" (OuterVolumeSpecName: "kube-api-access-q4tk4") pod "b3a98b87-6d7d-483b-b456-f9f80e1f2407" (UID: "b3a98b87-6d7d-483b-b456-f9f80e1f2407"). InnerVolumeSpecName "kube-api-access-q4tk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:30:55.943530 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:55.943509    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3a98b87-6d7d-483b-b456-f9f80e1f2407-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b3a98b87-6d7d-483b-b456-f9f80e1f2407" (UID: "b3a98b87-6d7d-483b-b456-f9f80e1f2407"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:30:56.042925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:56.042890    2558 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-bound-sa-token\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:30:56.042925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:56.042921    2558 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b3a98b87-6d7d-483b-b456-f9f80e1f2407-installation-pull-secrets\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:30:56.043094 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:56.042937    2558 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b3a98b87-6d7d-483b-b456-f9f80e1f2407-image-registry-private-configuration\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:30:56.043094 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:56.042950    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4tk4\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-kube-api-access-q4tk4\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:30:56.776059 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:56.776028    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79cfc578bf-bccp4"
Sep 11 19:30:56.804257 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:56.804229    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-79cfc578bf-bccp4"]
Sep 11 19:30:56.807748 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:56.807723    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-79cfc578bf-bccp4"]
Sep 11 19:30:56.848622 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:56.848595    2558 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b3a98b87-6d7d-483b-b456-f9f80e1f2407-registry-tls\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:30:57.811415 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:30:57.811379    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3a98b87-6d7d-483b-b456-f9f80e1f2407" path="/var/lib/kubelet/pods/b3a98b87-6d7d-483b-b456-f9f80e1f2407/volumes"
Sep 11 19:31:03.498193 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:03.498160    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-s9wws"]
Sep 11 19:31:03.503174 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:03.503158    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-s9wws"
Sep 11 19:31:03.505481 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:03.505459    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-fg7w7\""
Sep 11 19:31:03.505566 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:03.505465    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 11 19:31:03.511352 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:03.511331    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-s9wws"]
Sep 11 19:31:03.592777 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:03.592742    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/870f9d8f-89f3-4184-b005-ddd6c7e06f6e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-s9wws\" (UID: \"870f9d8f-89f3-4184-b005-ddd6c7e06f6e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-s9wws"
Sep 11 19:31:03.693868 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:03.693837    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/870f9d8f-89f3-4184-b005-ddd6c7e06f6e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-s9wws\" (UID: \"870f9d8f-89f3-4184-b005-ddd6c7e06f6e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-s9wws"
Sep 11 19:31:03.694002 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:31:03.693975    2558 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 11 19:31:03.694042 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:31:03.694031    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/870f9d8f-89f3-4184-b005-ddd6c7e06f6e-tls-certificates podName:870f9d8f-89f3-4184-b005-ddd6c7e06f6e nodeName:}" failed. No retries permitted until 2026-09-11 19:31:04.194016529 +0000 UTC m=+280.811185997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/870f9d8f-89f3-4184-b005-ddd6c7e06f6e-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-s9wws" (UID: "870f9d8f-89f3-4184-b005-ddd6c7e06f6e") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 11 19:31:04.198002 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:04.197967    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/870f9d8f-89f3-4184-b005-ddd6c7e06f6e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-s9wws\" (UID: \"870f9d8f-89f3-4184-b005-ddd6c7e06f6e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-s9wws"
Sep 11 19:31:04.200375 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:04.200349    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/870f9d8f-89f3-4184-b005-ddd6c7e06f6e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-s9wws\" (UID: \"870f9d8f-89f3-4184-b005-ddd6c7e06f6e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-s9wws"
Sep 11 19:31:04.412700 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:04.412660    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-s9wws"
Sep 11 19:31:04.679688 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:04.679653    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-s9wws"]
Sep 11 19:31:04.682575 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:31:04.682551    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870f9d8f_89f3_4184_b005_ddd6c7e06f6e.slice/crio-ec73ce532020c0a0226a57909aab7429ea2e7bb98e6e0d0d76350878441ebaed WatchSource:0}: Error finding container ec73ce532020c0a0226a57909aab7429ea2e7bb98e6e0d0d76350878441ebaed: Status 404 returned error can't find the container with id ec73ce532020c0a0226a57909aab7429ea2e7bb98e6e0d0d76350878441ebaed
Sep 11 19:31:04.801154 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:04.801075    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-s9wws" event={"ID":"870f9d8f-89f3-4184-b005-ddd6c7e06f6e","Type":"ContainerStarted","Data":"ec73ce532020c0a0226a57909aab7429ea2e7bb98e6e0d0d76350878441ebaed"}
Sep 11 19:31:05.436366 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:31:05.436308    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-qq9hm" podUID="22079f6a-8b4b-46a7-a9c9-5656255a8144"
Sep 11 19:31:05.804343 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:05.804272    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:31:06.808602 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:06.808566    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-s9wws" event={"ID":"870f9d8f-89f3-4184-b005-ddd6c7e06f6e","Type":"ContainerStarted","Data":"e7bc1bd19eecf581ad517ac43adcd39bab553b4808751714dc47fbd837ed0501"}
Sep 11 19:31:06.809007 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:06.808781    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-s9wws"
Sep 11 19:31:06.813462 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:06.813436    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-s9wws"
Sep 11 19:31:06.830346 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:06.828590    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-s9wws" podStartSLOduration=2.602561699 podStartE2EDuration="3.828568632s" totalImagesPullingTime="1.226006933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:31:06.826250463 +0000 UTC m=+283.443419954" watchObservedRunningTime="2026-09-11 19:31:06.828568632 +0000 UTC m=+283.445738124"
Sep 11 19:31:07.552389 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.552356    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff"]
Sep 11 19:31:07.555437 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.555419    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff"
Sep 11 19:31:07.558443 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.558328    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 19:31:07.558443 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.558349    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 19:31:07.558443 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.558345    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lk6qt\""
Sep 11 19:31:07.558443 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.558420    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 19:31:07.558443 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.558423    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 19:31:07.558826 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.558653    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 19:31:07.564892 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.564865    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff"]
Sep 11 19:31:07.625206 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.625166    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cd4631c-9b63-41a2-9fd0-16e7a88bf834-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-gn2ff\" (UID: \"7cd4631c-9b63-41a2-9fd0-16e7a88bf834\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff"
Sep 11 19:31:07.625206 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.625209    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cd4631c-9b63-41a2-9fd0-16e7a88bf834-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-gn2ff\" (UID: \"7cd4631c-9b63-41a2-9fd0-16e7a88bf834\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff"
Sep 11 19:31:07.625403 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.625296    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m25zz\" (UniqueName: \"kubernetes.io/projected/7cd4631c-9b63-41a2-9fd0-16e7a88bf834-kube-api-access-m25zz\") pod \"prometheus-operator-68bcd7f987-gn2ff\" (UID: \"7cd4631c-9b63-41a2-9fd0-16e7a88bf834\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff"
Sep 11 19:31:07.625403 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.625360    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cd4631c-9b63-41a2-9fd0-16e7a88bf834-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-gn2ff\" (UID: \"7cd4631c-9b63-41a2-9fd0-16e7a88bf834\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff"
Sep 11 19:31:07.726035 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.725998    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cd4631c-9b63-41a2-9fd0-16e7a88bf834-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-gn2ff\" (UID: \"7cd4631c-9b63-41a2-9fd0-16e7a88bf834\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff"
Sep 11 19:31:07.726035 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.726037    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cd4631c-9b63-41a2-9fd0-16e7a88bf834-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-gn2ff\" (UID: \"7cd4631c-9b63-41a2-9fd0-16e7a88bf834\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff"
Sep 11 19:31:07.726288 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.726057    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cd4631c-9b63-41a2-9fd0-16e7a88bf834-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-gn2ff\" (UID: \"7cd4631c-9b63-41a2-9fd0-16e7a88bf834\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff"
Sep 11 19:31:07.726288 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.726092    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m25zz\" (UniqueName: \"kubernetes.io/projected/7cd4631c-9b63-41a2-9fd0-16e7a88bf834-kube-api-access-m25zz\") pod \"prometheus-operator-68bcd7f987-gn2ff\" (UID: \"7cd4631c-9b63-41a2-9fd0-16e7a88bf834\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff"
Sep 11 19:31:07.726778 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.726756    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cd4631c-9b63-41a2-9fd0-16e7a88bf834-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-gn2ff\" (UID: \"7cd4631c-9b63-41a2-9fd0-16e7a88bf834\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff"
Sep 11 19:31:07.728696 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.728668    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cd4631c-9b63-41a2-9fd0-16e7a88bf834-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-gn2ff\" (UID: \"7cd4631c-9b63-41a2-9fd0-16e7a88bf834\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff"
Sep 11 19:31:07.728797 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.728774    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7cd4631c-9b63-41a2-9fd0-16e7a88bf834-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-gn2ff\" (UID: \"7cd4631c-9b63-41a2-9fd0-16e7a88bf834\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff"
Sep 11 19:31:07.733856 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.733833    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m25zz\" (UniqueName: \"kubernetes.io/projected/7cd4631c-9b63-41a2-9fd0-16e7a88bf834-kube-api-access-m25zz\") pod \"prometheus-operator-68bcd7f987-gn2ff\" (UID: \"7cd4631c-9b63-41a2-9fd0-16e7a88bf834\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff"
Sep 11 19:31:07.865087 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:07.864986    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff"
Sep 11 19:31:08.133586 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:08.133506    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff"]
Sep 11 19:31:08.136473 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:31:08.136448    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd4631c_9b63_41a2_9fd0_16e7a88bf834.slice/crio-2abe6001249b98b1c968962a27598af43330355700552629ecc63a6813ff0c6e WatchSource:0}: Error finding container 2abe6001249b98b1c968962a27598af43330355700552629ecc63a6813ff0c6e: Status 404 returned error can't find the container with id 2abe6001249b98b1c968962a27598af43330355700552629ecc63a6813ff0c6e
Sep 11 19:31:08.731961 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:08.731923    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:31:08.734239 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:08.734216    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/22079f6a-8b4b-46a7-a9c9-5656255a8144-metrics-tls\") pod \"dns-default-qq9hm\" (UID: \"22079f6a-8b4b-46a7-a9c9-5656255a8144\") " pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:31:08.807368 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:08.807345    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x76hz\""
Sep 11 19:31:08.815314 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:08.815292    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:31:08.815894 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:08.815871    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff" event={"ID":"7cd4631c-9b63-41a2-9fd0-16e7a88bf834","Type":"ContainerStarted","Data":"2abe6001249b98b1c968962a27598af43330355700552629ecc63a6813ff0c6e"}
Sep 11 19:31:09.084365 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:09.084293    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qq9hm"]
Sep 11 19:31:09.087606 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:31:09.087582    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22079f6a_8b4b_46a7_a9c9_5656255a8144.slice/crio-f554ebb7a0c926d19419a76aaef6a50286582d8f5dbb8dab5d7a03f70c07bdc4 WatchSource:0}: Error finding container f554ebb7a0c926d19419a76aaef6a50286582d8f5dbb8dab5d7a03f70c07bdc4: Status 404 returned error can't find the container with id f554ebb7a0c926d19419a76aaef6a50286582d8f5dbb8dab5d7a03f70c07bdc4
Sep 11 19:31:09.823227 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:09.823196    2558 generic.go:356] "Generic (PLEG): container finished" podID="4122f5ec-feaf-4a04-be80-d8231f2013aa" containerID="17c584abe1bd0211293c5965697b69fadb9213e1f4e134a94e5d0a6d5c760688" exitCode=1
Sep 11 19:31:09.823392 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:09.823264    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj" event={"ID":"4122f5ec-feaf-4a04-be80-d8231f2013aa","Type":"ContainerDied","Data":"17c584abe1bd0211293c5965697b69fadb9213e1f4e134a94e5d0a6d5c760688"}
Sep 11 19:31:09.823392 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:09.823336    2558 scope.go:117] "RemoveContainer" containerID="b621192b66979af6474e06c32769a9e11206cb52ae917754df5d630cb5190383"
Sep 11 19:31:09.823572 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:09.823559    2558 scope.go:117] "RemoveContainer" containerID="17c584abe1bd0211293c5965697b69fadb9213e1f4e134a94e5d0a6d5c760688"
Sep 11 19:31:09.823755 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:31:09.823733    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-6d8b86cd98-dcwtj_open-cluster-management-agent-addon(4122f5ec-feaf-4a04-be80-d8231f2013aa)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj" podUID="4122f5ec-feaf-4a04-be80-d8231f2013aa"
Sep 11 19:31:09.825686 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:09.825644    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff" event={"ID":"7cd4631c-9b63-41a2-9fd0-16e7a88bf834","Type":"ContainerStarted","Data":"d62a2b5eff74c67ca68fab43db53d0dbf806cebc670896d33c696b47029943e2"}
Sep 11 19:31:09.825820 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:09.825700    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff" event={"ID":"7cd4631c-9b63-41a2-9fd0-16e7a88bf834","Type":"ContainerStarted","Data":"f4c5d770dacc2550a631b48ffc2d6c27bfde819ee2d58f57a5c8845265ec494c"}
Sep 11 19:31:09.826994 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:09.826969    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qq9hm" event={"ID":"22079f6a-8b4b-46a7-a9c9-5656255a8144","Type":"ContainerStarted","Data":"f554ebb7a0c926d19419a76aaef6a50286582d8f5dbb8dab5d7a03f70c07bdc4"}
Sep 11 19:31:09.863365 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:09.863322    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-gn2ff" podStartSLOduration=1.35640483 podStartE2EDuration="2.863308281s" totalImagesPullingTime="1.506903451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:31:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:31:09.862978665 +0000 UTC m=+286.480148158" watchObservedRunningTime="2026-09-11 19:31:09.863308281 +0000 UTC m=+286.480477771"
Sep 11 19:31:10.458562 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:10.458523    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"
Sep 11 19:31:10.718934 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:10.718855    2558 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"
Sep 11 19:31:10.832424 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:10.832398    2558 scope.go:117] "RemoveContainer" containerID="17c584abe1bd0211293c5965697b69fadb9213e1f4e134a94e5d0a6d5c760688"
Sep 11 19:31:10.832606 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:31:10.832584    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-6d8b86cd98-dcwtj_open-cluster-management-agent-addon(4122f5ec-feaf-4a04-be80-d8231f2013aa)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj" podUID="4122f5ec-feaf-4a04-be80-d8231f2013aa"
Sep 11 19:31:11.836795 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.836683    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qq9hm" event={"ID":"22079f6a-8b4b-46a7-a9c9-5656255a8144","Type":"ContainerStarted","Data":"1b0dcb63abb542c6a908dc7c8c0c20e1571a5b3e23d0cc80889cbfd2e8881c00"}
Sep 11 19:31:11.836795 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.836770    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:31:11.836795 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.836793    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qq9hm" event={"ID":"22079f6a-8b4b-46a7-a9c9-5656255a8144","Type":"ContainerStarted","Data":"a25f4db088921be14fed434e252bd389de03f6c1dcb13ee307991859a0705663"}
Sep 11 19:31:11.838669 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.838643    2558 generic.go:356] "Generic (PLEG): container finished" podID="6039f54e-4387-4ba3-979a-21ade90fee05" containerID="5c3f4f0547176dbcdc41ce89615a1a35d0ccba8b13f59049386ae684f540602c" exitCode=255
Sep 11 19:31:11.838791 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.838726    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9" event={"ID":"6039f54e-4387-4ba3-979a-21ade90fee05","Type":"ContainerDied","Data":"5c3f4f0547176dbcdc41ce89615a1a35d0ccba8b13f59049386ae684f540602c"}
Sep 11 19:31:11.838791 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.838768    2558 scope.go:117] "RemoveContainer" containerID="a8342ff154442ff765108774a562f5aeccf2c52b161eb2903c1cafe7360678dc"
Sep 11 19:31:11.839018 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.838991    2558 scope.go:117] "RemoveContainer" containerID="17c584abe1bd0211293c5965697b69fadb9213e1f4e134a94e5d0a6d5c760688"
Sep 11 19:31:11.839176 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:31:11.839154    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-6d8b86cd98-dcwtj_open-cluster-management-agent-addon(4122f5ec-feaf-4a04-be80-d8231f2013aa)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj" podUID="4122f5ec-feaf-4a04-be80-d8231f2013aa"
Sep 11 19:31:11.844803 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.844779    2558 scope.go:117] "RemoveContainer" containerID="5c3f4f0547176dbcdc41ce89615a1a35d0ccba8b13f59049386ae684f540602c"
Sep 11 19:31:11.844938 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:31:11.844919    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-6689948644-7q7k9_open-cluster-management-agent-addon(6039f54e-4387-4ba3-979a-21ade90fee05)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9" podUID="6039f54e-4387-4ba3-979a-21ade90fee05"
Sep 11 19:31:11.853762 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.853702    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qq9hm" podStartSLOduration=251.867434805 podStartE2EDuration="4m13.853683513s" totalImagesPullingTime="1.986248708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:31:11.853598667 +0000 UTC m=+288.470768159" watchObservedRunningTime="2026-09-11 19:31:11.853683513 +0000 UTC m=+288.470853006"
Sep 11 19:31:11.881650 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.881613    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72"]
Sep 11 19:31:11.885982 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.885963    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72"
Sep 11 19:31:11.887863 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.887837    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 11 19:31:11.887980 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.887875    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 11 19:31:11.887980 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.887893    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wnjht\""
Sep 11 19:31:11.893520 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.893490    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72"]
Sep 11 19:31:11.917161 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.917133    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-b4fxt"]
Sep 11 19:31:11.920388 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.920363    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:11.923477 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.923455    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 19:31:11.923574 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.923554    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 19:31:11.923910 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.923895    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 19:31:11.926144 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.926125    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nfn9q\""
Sep 11 19:31:11.953089 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.953047    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:11.953256 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.953094    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d094cc80-b4b7-4f94-bd57-cb159354ea85-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-mkv72\" (UID: \"d094cc80-b4b7-4f94-bd57-cb159354ea85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72"
Sep 11 19:31:11.953256 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.953128    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-root\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:11.953256 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.953158    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-metrics-client-ca\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:11.953256 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.953188    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdct4\" (UniqueName: \"kubernetes.io/projected/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-kube-api-access-vdct4\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:11.953256 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.953226    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-node-exporter-textfile\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:11.953437 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.953261    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-node-exporter-accelerators-collector-config\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:11.953437 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.953326    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-sys\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:11.953437 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.953371    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-node-exporter-wtmp\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:11.953437 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.953430    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d094cc80-b4b7-4f94-bd57-cb159354ea85-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-mkv72\" (UID: \"d094cc80-b4b7-4f94-bd57-cb159354ea85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72"
Sep 11 19:31:11.953634 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.953501    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgglw\" (UniqueName: \"kubernetes.io/projected/d094cc80-b4b7-4f94-bd57-cb159354ea85-kube-api-access-vgglw\") pod \"openshift-state-metrics-67dcf8c979-mkv72\" (UID: \"d094cc80-b4b7-4f94-bd57-cb159354ea85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72"
Sep 11 19:31:11.953634 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.953588    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d094cc80-b4b7-4f94-bd57-cb159354ea85-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-mkv72\" (UID: \"d094cc80-b4b7-4f94-bd57-cb159354ea85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72"
Sep 11 19:31:11.953757 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:11.953642    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-node-exporter-tls\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:12.054619 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.054581    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-sys\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:12.054619 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.054623    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-node-exporter-wtmp\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:12.054899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.054643    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d094cc80-b4b7-4f94-bd57-cb159354ea85-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-mkv72\" (UID: \"d094cc80-b4b7-4f94-bd57-cb159354ea85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72"
Sep 11 19:31:12.054899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.054702    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-sys\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:12.054899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.054769    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgglw\" (UniqueName: \"kubernetes.io/projected/d094cc80-b4b7-4f94-bd57-cb159354ea85-kube-api-access-vgglw\") pod \"openshift-state-metrics-67dcf8c979-mkv72\" (UID: \"d094cc80-b4b7-4f94-bd57-cb159354ea85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72"
Sep 11 19:31:12.054899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.054800    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-node-exporter-wtmp\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:12.054899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.054816    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d094cc80-b4b7-4f94-bd57-cb159354ea85-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-mkv72\" (UID: \"d094cc80-b4b7-4f94-bd57-cb159354ea85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72"
Sep 11 19:31:12.054899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.054847    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-node-exporter-tls\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:12.054899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.054881    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:12.055253 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.054914    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d094cc80-b4b7-4f94-bd57-cb159354ea85-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-mkv72\" (UID: \"d094cc80-b4b7-4f94-bd57-cb159354ea85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72"
Sep 11 19:31:12.055253 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.054946    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-root\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:12.055253 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:31:12.054951    2558 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 11 19:31:12.055253 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.054975    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-metrics-client-ca\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:12.055253 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:31:12.055017    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d094cc80-b4b7-4f94-bd57-cb159354ea85-openshift-state-metrics-tls podName:d094cc80-b4b7-4f94-bd57-cb159354ea85 nodeName:}" failed. No retries permitted until 2026-09-11 19:31:12.554995731 +0000 UTC m=+289.172165206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d094cc80-b4b7-4f94-bd57-cb159354ea85-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-mkv72" (UID: "d094cc80-b4b7-4f94-bd57-cb159354ea85") : secret "openshift-state-metrics-tls" not found
Sep 11 19:31:12.055253 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.055038    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdct4\" (UniqueName: \"kubernetes.io/projected/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-kube-api-access-vdct4\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:12.055253 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.055081    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-node-exporter-textfile\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:12.055253 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.055116    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-node-exporter-accelerators-collector-config\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:12.055253 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.055158    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-root\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:12.055690 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.055580    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-node-exporter-textfile\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:12.055690 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.055589    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-metrics-client-ca\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:12.055690 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.055612    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-node-exporter-accelerators-collector-config\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:12.055959 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.055938    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d094cc80-b4b7-4f94-bd57-cb159354ea85-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-mkv72\" (UID: \"d094cc80-b4b7-4f94-bd57-cb159354ea85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72"
Sep 11 19:31:12.057303 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.057280    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d094cc80-b4b7-4f94-bd57-cb159354ea85-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-mkv72\" (UID: \"d094cc80-b4b7-4f94-bd57-cb159354ea85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72"
Sep 11 19:31:12.057592 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.057575    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-node-exporter-tls\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:12.057677 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.057657    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:12.062878 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.062850    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdct4\" (UniqueName: \"kubernetes.io/projected/5bd3b61e-bae5-4fbb-a921-ffd2f6c74675-kube-api-access-vdct4\") pod \"node-exporter-b4fxt\" (UID: \"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675\") " pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:12.062956 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.062885    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgglw\" (UniqueName: \"kubernetes.io/projected/d094cc80-b4b7-4f94-bd57-cb159354ea85-kube-api-access-vgglw\") pod \"openshift-state-metrics-67dcf8c979-mkv72\" (UID: \"d094cc80-b4b7-4f94-bd57-cb159354ea85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72"
Sep 11 19:31:12.229178 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.229142    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b4fxt"
Sep 11 19:31:12.240298 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:31:12.240266    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd3b61e_bae5_4fbb_a921_ffd2f6c74675.slice/crio-3b573a76c358d02ae63573d4970f25253d4e2b3880b8782e6a1abc56e7fd93b4 WatchSource:0}: Error finding container 3b573a76c358d02ae63573d4970f25253d4e2b3880b8782e6a1abc56e7fd93b4: Status 404 returned error can't find the container with id 3b573a76c358d02ae63573d4970f25253d4e2b3880b8782e6a1abc56e7fd93b4
Sep 11 19:31:12.558799 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.558677    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d094cc80-b4b7-4f94-bd57-cb159354ea85-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-mkv72\" (UID: \"d094cc80-b4b7-4f94-bd57-cb159354ea85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72"
Sep 11 19:31:12.560991 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.560969    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d094cc80-b4b7-4f94-bd57-cb159354ea85-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-mkv72\" (UID: \"d094cc80-b4b7-4f94-bd57-cb159354ea85\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72"
Sep 11 19:31:12.812061 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.811978    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72"
Sep 11 19:31:12.848871 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:12.848834    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b4fxt" event={"ID":"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675","Type":"ContainerStarted","Data":"3b573a76c358d02ae63573d4970f25253d4e2b3880b8782e6a1abc56e7fd93b4"}
Sep 11 19:31:13.095167 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:13.095098    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72"]
Sep 11 19:31:13.284334 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:31:13.284286    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd094cc80_b4b7_4f94_bd57_cb159354ea85.slice/crio-5d56ab7f49a6174f7d5174fa43da69beab4a0980d87fac9984172ec42a1aee87 WatchSource:0}: Error finding container 5d56ab7f49a6174f7d5174fa43da69beab4a0980d87fac9984172ec42a1aee87: Status 404 returned error can't find the container with id 5d56ab7f49a6174f7d5174fa43da69beab4a0980d87fac9984172ec42a1aee87
Sep 11 19:31:13.855526 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:13.855490    2558 generic.go:356] "Generic (PLEG): container finished" podID="5bd3b61e-bae5-4fbb-a921-ffd2f6c74675" containerID="4de7858d2df5d8f0ce74be11a399c56b718d2fd76765a570ef2ae089723b00d7" exitCode=0
Sep 11 19:31:13.855963 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:13.855569    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b4fxt" event={"ID":"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675","Type":"ContainerDied","Data":"4de7858d2df5d8f0ce74be11a399c56b718d2fd76765a570ef2ae089723b00d7"}
Sep 11 19:31:13.857363 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:13.857341    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72" event={"ID":"d094cc80-b4b7-4f94-bd57-cb159354ea85","Type":"ContainerStarted","Data":"cb22a1f1634a0ff05ccc12a76f4e99660bd57129318c9f675be71dd749941f2c"}
Sep 11 19:31:13.857448 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:13.857384    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72" event={"ID":"d094cc80-b4b7-4f94-bd57-cb159354ea85","Type":"ContainerStarted","Data":"8d18da1eece717b15c1b8c46e3d3e14717b666e2140347046fa5b407aa9e0270"}
Sep 11 19:31:13.857448 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:13.857401    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72" event={"ID":"d094cc80-b4b7-4f94-bd57-cb159354ea85","Type":"ContainerStarted","Data":"5d56ab7f49a6174f7d5174fa43da69beab4a0980d87fac9984172ec42a1aee87"}
Sep 11 19:31:14.863533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:14.863441    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b4fxt" event={"ID":"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675","Type":"ContainerStarted","Data":"16675544aefe2f77bf798fc9981263f4d2a7ce294b5e03e745ca51e2601c3abc"}
Sep 11 19:31:14.863533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:14.863504    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b4fxt" event={"ID":"5bd3b61e-bae5-4fbb-a921-ffd2f6c74675","Type":"ContainerStarted","Data":"a15d12193d4f517f7dd8bf46cb7601e082db54fe115d570af829a9167bad7435"}
Sep 11 19:31:14.865504 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:14.865480    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72" event={"ID":"d094cc80-b4b7-4f94-bd57-cb159354ea85","Type":"ContainerStarted","Data":"64b1bb2de3aa8a7b68330ab178049255ff8b9f3cab8c7e9577406ed2ab062939"}
Sep 11 19:31:14.883317 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:14.883262    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-b4fxt" podStartSLOduration=2.732607656 podStartE2EDuration="3.883247694s" totalImagesPullingTime="1.087819788s" totalInitContainerRuntime="62.82025ms" isStatefulPod=true podCreationTimestamp="2026-09-11 19:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:31:14.881860702 +0000 UTC m=+291.499030192" watchObservedRunningTime="2026-09-11 19:31:14.883247694 +0000 UTC m=+291.500417179"
Sep 11 19:31:14.906774 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:14.906728    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-mkv72" podStartSLOduration=2.859050577 podStartE2EDuration="3.90669485s" totalImagesPullingTime="1.047644273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:31:14.905558053 +0000 UTC m=+291.522727543" watchObservedRunningTime="2026-09-11 19:31:14.90669485 +0000 UTC m=+291.523864340"
Sep 11 19:31:16.780425 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:16.780398    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8dc4fc6b6-2l26s"
Sep 11 19:31:17.808783 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:31:17.808738    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-zzscf" podUID="e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef"
Sep 11 19:31:18.124394 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.124313    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 19:31:18.127572 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.127547    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.130922 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.130728    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7tun6f5d38rkr\""
Sep 11 19:31:18.131040 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.131023    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 19:31:18.131241 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.131225    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 19:31:18.131718 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.131685    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 19:31:18.131858 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.131837    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 19:31:18.131960 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.131858    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 19:31:18.131960 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.131868    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-82bfd\""
Sep 11 19:31:18.131960 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.131927    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 19:31:18.131960 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.131790    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 19:31:18.132155 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.131823    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 19:31:18.132155 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.131980    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 19:31:18.132155 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.131730    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 19:31:18.132155 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.131809    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 19:31:18.132155 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.132110    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 19:31:18.132155 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.132148    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 19:31:18.132542 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.132247    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 19:31:18.133664 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.133647    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 19:31:18.151024 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.150994    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 19:31:18.203068 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.203033    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.203208 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.203074    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.203208 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.203106    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.203208 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.203130    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.203208 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.203155    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.203208 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.203188    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-config\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.203412 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.203251    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.203412 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.203278    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.203412 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.203299    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.203412 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.203319    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-web-config\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.203412 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.203397    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7gpb\" (UniqueName: \"kubernetes.io/projected/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-kube-api-access-t7gpb\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.203647 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.203442    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.203647 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.203490    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.203647 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.203541    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.203647 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.203570    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.203647 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.203599    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.203647 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.203638    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.203867 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.203659    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.203867 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.203675    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.203867 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.203690    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-config-out\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.304601 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.304551    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.304770 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.304606    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.304770 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.304635    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-web-config\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.304770 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.304668    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7gpb\" (UniqueName: \"kubernetes.io/projected/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-kube-api-access-t7gpb\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.304770 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.304699    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.304770 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.304746    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.305038 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.304782    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.305038 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.304810    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.305038 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.304839    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.305038 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.304870    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.305038 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.304898    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.305038 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.304926    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.305038 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.304951    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-config-out\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.305038 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.305024    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.305927 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.305838    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.305927 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.305870    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.307817 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.306786    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.307817 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.306784    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.307817 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.307332    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.307817 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.306816    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.307817 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.307427    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.307817 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.307454    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.307817 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.307497    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-config\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.307817 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.307513    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.307817 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.307531    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.308367 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.308008    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.308367 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.308082    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.308367 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.308157    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.308518 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.308378    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-web-config\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.309484 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.308650    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.311120 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.311094    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.311219 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.311155    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-config-out\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.311594 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.311571    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.311679 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.311567    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.312033 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.312012    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.312103 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.312036    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.312952 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.312911    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-config\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.313285 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.313261    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.313635 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.313610    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.315024 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.314998    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7gpb\" (UniqueName: \"kubernetes.io/projected/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-kube-api-access-t7gpb\") pod \"prometheus-k8s-0\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.439959 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.439919    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:18.729859 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.729837    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 19:31:18.732144 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:31:18.732118    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3cefa7a_c8b4_4d19_8253_1150614c4e6e.slice/crio-f4656a28466eb03e823d438611a434e92744fbfacc7959ff6a0bea08a4f1d133 WatchSource:0}: Error finding container f4656a28466eb03e823d438611a434e92744fbfacc7959ff6a0bea08a4f1d133: Status 404 returned error can't find the container with id f4656a28466eb03e823d438611a434e92744fbfacc7959ff6a0bea08a4f1d133
Sep 11 19:31:18.879345 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:18.879313    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3cefa7a-c8b4-4d19-8253-1150614c4e6e","Type":"ContainerStarted","Data":"f4656a28466eb03e823d438611a434e92744fbfacc7959ff6a0bea08a4f1d133"}
Sep 11 19:31:19.883972 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:19.883938    2558 generic.go:356] "Generic (PLEG): container finished" podID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerID="3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0" exitCode=0
Sep 11 19:31:19.884306 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:19.883990    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3cefa7a-c8b4-4d19-8253-1150614c4e6e","Type":"ContainerDied","Data":"3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0"}
Sep 11 19:31:20.707425 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:20.707389    2558 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9"
Sep 11 19:31:20.714691 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:20.714665    2558 scope.go:117] "RemoveContainer" containerID="5c3f4f0547176dbcdc41ce89615a1a35d0ccba8b13f59049386ae684f540602c"
Sep 11 19:31:20.714922 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:31:20.714899    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"addon-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=addon-agent pod=managed-serviceaccount-addon-agent-6689948644-7q7k9_open-cluster-management-agent-addon(6039f54e-4387-4ba3-979a-21ade90fee05)\"" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9" podUID="6039f54e-4387-4ba3-979a-21ade90fee05"
Sep 11 19:31:21.854771 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:21.854735    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qq9hm"
Sep 11 19:31:22.899100 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:22.899068    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3cefa7a-c8b4-4d19-8253-1150614c4e6e","Type":"ContainerStarted","Data":"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e"}
Sep 11 19:31:23.736748 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:23.736701    2558 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 19:31:23.905216 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:23.905066    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3cefa7a-c8b4-4d19-8253-1150614c4e6e","Type":"ContainerStarted","Data":"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766"}
Sep 11 19:31:24.912630 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:24.912588    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3cefa7a-c8b4-4d19-8253-1150614c4e6e","Type":"ContainerStarted","Data":"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204"}
Sep 11 19:31:25.921212 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:25.920154    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3cefa7a-c8b4-4d19-8253-1150614c4e6e","Type":"ContainerStarted","Data":"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43"}
Sep 11 19:31:25.921212 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:25.920239    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3cefa7a-c8b4-4d19-8253-1150614c4e6e","Type":"ContainerStarted","Data":"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673"}
Sep 11 19:31:25.921212 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:25.920278    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3cefa7a-c8b4-4d19-8253-1150614c4e6e","Type":"ContainerStarted","Data":"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff"}
Sep 11 19:31:25.953442 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:25.953373    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.107325798 podStartE2EDuration="7.953316132s" totalImagesPullingTime="5.823008672s" totalInitContainerRuntime="22.981662ms" isStatefulPod=false podCreationTimestamp="2026-09-11 19:31:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:31:25.952178832 +0000 UTC m=+302.569348347" watchObservedRunningTime="2026-09-11 19:31:25.953316132 +0000 UTC m=+302.570485624"
Sep 11 19:31:26.808205 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:26.808179    2558 scope.go:117] "RemoveContainer" containerID="17c584abe1bd0211293c5965697b69fadb9213e1f4e134a94e5d0a6d5c760688"
Sep 11 19:31:26.926389 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:26.926353    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj" event={"ID":"4122f5ec-feaf-4a04-be80-d8231f2013aa","Type":"ContainerStarted","Data":"a9d714da147489b1bcb0c5c7927d9319f7795d4adc582cbf9cb74861f40632cd"}
Sep 11 19:31:26.926979 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:26.926953    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"
Sep 11 19:31:26.928470 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:26.928448    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d8b86cd98-dcwtj"
Sep 11 19:31:28.440454 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:28.440421    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:31:29.807624 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:29.807583    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:31:33.819550 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:33.819521    2558 scope.go:117] "RemoveContainer" containerID="5c3f4f0547176dbcdc41ce89615a1a35d0ccba8b13f59049386ae684f540602c"
Sep 11 19:31:33.952551 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:31:33.952514    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6689948644-7q7k9" event={"ID":"6039f54e-4387-4ba3-979a-21ade90fee05","Type":"ContainerStarted","Data":"0e2ecc633bc8a79e4ce1a29bf2cf0458c34cc41b1ca032821ec76880a9fe986c"}
Sep 11 19:32:18.440335 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:18.440235    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:18.459336 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:18.459309    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:19.129477 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:19.129447    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:36.468342 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.468302    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 19:32:36.468851 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.468683    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="prometheus" containerID="cri-o://154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e" gracePeriod=600
Sep 11 19:32:36.468851 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.468720    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="kube-rbac-proxy" containerID="cri-o://a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673" gracePeriod=600
Sep 11 19:32:36.468851 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.468720    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="kube-rbac-proxy-web" containerID="cri-o://e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff" gracePeriod=600
Sep 11 19:32:36.468851 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.468740    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="thanos-sidecar" containerID="cri-o://a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204" gracePeriod=600
Sep 11 19:32:36.468851 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.468767    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="config-reloader" containerID="cri-o://96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766" gracePeriod=600
Sep 11 19:32:36.468851 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.468795    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="kube-rbac-proxy-thanos" containerID="cri-o://fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43" gracePeriod=600
Sep 11 19:32:36.711574 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.711544    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:36.845084 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845004    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-rulefiles-0\") pod \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") "
Sep 11 19:32:36.845084 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845042    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") "
Sep 11 19:32:36.845084 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845062    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7gpb\" (UniqueName: \"kubernetes.io/projected/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-kube-api-access-t7gpb\") pod \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") "
Sep 11 19:32:36.845084 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845077    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-trusted-ca-bundle\") pod \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") "
Sep 11 19:32:36.845388 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845104    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-configmap-serving-certs-ca-bundle\") pod \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") "
Sep 11 19:32:36.845388 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845224    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-config-out\") pod \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") "
Sep 11 19:32:36.845388 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845271    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-tls-assets\") pod \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") "
Sep 11 19:32:36.845388 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845305    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-grpc-tls\") pod \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") "
Sep 11 19:32:36.845388 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845338    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-configmap-metrics-client-ca\") pod \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") "
Sep 11 19:32:36.845388 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845380    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-metrics-client-certs\") pod \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") "
Sep 11 19:32:36.845692 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845406    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-prometheus-k8s-tls\") pod \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") "
Sep 11 19:32:36.845692 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845439    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-rulefiles-2\") pod \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") "
Sep 11 19:32:36.845692 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845453    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "e3cefa7a-c8b4-4d19-8253-1150614c4e6e" (UID: "e3cefa7a-c8b4-4d19-8253-1150614c4e6e"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:32:36.845692 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845487    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") "
Sep 11 19:32:36.845692 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845515    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-rulefiles-1\") pod \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") "
Sep 11 19:32:36.845692 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845518    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "e3cefa7a-c8b4-4d19-8253-1150614c4e6e" (UID: "e3cefa7a-c8b4-4d19-8253-1150614c4e6e"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:32:36.845692 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845549    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-kube-rbac-proxy\") pod \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") "
Sep 11 19:32:36.846064 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845803    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "e3cefa7a-c8b4-4d19-8253-1150614c4e6e" (UID: "e3cefa7a-c8b4-4d19-8253-1150614c4e6e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:32:36.846064 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845839    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "e3cefa7a-c8b4-4d19-8253-1150614c4e6e" (UID: "e3cefa7a-c8b4-4d19-8253-1150614c4e6e"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:32:36.846064 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845846    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-db\") pod \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") "
Sep 11 19:32:36.846064 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845909    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-web-config\") pod \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") "
Sep 11 19:32:36.846064 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845965    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-thanos-prometheus-http-client-file\") pod \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") "
Sep 11 19:32:36.846064 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.845994    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-config\") pod \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") "
Sep 11 19:32:36.846064 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.846026    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-configmap-kubelet-serving-ca-bundle\") pod \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\" (UID: \"e3cefa7a-c8b4-4d19-8253-1150614c4e6e\") "
Sep 11 19:32:36.846414 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.846272    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:32:36.846414 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.846290    2558 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:32:36.846414 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.846303    2558 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-configmap-metrics-client-ca\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:32:36.846414 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.846319    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:32:36.846624 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.846422    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "e3cefa7a-c8b4-4d19-8253-1150614c4e6e" (UID: "e3cefa7a-c8b4-4d19-8253-1150614c4e6e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:32:36.846624 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.846613    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "e3cefa7a-c8b4-4d19-8253-1150614c4e6e" (UID: "e3cefa7a-c8b4-4d19-8253-1150614c4e6e"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:32:36.847416 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.847086    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "e3cefa7a-c8b4-4d19-8253-1150614c4e6e" (UID: "e3cefa7a-c8b4-4d19-8253-1150614c4e6e"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:32:36.847416 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.847184    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "e3cefa7a-c8b4-4d19-8253-1150614c4e6e" (UID: "e3cefa7a-c8b4-4d19-8253-1150614c4e6e"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:32:36.849467 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.849436    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "e3cefa7a-c8b4-4d19-8253-1150614c4e6e" (UID: "e3cefa7a-c8b4-4d19-8253-1150614c4e6e"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:32:36.849582 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.849468    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e3cefa7a-c8b4-4d19-8253-1150614c4e6e" (UID: "e3cefa7a-c8b4-4d19-8253-1150614c4e6e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:32:36.849651 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.849625    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "e3cefa7a-c8b4-4d19-8253-1150614c4e6e" (UID: "e3cefa7a-c8b4-4d19-8253-1150614c4e6e"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:32:36.849651 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.849625    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-kube-api-access-t7gpb" (OuterVolumeSpecName: "kube-api-access-t7gpb") pod "e3cefa7a-c8b4-4d19-8253-1150614c4e6e" (UID: "e3cefa7a-c8b4-4d19-8253-1150614c4e6e"). InnerVolumeSpecName "kube-api-access-t7gpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:32:36.849780 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.849730    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "e3cefa7a-c8b4-4d19-8253-1150614c4e6e" (UID: "e3cefa7a-c8b4-4d19-8253-1150614c4e6e"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:32:36.849983 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.849959    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-config-out" (OuterVolumeSpecName: "config-out") pod "e3cefa7a-c8b4-4d19-8253-1150614c4e6e" (UID: "e3cefa7a-c8b4-4d19-8253-1150614c4e6e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:32:36.850232 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.850205    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "e3cefa7a-c8b4-4d19-8253-1150614c4e6e" (UID: "e3cefa7a-c8b4-4d19-8253-1150614c4e6e"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:32:36.850565 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.850545    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-config" (OuterVolumeSpecName: "config") pod "e3cefa7a-c8b4-4d19-8253-1150614c4e6e" (UID: "e3cefa7a-c8b4-4d19-8253-1150614c4e6e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:32:36.850643 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.850567    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "e3cefa7a-c8b4-4d19-8253-1150614c4e6e" (UID: "e3cefa7a-c8b4-4d19-8253-1150614c4e6e"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:32:36.850643 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.850599    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "e3cefa7a-c8b4-4d19-8253-1150614c4e6e" (UID: "e3cefa7a-c8b4-4d19-8253-1150614c4e6e"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:32:36.851073 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.851054    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "e3cefa7a-c8b4-4d19-8253-1150614c4e6e" (UID: "e3cefa7a-c8b4-4d19-8253-1150614c4e6e"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:32:36.863597 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.863386    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-web-config" (OuterVolumeSpecName: "web-config") pod "e3cefa7a-c8b4-4d19-8253-1150614c4e6e" (UID: "e3cefa7a-c8b4-4d19-8253-1150614c4e6e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:32:36.947680 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.947641    2558 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:32:36.947680 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.947671    2558 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-config\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:32:36.947680 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.947683    2558 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:32:36.947921 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.947694    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:32:36.947921 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.947731    2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:32:36.947921 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.947742    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7gpb\" (UniqueName: \"kubernetes.io/projected/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-kube-api-access-t7gpb\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:32:36.947921 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.947750    2558 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-config-out\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:32:36.947921 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.947758    2558 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-tls-assets\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:32:36.947921 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.947765    2558 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-grpc-tls\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:32:36.947921 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.947773    2558 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-metrics-client-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:32:36.947921 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.947781    2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:32:36.947921 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.947790    2558 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:32:36.947921 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.947798    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:32:36.947921 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.947806    2558 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-secret-kube-rbac-proxy\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:32:36.947921 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.947813    2558 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-prometheus-k8s-db\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:32:36.947921 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:36.947821    2558 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e3cefa7a-c8b4-4d19-8253-1150614c4e6e-web-config\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:32:37.186821 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.186788    2558 generic.go:356] "Generic (PLEG): container finished" podID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerID="fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43" exitCode=0
Sep 11 19:32:37.186821 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.186816    2558 generic.go:356] "Generic (PLEG): container finished" podID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerID="a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673" exitCode=0
Sep 11 19:32:37.186821 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.186822    2558 generic.go:356] "Generic (PLEG): container finished" podID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerID="e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff" exitCode=0
Sep 11 19:32:37.186821 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.186829    2558 generic.go:356] "Generic (PLEG): container finished" podID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerID="a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204" exitCode=0
Sep 11 19:32:37.187063 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.186838    2558 generic.go:356] "Generic (PLEG): container finished" podID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerID="96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766" exitCode=0
Sep 11 19:32:37.187063 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.186846    2558 generic.go:356] "Generic (PLEG): container finished" podID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerID="154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e" exitCode=0
Sep 11 19:32:37.187063 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.186881    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.187063 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.186874    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3cefa7a-c8b4-4d19-8253-1150614c4e6e","Type":"ContainerDied","Data":"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43"}
Sep 11 19:32:37.187063 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.187018    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3cefa7a-c8b4-4d19-8253-1150614c4e6e","Type":"ContainerDied","Data":"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673"}
Sep 11 19:32:37.187239 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.187064    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3cefa7a-c8b4-4d19-8253-1150614c4e6e","Type":"ContainerDied","Data":"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff"}
Sep 11 19:32:37.187239 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.187079    2558 scope.go:117] "RemoveContainer" containerID="fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43"
Sep 11 19:32:37.187239 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.187107    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3cefa7a-c8b4-4d19-8253-1150614c4e6e","Type":"ContainerDied","Data":"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204"}
Sep 11 19:32:37.187239 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.187149    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3cefa7a-c8b4-4d19-8253-1150614c4e6e","Type":"ContainerDied","Data":"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766"}
Sep 11 19:32:37.187239 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.187190    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3cefa7a-c8b4-4d19-8253-1150614c4e6e","Type":"ContainerDied","Data":"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e"}
Sep 11 19:32:37.187239 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.187229    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e3cefa7a-c8b4-4d19-8253-1150614c4e6e","Type":"ContainerDied","Data":"f4656a28466eb03e823d438611a434e92744fbfacc7959ff6a0bea08a4f1d133"}
Sep 11 19:32:37.195574 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.195545    2558 scope.go:117] "RemoveContainer" containerID="a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673"
Sep 11 19:32:37.202737 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.202691    2558 scope.go:117] "RemoveContainer" containerID="e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff"
Sep 11 19:32:37.209854 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.209835    2558 scope.go:117] "RemoveContainer" containerID="a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204"
Sep 11 19:32:37.212231 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.212208    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 19:32:37.215975 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.215950    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 19:32:37.217792 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.217776    2558 scope.go:117] "RemoveContainer" containerID="96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766"
Sep 11 19:32:37.224607 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.224584    2558 scope.go:117] "RemoveContainer" containerID="154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e"
Sep 11 19:32:37.231923 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.231903    2558 scope.go:117] "RemoveContainer" containerID="3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0"
Sep 11 19:32:37.238570 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.238548    2558 scope.go:117] "RemoveContainer" containerID="fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43"
Sep 11 19:32:37.238988 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:32:37.238961    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43\": container with ID starting with fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43 not found: ID does not exist" containerID="fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43"
Sep 11 19:32:37.239092 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.238995    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43"} err="failed to get container status \"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43\": rpc error: code = NotFound desc = could not find container \"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43\": container with ID starting with fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43 not found: ID does not exist"
Sep 11 19:32:37.239092 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.239031    2558 scope.go:117] "RemoveContainer" containerID="a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673"
Sep 11 19:32:37.239407 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:32:37.239382    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673\": container with ID starting with a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673 not found: ID does not exist" containerID="a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673"
Sep 11 19:32:37.239504 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.239411    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673"} err="failed to get container status \"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673\": rpc error: code = NotFound desc = could not find container \"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673\": container with ID starting with a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673 not found: ID does not exist"
Sep 11 19:32:37.239504 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.239431    2558 scope.go:117] "RemoveContainer" containerID="e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff"
Sep 11 19:32:37.239895 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:32:37.239729    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff\": container with ID starting with e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff not found: ID does not exist" containerID="e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff"
Sep 11 19:32:37.239895 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.239748    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff"} err="failed to get container status \"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff\": rpc error: code = NotFound desc = could not find container \"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff\": container with ID starting with e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff not found: ID does not exist"
Sep 11 19:32:37.239895 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.239760    2558 scope.go:117] "RemoveContainer" containerID="a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204"
Sep 11 19:32:37.240078 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:32:37.240023    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204\": container with ID starting with a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204 not found: ID does not exist" containerID="a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204"
Sep 11 19:32:37.240078 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.240056    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204"} err="failed to get container status \"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204\": rpc error: code = NotFound desc = could not find container \"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204\": container with ID starting with a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204 not found: ID does not exist"
Sep 11 19:32:37.240078 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.240071    2558 scope.go:117] "RemoveContainer" containerID="96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766"
Sep 11 19:32:37.240334 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:32:37.240316    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766\": container with ID starting with 96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766 not found: ID does not exist" containerID="96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766"
Sep 11 19:32:37.240389 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.240335    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766"} err="failed to get container status \"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766\": rpc error: code = NotFound desc = could not find container \"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766\": container with ID starting with 96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766 not found: ID does not exist"
Sep 11 19:32:37.240389 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.240348    2558 scope.go:117] "RemoveContainer" containerID="154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e"
Sep 11 19:32:37.240617 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:32:37.240598    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e\": container with ID starting with 154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e not found: ID does not exist" containerID="154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e"
Sep 11 19:32:37.240671 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.240621    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e"} err="failed to get container status \"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e\": rpc error: code = NotFound desc = could not find container \"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e\": container with ID starting with 154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e not found: ID does not exist"
Sep 11 19:32:37.240671 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.240632    2558 scope.go:117] "RemoveContainer" containerID="3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0"
Sep 11 19:32:37.240841 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.240822    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 19:32:37.240935 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:32:37.240916    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0\": container with ID starting with 3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0 not found: ID does not exist" containerID="3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0"
Sep 11 19:32:37.240990 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.240939    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0"} err="failed to get container status \"3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0\": rpc error: code = NotFound desc = could not find container \"3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0\": container with ID starting with 3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0 not found: ID does not exist"
Sep 11 19:32:37.240990 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.240952    2558 scope.go:117] "RemoveContainer" containerID="fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43"
Sep 11 19:32:37.241158 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241142    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="prometheus"
Sep 11 19:32:37.241214 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241150    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43"} err="failed to get container status \"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43\": rpc error: code = NotFound desc = could not find container \"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43\": container with ID starting with fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43 not found: ID does not exist"
Sep 11 19:32:37.241214 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241163    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="prometheus"
Sep 11 19:32:37.241214 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241178    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="kube-rbac-proxy-thanos"
Sep 11 19:32:37.241214 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241188    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="kube-rbac-proxy-thanos"
Sep 11 19:32:37.241214 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241165    2558 scope.go:117] "RemoveContainer" containerID="a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673"
Sep 11 19:32:37.241214 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241199    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="init-config-reloader"
Sep 11 19:32:37.241214 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241210    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="init-config-reloader"
Sep 11 19:32:37.241541 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241220    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="kube-rbac-proxy-web"
Sep 11 19:32:37.241541 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241228    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="kube-rbac-proxy-web"
Sep 11 19:32:37.241541 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241237    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="kube-rbac-proxy"
Sep 11 19:32:37.241541 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241245    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="kube-rbac-proxy"
Sep 11 19:32:37.241541 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241258    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="thanos-sidecar"
Sep 11 19:32:37.241541 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241266    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="thanos-sidecar"
Sep 11 19:32:37.241541 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241294    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="config-reloader"
Sep 11 19:32:37.241541 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241302    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="config-reloader"
Sep 11 19:32:37.241541 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241366    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="config-reloader"
Sep 11 19:32:37.241541 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241377    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="kube-rbac-proxy"
Sep 11 19:32:37.241541 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241387    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="thanos-sidecar"
Sep 11 19:32:37.241541 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241388    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673"} err="failed to get container status \"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673\": rpc error: code = NotFound desc = could not find container \"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673\": container with ID starting with a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673 not found: ID does not exist"
Sep 11 19:32:37.241541 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241397    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="kube-rbac-proxy-web"
Sep 11 19:32:37.241541 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241400    2558 scope.go:117] "RemoveContainer" containerID="e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff"
Sep 11 19:32:37.241541 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241405    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="kube-rbac-proxy-thanos"
Sep 11 19:32:37.241541 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241416    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" containerName="prometheus"
Sep 11 19:32:37.242308 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241626    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff"} err="failed to get container status \"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff\": rpc error: code = NotFound desc = could not find container \"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff\": container with ID starting with e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff not found: ID does not exist"
Sep 11 19:32:37.242308 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241640    2558 scope.go:117] "RemoveContainer" containerID="a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204"
Sep 11 19:32:37.242308 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241863    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204"} err="failed to get container status \"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204\": rpc error: code = NotFound desc = could not find container \"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204\": container with ID starting with a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204 not found: ID does not exist"
Sep 11 19:32:37.242308 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.241880    2558 scope.go:117] "RemoveContainer" containerID="96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766"
Sep 11 19:32:37.242308 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.242067    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766"} err="failed to get container status \"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766\": rpc error: code = NotFound desc = could not find container \"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766\": container with ID starting with 96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766 not found: ID does not exist"
Sep 11 19:32:37.242308 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.242078    2558 scope.go:117] "RemoveContainer" containerID="154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e"
Sep 11 19:32:37.242308 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.242250    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e"} err="failed to get container status \"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e\": rpc error: code = NotFound desc = could not find container \"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e\": container with ID starting with 154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e not found: ID does not exist"
Sep 11 19:32:37.242308 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.242265    2558 scope.go:117] "RemoveContainer" containerID="3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0"
Sep 11 19:32:37.242752 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.242474    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0"} err="failed to get container status \"3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0\": rpc error: code = NotFound desc = could not find container \"3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0\": container with ID starting with 3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0 not found: ID does not exist"
Sep 11 19:32:37.242752 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.242487    2558 scope.go:117] "RemoveContainer" containerID="fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43"
Sep 11 19:32:37.242752 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.242686    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43"} err="failed to get container status \"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43\": rpc error: code = NotFound desc = could not find container \"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43\": container with ID starting with fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43 not found: ID does not exist"
Sep 11 19:32:37.242752 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.242724    2558 scope.go:117] "RemoveContainer" containerID="a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673"
Sep 11 19:32:37.242999 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.242980    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673"} err="failed to get container status \"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673\": rpc error: code = NotFound desc = could not find container \"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673\": container with ID starting with a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673 not found: ID does not exist"
Sep 11 19:32:37.243040 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.243000    2558 scope.go:117] "RemoveContainer" containerID="e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff"
Sep 11 19:32:37.243268 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.243250    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff"} err="failed to get container status \"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff\": rpc error: code = NotFound desc = could not find container \"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff\": container with ID starting with e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff not found: ID does not exist"
Sep 11 19:32:37.243268 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.243268    2558 scope.go:117] "RemoveContainer" containerID="a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204"
Sep 11 19:32:37.243581 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.243560    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204"} err="failed to get container status \"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204\": rpc error: code = NotFound desc = could not find container \"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204\": container with ID starting with a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204 not found: ID does not exist"
Sep 11 19:32:37.243668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.243582    2558 scope.go:117] "RemoveContainer" containerID="96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766"
Sep 11 19:32:37.243857 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.243837    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766"} err="failed to get container status \"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766\": rpc error: code = NotFound desc = could not find container \"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766\": container with ID starting with 96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766 not found: ID does not exist"
Sep 11 19:32:37.243923 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.243858    2558 scope.go:117] "RemoveContainer" containerID="154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e"
Sep 11 19:32:37.246090 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.246061    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e"} err="failed to get container status \"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e\": rpc error: code = NotFound desc = could not find container \"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e\": container with ID starting with 154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e not found: ID does not exist"
Sep 11 19:32:37.246090 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.246089    2558 scope.go:117] "RemoveContainer" containerID="3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0"
Sep 11 19:32:37.246548 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.246527    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0"} err="failed to get container status \"3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0\": rpc error: code = NotFound desc = could not find container \"3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0\": container with ID starting with 3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0 not found: ID does not exist"
Sep 11 19:32:37.246631 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.246551    2558 scope.go:117] "RemoveContainer" containerID="fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43"
Sep 11 19:32:37.246877 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.246852    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43"} err="failed to get container status \"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43\": rpc error: code = NotFound desc = could not find container \"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43\": container with ID starting with fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43 not found: ID does not exist"
Sep 11 19:32:37.246877 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.246877    2558 scope.go:117] "RemoveContainer" containerID="a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673"
Sep 11 19:32:37.247188 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.247162    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673"} err="failed to get container status \"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673\": rpc error: code = NotFound desc = could not find container \"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673\": container with ID starting with a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673 not found: ID does not exist"
Sep 11 19:32:37.247188 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.247187    2558 scope.go:117] "RemoveContainer" containerID="e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff"
Sep 11 19:32:37.247408 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.247379    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.250036 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.249868    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff"} err="failed to get container status \"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff\": rpc error: code = NotFound desc = could not find container \"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff\": container with ID starting with e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff not found: ID does not exist"
Sep 11 19:32:37.250036 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.249899    2558 scope.go:117] "RemoveContainer" containerID="a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204"
Sep 11 19:32:37.250036 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.250033    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 19:32:37.250869 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.250073    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 19:32:37.250869 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.250109    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 19:32:37.250869 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.250142    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 19:32:37.250869 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.250309    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204"} err="failed to get container status \"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204\": rpc error: code = NotFound desc = could not find container \"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204\": container with ID starting with a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204 not found: ID does not exist"
Sep 11 19:32:37.250869 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.250332    2558 scope.go:117] "RemoveContainer" containerID="96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766"
Sep 11 19:32:37.250869 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.250348    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 19:32:37.250869 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.250358    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 19:32:37.250869 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.250361    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 19:32:37.250869 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.250694    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766"} err="failed to get container status \"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766\": rpc error: code = NotFound desc = could not find container \"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766\": container with ID starting with 96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766 not found: ID does not exist"
Sep 11 19:32:37.250869 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.250742    2558 scope.go:117] "RemoveContainer" containerID="154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e"
Sep 11 19:32:37.250869 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.250785    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 19:32:37.251464 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.251110    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 19:32:37.251464 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.251104    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e"} err="failed to get container status \"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e\": rpc error: code = NotFound desc = could not find container \"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e\": container with ID starting with 154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e not found: ID does not exist"
Sep 11 19:32:37.251464 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.251146    2558 scope.go:117] "RemoveContainer" containerID="3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0"
Sep 11 19:32:37.251464 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.251164    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 19:32:37.251464 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.251183    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 19:32:37.251464 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.251183    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7tun6f5d38rkr\""
Sep 11 19:32:37.251464 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.251250    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-82bfd\""
Sep 11 19:32:37.251464 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.251388    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0"} err="failed to get container status \"3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0\": rpc error: code = NotFound desc = could not find container \"3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0\": container with ID starting with 3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0 not found: ID does not exist"
Sep 11 19:32:37.251464 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.251404    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 19:32:37.251464 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.251408    2558 scope.go:117] "RemoveContainer" containerID="fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43"
Sep 11 19:32:37.251887 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.251471    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 19:32:37.251887 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.251683    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43"} err="failed to get container status \"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43\": rpc error: code = NotFound desc = could not find container \"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43\": container with ID starting with fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43 not found: ID does not exist"
Sep 11 19:32:37.251887 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.251701    2558 scope.go:117] "RemoveContainer" containerID="a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673"
Sep 11 19:32:37.252013 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.251968    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673"} err="failed to get container status \"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673\": rpc error: code = NotFound desc = could not find container \"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673\": container with ID starting with a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673 not found: ID does not exist"
Sep 11 19:32:37.252013 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.251991    2558 scope.go:117] "RemoveContainer" containerID="e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff"
Sep 11 19:32:37.252299 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.252270    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff"} err="failed to get container status \"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff\": rpc error: code = NotFound desc = could not find container \"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff\": container with ID starting with e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff not found: ID does not exist"
Sep 11 19:32:37.252354 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.252302    2558 scope.go:117] "RemoveContainer" containerID="a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204"
Sep 11 19:32:37.252565 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.252537    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204"} err="failed to get container status \"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204\": rpc error: code = NotFound desc = could not find container \"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204\": container with ID starting with a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204 not found: ID does not exist"
Sep 11 19:32:37.252565 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.252559    2558 scope.go:117] "RemoveContainer" containerID="96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766"
Sep 11 19:32:37.253098 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.253067    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766"} err="failed to get container status \"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766\": rpc error: code = NotFound desc = could not find container \"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766\": container with ID starting with 96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766 not found: ID does not exist"
Sep 11 19:32:37.253098 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.253096    2558 scope.go:117] "RemoveContainer" containerID="154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e"
Sep 11 19:32:37.253444 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.253416    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e"} err="failed to get container status \"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e\": rpc error: code = NotFound desc = could not find container \"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e\": container with ID starting with 154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e not found: ID does not exist"
Sep 11 19:32:37.253444 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.253444    2558 scope.go:117] "RemoveContainer" containerID="3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0"
Sep 11 19:32:37.253793 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.253762    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0"} err="failed to get container status \"3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0\": rpc error: code = NotFound desc = could not find container \"3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0\": container with ID starting with 3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0 not found: ID does not exist"
Sep 11 19:32:37.253793 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.253790    2558 scope.go:117] "RemoveContainer" containerID="fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43"
Sep 11 19:32:37.254009 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.253990    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 19:32:37.254090 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.254055    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43"} err="failed to get container status \"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43\": rpc error: code = NotFound desc = could not find container \"fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43\": container with ID starting with fca70a367f662c7c7f3ce8ee0ffcbaccdce536dfdfa4ff1121ab8479c74e1d43 not found: ID does not exist"
Sep 11 19:32:37.254090 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.254071    2558 scope.go:117] "RemoveContainer" containerID="a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673"
Sep 11 19:32:37.254341 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.254321    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673"} err="failed to get container status \"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673\": rpc error: code = NotFound desc = could not find container \"a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673\": container with ID starting with a54e8269c78ed7662314231371cdc5c23a4b337a7efa226bf5276f2a03ee1673 not found: ID does not exist"
Sep 11 19:32:37.254419 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.254342    2558 scope.go:117] "RemoveContainer" containerID="e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff"
Sep 11 19:32:37.254611 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.254580    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff"} err="failed to get container status \"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff\": rpc error: code = NotFound desc = could not find container \"e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff\": container with ID starting with e864ae4c16600a0d7274a2035ee7f45a0291b6a1a22cb752a03717768fb9a9ff not found: ID does not exist"
Sep 11 19:32:37.254611 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.254613    2558 scope.go:117] "RemoveContainer" containerID="a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204"
Sep 11 19:32:37.254900 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.254876    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204"} err="failed to get container status \"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204\": rpc error: code = NotFound desc = could not find container \"a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204\": container with ID starting with a96ad99dd509b48ab8478d6a070fb6a0e0e4e3b7f43e7e85a05a28619c0e3204 not found: ID does not exist"
Sep 11 19:32:37.254972 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.254903    2558 scope.go:117] "RemoveContainer" containerID="96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766"
Sep 11 19:32:37.255210 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.255186    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766"} err="failed to get container status \"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766\": rpc error: code = NotFound desc = could not find container \"96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766\": container with ID starting with 96287ae55ed4c19b61b9dd91d1ac013f087f39021132c8477570b09765b82766 not found: ID does not exist"
Sep 11 19:32:37.255275 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.255212    2558 scope.go:117] "RemoveContainer" containerID="154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e"
Sep 11 19:32:37.255538 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.255494    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e"} err="failed to get container status \"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e\": rpc error: code = NotFound desc = could not find container \"154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e\": container with ID starting with 154c70a810731ef202caf430c78e0f4af6df078ba3b6f0ee27fb32495f26d75e not found: ID does not exist"
Sep 11 19:32:37.255538 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.255535    2558 scope.go:117] "RemoveContainer" containerID="3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0"
Sep 11 19:32:37.255853 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.255828    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0"} err="failed to get container status \"3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0\": rpc error: code = NotFound desc = could not find container \"3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0\": container with ID starting with 3c122d2a1385106418584e0aba8142d5c458341ee4f7e0e602430f0aec1d56a0 not found: ID does not exist"
Sep 11 19:32:37.257306 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.257283    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 19:32:37.261561 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.261537    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 19:32:37.351165 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.351121    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.351165 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.351164    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.351364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.351194    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.351364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.351232    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.351364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.351252    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-config\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.351364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.351275    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-config-out\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.351364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.351303    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.351364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.351322    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.351364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.351336    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.351622 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.351376    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.351622 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.351409    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf5nc\" (UniqueName: \"kubernetes.io/projected/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-kube-api-access-jf5nc\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.351622 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.351436    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.351622 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.351466    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.351622 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.351492    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.351622 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.351506    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.351622 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.351525    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.351622 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.351541    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.351622 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.351557    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.351622 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.351597    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.351622 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.351618    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-web-config\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453058 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.452965    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453058 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453006    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453058 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453027    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453058 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453050    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453068    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-config\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453088    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-config-out\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453111    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453127    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453144    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453171    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453196    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf5nc\" (UniqueName: \"kubernetes.io/projected/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-kube-api-access-jf5nc\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453230    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453275    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453305    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453327    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453357    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453969 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453385    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453969 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453410    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453969 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453439    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453969 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453465    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-web-config\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453969 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453807    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.453969 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.453941    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.454318 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.454289    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.456145 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.456117    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-config-out\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.456385 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.456361    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.456461 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.456361    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.456461 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.456421    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-web-config\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.456575 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.456468    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-config\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.456575 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.456525    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.456678 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.456597    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.456817 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.456794    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.457123 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.457062    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.457228 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.457183    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.457606 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.457576    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.457700 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.457676    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.458813 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.458796    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.458946 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.458930    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.459349 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.459329    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.460036 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.460018    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.464868 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.464850    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf5nc\" (UniqueName: \"kubernetes.io/projected/5c8b0a80-0b86-4c1c-a5c9-642029c131ee-kube-api-access-jf5nc\") pod \"prometheus-k8s-0\" (UID: \"5c8b0a80-0b86-4c1c-a5c9-642029c131ee\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.560340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.560297    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:32:37.756591 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.756491    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs\") pod \"network-metrics-daemon-zzscf\" (UID: \"e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef\") " pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:32:37.758848 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.758821    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef-metrics-certs\") pod \"network-metrics-daemon-zzscf\" (UID: \"e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef\") " pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:32:37.812632 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.812594    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3cefa7a-c8b4-4d19-8253-1150614c4e6e" path="/var/lib/kubelet/pods/e3cefa7a-c8b4-4d19-8253-1150614c4e6e/volumes"
Sep 11 19:32:37.840422 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.840387    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 19:32:37.843377 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:32:37.843342    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8b0a80_0b86_4c1c_a5c9_642029c131ee.slice/crio-a201ab54e4f0af7901ca84e411df48b3195bffc116e5ba9d1e3a6ae3239a65ac WatchSource:0}: Error finding container a201ab54e4f0af7901ca84e411df48b3195bffc116e5ba9d1e3a6ae3239a65ac: Status 404 returned error can't find the container with id a201ab54e4f0af7901ca84e411df48b3195bffc116e5ba9d1e3a6ae3239a65ac
Sep 11 19:32:37.910309 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.910278    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k4fsc\""
Sep 11 19:32:37.919023 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:37.918993    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zzscf"
Sep 11 19:32:38.186628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:38.186596    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zzscf"]
Sep 11 19:32:38.190129 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:32:38.190099    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92e2eb8_6205_4e9a_9f99_9afdb7cbcdef.slice/crio-a53d403d297f06639765ba69801d59b0a630935b4a3b7d8ab8d0e2a8b780d901 WatchSource:0}: Error finding container a53d403d297f06639765ba69801d59b0a630935b4a3b7d8ab8d0e2a8b780d901: Status 404 returned error can't find the container with id a53d403d297f06639765ba69801d59b0a630935b4a3b7d8ab8d0e2a8b780d901
Sep 11 19:32:38.192322 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:38.192300    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 19:32:38.193016 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:38.192995    2558 generic.go:356] "Generic (PLEG): container finished" podID="5c8b0a80-0b86-4c1c-a5c9-642029c131ee" containerID="2b61c8f4dedc4328923db213088321f0c7da3d88af1c047acac2a55da5a9757c" exitCode=0
Sep 11 19:32:38.193081 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:38.193067    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c8b0a80-0b86-4c1c-a5c9-642029c131ee","Type":"ContainerDied","Data":"2b61c8f4dedc4328923db213088321f0c7da3d88af1c047acac2a55da5a9757c"}
Sep 11 19:32:38.193144 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:38.193130    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c8b0a80-0b86-4c1c-a5c9-642029c131ee","Type":"ContainerStarted","Data":"a201ab54e4f0af7901ca84e411df48b3195bffc116e5ba9d1e3a6ae3239a65ac"}
Sep 11 19:32:39.201253 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:39.201222    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c8b0a80-0b86-4c1c-a5c9-642029c131ee","Type":"ContainerStarted","Data":"1ab77653587b363b63e9f1cc92e82858fbe932a22dbafd53c382153accfda045"}
Sep 11 19:32:39.201617 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:39.201321    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c8b0a80-0b86-4c1c-a5c9-642029c131ee","Type":"ContainerStarted","Data":"4f05e193d26b3d55315374427df141fe0173ce6ef3b91253aa6e4b4698b5c7db"}
Sep 11 19:32:39.201617 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:39.201350    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c8b0a80-0b86-4c1c-a5c9-642029c131ee","Type":"ContainerStarted","Data":"9cf7fd32f165ad6662e1f70d8a8e8e4eeb4229a0775472109f05674601736fc9"}
Sep 11 19:32:39.201617 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:39.201380    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c8b0a80-0b86-4c1c-a5c9-642029c131ee","Type":"ContainerStarted","Data":"4c2f5dac46e211a05773babd2c9d3334c79ccfba48fadde327df0a970d205a7f"}
Sep 11 19:32:39.201617 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:39.201422    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c8b0a80-0b86-4c1c-a5c9-642029c131ee","Type":"ContainerStarted","Data":"efa53b9d25d369482ff8f46b06e9f0d5f40cc67d5b84034b0dadaf072fa1de5e"}
Sep 11 19:32:39.201617 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:39.201450    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5c8b0a80-0b86-4c1c-a5c9-642029c131ee","Type":"ContainerStarted","Data":"0f4d2f41639f834d93185e43d2f8772d3bce852de31cd604e38cb5294c1e9673"}
Sep 11 19:32:39.202846 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:39.202822    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zzscf" event={"ID":"e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef","Type":"ContainerStarted","Data":"097ff3591ad2da32f2647af2b9400210cb0b0c8080bf952637c4b0f9b8e0f1b6"}
Sep 11 19:32:39.202926 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:39.202873    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zzscf" event={"ID":"e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef","Type":"ContainerStarted","Data":"a53d403d297f06639765ba69801d59b0a630935b4a3b7d8ab8d0e2a8b780d901"}
Sep 11 19:32:39.234369 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:39.234313    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.214329946 podStartE2EDuration="2.234296455s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.966509ms" isStatefulPod=false podCreationTimestamp="2026-09-11 19:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:32:39.23136872 +0000 UTC m=+375.848538213" watchObservedRunningTime="2026-09-11 19:32:39.234296455 +0000 UTC m=+375.851465946"
Sep 11 19:32:40.209315 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:40.209276    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zzscf" event={"ID":"e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef","Type":"ContainerStarted","Data":"7f5c0fc78252d492dfb9e15f188c74024bd4531bb6493922e29f5d6f0b1a9998"}
Sep 11 19:32:40.225077 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:40.225025    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zzscf" podStartSLOduration=375.315514125 podStartE2EDuration="6m16.225009776s" totalImagesPullingTime="909.495651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:32:40.223601368 +0000 UTC m=+376.840770858" watchObservedRunningTime="2026-09-11 19:32:40.225009776 +0000 UTC m=+376.842179266"
Sep 11 19:32:42.561325 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:32:42.561278    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:33:37.560618 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:33:37.560574    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:33:37.575786 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:33:37.575759    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:33:38.440026 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:33:38.439996    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 19:35:24.628693 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:24.628601    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xmsv9"]
Sep 11 19:35:24.631924 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:24.631902    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xmsv9"
Sep 11 19:35:24.634271 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:24.634245    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 19:35:24.634674 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:24.634655    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 19:35:24.634808 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:24.634673    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6h8px\""
Sep 11 19:35:24.641459 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:24.641437    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xmsv9"]
Sep 11 19:35:24.707250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:24.707216    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgtnq\" (UniqueName: \"kubernetes.io/projected/7ab88c1d-216b-4c86-8abf-80468a16c5b5-kube-api-access-rgtnq\") pod \"openshift-lws-operator-fd8ccff4c-xmsv9\" (UID: \"7ab88c1d-216b-4c86-8abf-80468a16c5b5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xmsv9"
Sep 11 19:35:24.707395 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:24.707326    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ab88c1d-216b-4c86-8abf-80468a16c5b5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xmsv9\" (UID: \"7ab88c1d-216b-4c86-8abf-80468a16c5b5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xmsv9"
Sep 11 19:35:24.808512 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:24.808476    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ab88c1d-216b-4c86-8abf-80468a16c5b5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xmsv9\" (UID: \"7ab88c1d-216b-4c86-8abf-80468a16c5b5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xmsv9"
Sep 11 19:35:24.808512 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:24.808518    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtnq\" (UniqueName: \"kubernetes.io/projected/7ab88c1d-216b-4c86-8abf-80468a16c5b5-kube-api-access-rgtnq\") pod \"openshift-lws-operator-fd8ccff4c-xmsv9\" (UID: \"7ab88c1d-216b-4c86-8abf-80468a16c5b5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xmsv9"
Sep 11 19:35:24.808922 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:24.808897    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ab88c1d-216b-4c86-8abf-80468a16c5b5-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xmsv9\" (UID: \"7ab88c1d-216b-4c86-8abf-80468a16c5b5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xmsv9"
Sep 11 19:35:24.816201 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:24.816179    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgtnq\" (UniqueName: \"kubernetes.io/projected/7ab88c1d-216b-4c86-8abf-80468a16c5b5-kube-api-access-rgtnq\") pod \"openshift-lws-operator-fd8ccff4c-xmsv9\" (UID: \"7ab88c1d-216b-4c86-8abf-80468a16c5b5\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xmsv9"
Sep 11 19:35:24.941617 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:24.941579    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xmsv9"
Sep 11 19:35:25.209269 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:25.209191    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xmsv9"]
Sep 11 19:35:25.212357 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:35:25.212326    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ab88c1d_216b_4c86_8abf_80468a16c5b5.slice/crio-3388c91b749d78f604d6c364a08f176b4b7eb69fac87dab11e01e425a0fefe9e WatchSource:0}: Error finding container 3388c91b749d78f604d6c364a08f176b4b7eb69fac87dab11e01e425a0fefe9e: Status 404 returned error can't find the container with id 3388c91b749d78f604d6c364a08f176b4b7eb69fac87dab11e01e425a0fefe9e
Sep 11 19:35:25.829010 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:25.828968    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xmsv9" event={"ID":"7ab88c1d-216b-4c86-8abf-80468a16c5b5","Type":"ContainerStarted","Data":"3388c91b749d78f604d6c364a08f176b4b7eb69fac87dab11e01e425a0fefe9e"}
Sep 11 19:35:28.843872 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:28.843838    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xmsv9" event={"ID":"7ab88c1d-216b-4c86-8abf-80468a16c5b5","Type":"ContainerStarted","Data":"053641195c21064e9c21e4162c7869057f32c7c7c2cc9051528b0b37cbe38cd3"}
Sep 11 19:35:28.862434 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:28.862386    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xmsv9" podStartSLOduration=2.183144474 podStartE2EDuration="4.862368899s" totalImagesPullingTime="2.679224425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:35:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:35:28.861108863 +0000 UTC m=+545.478278353" watchObservedRunningTime="2026-09-11 19:35:28.862368899 +0000 UTC m=+545.479538398"
Sep 11 19:35:44.362819 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.362786    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"]
Sep 11 19:35:44.366082 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.366064    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"
Sep 11 19:35:44.368444 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.368422    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 19:35:44.368746 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.368728    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 19:35:44.369211 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.369195    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 19:35:44.369536 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.369518    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 19:35:44.370045 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.370023    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2rjxh\""
Sep 11 19:35:44.385103 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.385071    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"]
Sep 11 19:35:44.471053 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.471021    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4d22b8c9-3e1c-445b-9c2c-e873033b0761-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-ndv2n\" (UID: \"4d22b8c9-3e1c-445b-9c2c-e873033b0761\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"
Sep 11 19:35:44.471180 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.471068    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d22b8c9-3e1c-445b-9c2c-e873033b0761-tmp\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-ndv2n\" (UID: \"4d22b8c9-3e1c-445b-9c2c-e873033b0761\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"
Sep 11 19:35:44.471180 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.471089    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28jqv\" (UniqueName: \"kubernetes.io/projected/4d22b8c9-3e1c-445b-9c2c-e873033b0761-kube-api-access-28jqv\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-ndv2n\" (UID: \"4d22b8c9-3e1c-445b-9c2c-e873033b0761\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"
Sep 11 19:35:44.471180 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.471159    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4d22b8c9-3e1c-445b-9c2c-e873033b0761-manifests\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-ndv2n\" (UID: \"4d22b8c9-3e1c-445b-9c2c-e873033b0761\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"
Sep 11 19:35:44.471285 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.471187    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d22b8c9-3e1c-445b-9c2c-e873033b0761-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-ndv2n\" (UID: \"4d22b8c9-3e1c-445b-9c2c-e873033b0761\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"
Sep 11 19:35:44.572315 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.572284    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4d22b8c9-3e1c-445b-9c2c-e873033b0761-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-ndv2n\" (UID: \"4d22b8c9-3e1c-445b-9c2c-e873033b0761\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"
Sep 11 19:35:44.572429 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.572329    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d22b8c9-3e1c-445b-9c2c-e873033b0761-tmp\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-ndv2n\" (UID: \"4d22b8c9-3e1c-445b-9c2c-e873033b0761\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"
Sep 11 19:35:44.572429 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.572347    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28jqv\" (UniqueName: \"kubernetes.io/projected/4d22b8c9-3e1c-445b-9c2c-e873033b0761-kube-api-access-28jqv\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-ndv2n\" (UID: \"4d22b8c9-3e1c-445b-9c2c-e873033b0761\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"
Sep 11 19:35:44.572500 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.572468    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4d22b8c9-3e1c-445b-9c2c-e873033b0761-manifests\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-ndv2n\" (UID: \"4d22b8c9-3e1c-445b-9c2c-e873033b0761\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"
Sep 11 19:35:44.572539 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.572509    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d22b8c9-3e1c-445b-9c2c-e873033b0761-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-ndv2n\" (UID: \"4d22b8c9-3e1c-445b-9c2c-e873033b0761\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"
Sep 11 19:35:44.572818 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.572801    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4d22b8c9-3e1c-445b-9c2c-e873033b0761-manifests\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-ndv2n\" (UID: \"4d22b8c9-3e1c-445b-9c2c-e873033b0761\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"
Sep 11 19:35:44.574648 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.574620    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d22b8c9-3e1c-445b-9c2c-e873033b0761-tmp\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-ndv2n\" (UID: \"4d22b8c9-3e1c-445b-9c2c-e873033b0761\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"
Sep 11 19:35:44.574793 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.574776    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4d22b8c9-3e1c-445b-9c2c-e873033b0761-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-ndv2n\" (UID: \"4d22b8c9-3e1c-445b-9c2c-e873033b0761\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"
Sep 11 19:35:44.574863 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.574849    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4d22b8c9-3e1c-445b-9c2c-e873033b0761-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-ndv2n\" (UID: \"4d22b8c9-3e1c-445b-9c2c-e873033b0761\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"
Sep 11 19:35:44.581739 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.581694    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28jqv\" (UniqueName: \"kubernetes.io/projected/4d22b8c9-3e1c-445b-9c2c-e873033b0761-kube-api-access-28jqv\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-ndv2n\" (UID: \"4d22b8c9-3e1c-445b-9c2c-e873033b0761\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"
Sep 11 19:35:44.675813 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.675786    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"
Sep 11 19:35:44.953991 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:44.953922    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"]
Sep 11 19:35:44.957744 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:35:44.957703    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d22b8c9_3e1c_445b_9c2c_e873033b0761.slice/crio-dab2fd93854f3b2584a05e0b487388dc0cd7aca6aafd7aa61749dc649cb59fc5 WatchSource:0}: Error finding container dab2fd93854f3b2584a05e0b487388dc0cd7aca6aafd7aa61749dc649cb59fc5: Status 404 returned error can't find the container with id dab2fd93854f3b2584a05e0b487388dc0cd7aca6aafd7aa61749dc649cb59fc5
Sep 11 19:35:45.909317 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:45.909272    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n" event={"ID":"4d22b8c9-3e1c-445b-9c2c-e873033b0761","Type":"ContainerStarted","Data":"dab2fd93854f3b2584a05e0b487388dc0cd7aca6aafd7aa61749dc649cb59fc5"}
Sep 11 19:35:48.923911 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:48.923880    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n" event={"ID":"4d22b8c9-3e1c-445b-9c2c-e873033b0761","Type":"ContainerStarted","Data":"037c2366678e7d901c538e9cf3b2d60d378be619113f86326f2a961a1cfcc5c0"}
Sep 11 19:35:49.928764 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:49.928724    2558 generic.go:356] "Generic (PLEG): container finished" podID="4d22b8c9-3e1c-445b-9c2c-e873033b0761" containerID="037c2366678e7d901c538e9cf3b2d60d378be619113f86326f2a961a1cfcc5c0" exitCode=0
Sep 11 19:35:49.929158 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:49.928807    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n" event={"ID":"4d22b8c9-3e1c-445b-9c2c-e873033b0761","Type":"ContainerDied","Data":"037c2366678e7d901c538e9cf3b2d60d378be619113f86326f2a961a1cfcc5c0"}
Sep 11 19:35:50.935276 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:50.935245    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n" event={"ID":"4d22b8c9-3e1c-445b-9c2c-e873033b0761","Type":"ContainerStarted","Data":"7c0dc684363f842c89e628cf2c6867540063e04ad97047f7108b56ae2c5fb0f4"}
Sep 11 19:35:50.935630 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:50.935373    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"
Sep 11 19:35:50.955775 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:35:50.955732    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n" podStartSLOduration=2.428982667 podStartE2EDuration="6.955701394s" totalImagesPullingTime="3.495771582s" totalInitContainerRuntime="1.030947145s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:35:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:35:50.953857877 +0000 UTC m=+567.571027392" watchObservedRunningTime="2026-09-11 19:35:50.955701394 +0000 UTC m=+567.572870885"
Sep 11 19:36:01.941818 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:01.941788    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-ndv2n"
Sep 11 19:36:21.772573 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:21.772540    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt"]
Sep 11 19:36:21.775744 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:21.775700    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt"
Sep 11 19:36:21.778506 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:21.778484    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 19:36:21.779156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:21.779125    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 19:36:21.779265 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:21.779153    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 19:36:21.779265 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:21.779172    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ndnhc\""
Sep 11 19:36:21.784583 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:21.784559    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt"]
Sep 11 19:36:21.859421 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:21.859391    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/69a666ce-ef0f-4db5-8753-7e3907d59888-metrics-cert\") pod \"lws-controller-manager-6d6bf85947-78cbt\" (UID: \"69a666ce-ef0f-4db5-8753-7e3907d59888\") " pod="openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt"
Sep 11 19:36:21.859604 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:21.859444    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2t68\" (UniqueName: \"kubernetes.io/projected/69a666ce-ef0f-4db5-8753-7e3907d59888-kube-api-access-x2t68\") pod \"lws-controller-manager-6d6bf85947-78cbt\" (UID: \"69a666ce-ef0f-4db5-8753-7e3907d59888\") " pod="openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt"
Sep 11 19:36:21.859604 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:21.859463    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69a666ce-ef0f-4db5-8753-7e3907d59888-cert\") pod \"lws-controller-manager-6d6bf85947-78cbt\" (UID: \"69a666ce-ef0f-4db5-8753-7e3907d59888\") " pod="openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt"
Sep 11 19:36:21.859604 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:21.859532    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/69a666ce-ef0f-4db5-8753-7e3907d59888-manager-config\") pod \"lws-controller-manager-6d6bf85947-78cbt\" (UID: \"69a666ce-ef0f-4db5-8753-7e3907d59888\") " pod="openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt"
Sep 11 19:36:21.960600 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:21.960559    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/69a666ce-ef0f-4db5-8753-7e3907d59888-metrics-cert\") pod \"lws-controller-manager-6d6bf85947-78cbt\" (UID: \"69a666ce-ef0f-4db5-8753-7e3907d59888\") " pod="openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt"
Sep 11 19:36:21.960782 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:21.960635    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2t68\" (UniqueName: \"kubernetes.io/projected/69a666ce-ef0f-4db5-8753-7e3907d59888-kube-api-access-x2t68\") pod \"lws-controller-manager-6d6bf85947-78cbt\" (UID: \"69a666ce-ef0f-4db5-8753-7e3907d59888\") " pod="openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt"
Sep 11 19:36:21.960782 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:21.960664    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69a666ce-ef0f-4db5-8753-7e3907d59888-cert\") pod \"lws-controller-manager-6d6bf85947-78cbt\" (UID: \"69a666ce-ef0f-4db5-8753-7e3907d59888\") " pod="openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt"
Sep 11 19:36:21.960782 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:21.960698    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/69a666ce-ef0f-4db5-8753-7e3907d59888-manager-config\") pod \"lws-controller-manager-6d6bf85947-78cbt\" (UID: \"69a666ce-ef0f-4db5-8753-7e3907d59888\") " pod="openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt"
Sep 11 19:36:21.961412 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:21.961387    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/69a666ce-ef0f-4db5-8753-7e3907d59888-manager-config\") pod \"lws-controller-manager-6d6bf85947-78cbt\" (UID: \"69a666ce-ef0f-4db5-8753-7e3907d59888\") " pod="openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt"
Sep 11 19:36:21.963103 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:21.963083    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/69a666ce-ef0f-4db5-8753-7e3907d59888-metrics-cert\") pod \"lws-controller-manager-6d6bf85947-78cbt\" (UID: \"69a666ce-ef0f-4db5-8753-7e3907d59888\") " pod="openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt"
Sep 11 19:36:21.963176 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:21.963115    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69a666ce-ef0f-4db5-8753-7e3907d59888-cert\") pod \"lws-controller-manager-6d6bf85947-78cbt\" (UID: \"69a666ce-ef0f-4db5-8753-7e3907d59888\") " pod="openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt"
Sep 11 19:36:21.969287 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:21.969269    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2t68\" (UniqueName: \"kubernetes.io/projected/69a666ce-ef0f-4db5-8753-7e3907d59888-kube-api-access-x2t68\") pod \"lws-controller-manager-6d6bf85947-78cbt\" (UID: \"69a666ce-ef0f-4db5-8753-7e3907d59888\") " pod="openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt"
Sep 11 19:36:22.086321 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:22.086234    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt"
Sep 11 19:36:22.357315 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:22.357289    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt"]
Sep 11 19:36:22.359524 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:36:22.359497    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a666ce_ef0f_4db5_8753_7e3907d59888.slice/crio-7eec415e7478620a4b3f8ab642cab17d31bed3851a472428c5c98949a17c5bae WatchSource:0}: Error finding container 7eec415e7478620a4b3f8ab642cab17d31bed3851a472428c5c98949a17c5bae: Status 404 returned error can't find the container with id 7eec415e7478620a4b3f8ab642cab17d31bed3851a472428c5c98949a17c5bae
Sep 11 19:36:23.064772 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:23.064737    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt" event={"ID":"69a666ce-ef0f-4db5-8753-7e3907d59888","Type":"ContainerStarted","Data":"7eec415e7478620a4b3f8ab642cab17d31bed3851a472428c5c98949a17c5bae"}
Sep 11 19:36:24.072568 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:24.072521    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt" event={"ID":"69a666ce-ef0f-4db5-8753-7e3907d59888","Type":"ContainerStarted","Data":"ce040a57c20e8e3744d631cc0670a46f713bd1ab8d83aca7410a9aa68394bd6a"}
Sep 11 19:36:24.073113 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:24.072871    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt"
Sep 11 19:36:24.090942 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:24.090876    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt" podStartSLOduration=1.484734995 podStartE2EDuration="3.090857355s" totalImagesPullingTime="1.60612236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:36:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:36:24.088827851 +0000 UTC m=+600.705997355" watchObservedRunningTime="2026-09-11 19:36:24.090857355 +0000 UTC m=+600.708026848"
Sep 11 19:36:24.120008 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:24.119972    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bvhv7"]
Sep 11 19:36:24.123558 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:24.123538    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bvhv7"
Sep 11 19:36:24.125470 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:24.125445    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 11 19:36:24.125579 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:24.125494    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8sl4b\""
Sep 11 19:36:24.131755 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:24.131730    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bvhv7"]
Sep 11 19:36:24.179395 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:24.179360    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d537e8db-07f2-4b1d-88cf-be3abd543e71-cert\") pod \"kserve-controller-manager-d6f4b65b8-bvhv7\" (UID: \"d537e8db-07f2-4b1d-88cf-be3abd543e71\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bvhv7"
Sep 11 19:36:24.179581 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:24.179436    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f96sz\" (UniqueName: \"kubernetes.io/projected/d537e8db-07f2-4b1d-88cf-be3abd543e71-kube-api-access-f96sz\") pod \"kserve-controller-manager-d6f4b65b8-bvhv7\" (UID: \"d537e8db-07f2-4b1d-88cf-be3abd543e71\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bvhv7"
Sep 11 19:36:24.280498 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:24.280397    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f96sz\" (UniqueName: \"kubernetes.io/projected/d537e8db-07f2-4b1d-88cf-be3abd543e71-kube-api-access-f96sz\") pod \"kserve-controller-manager-d6f4b65b8-bvhv7\" (UID: \"d537e8db-07f2-4b1d-88cf-be3abd543e71\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bvhv7"
Sep 11 19:36:24.280498 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:24.280448    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d537e8db-07f2-4b1d-88cf-be3abd543e71-cert\") pod \"kserve-controller-manager-d6f4b65b8-bvhv7\" (UID: \"d537e8db-07f2-4b1d-88cf-be3abd543e71\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bvhv7"
Sep 11 19:36:24.282926 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:24.282900    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d537e8db-07f2-4b1d-88cf-be3abd543e71-cert\") pod \"kserve-controller-manager-d6f4b65b8-bvhv7\" (UID: \"d537e8db-07f2-4b1d-88cf-be3abd543e71\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bvhv7"
Sep 11 19:36:24.289815 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:24.289794    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f96sz\" (UniqueName: \"kubernetes.io/projected/d537e8db-07f2-4b1d-88cf-be3abd543e71-kube-api-access-f96sz\") pod \"kserve-controller-manager-d6f4b65b8-bvhv7\" (UID: \"d537e8db-07f2-4b1d-88cf-be3abd543e71\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bvhv7"
Sep 11 19:36:24.436090 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:24.436057    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bvhv7"
Sep 11 19:36:24.704233 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:24.704202    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bvhv7"]
Sep 11 19:36:24.707203 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:36:24.707173    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd537e8db_07f2_4b1d_88cf_be3abd543e71.slice/crio-e9ba2fe8ebc2d307949fa589ae3eb49fb4af2b145ada5921bfa46ceb22affe83 WatchSource:0}: Error finding container e9ba2fe8ebc2d307949fa589ae3eb49fb4af2b145ada5921bfa46ceb22affe83: Status 404 returned error can't find the container with id e9ba2fe8ebc2d307949fa589ae3eb49fb4af2b145ada5921bfa46ceb22affe83
Sep 11 19:36:25.077596 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:25.077508    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bvhv7" event={"ID":"d537e8db-07f2-4b1d-88cf-be3abd543e71","Type":"ContainerStarted","Data":"e9ba2fe8ebc2d307949fa589ae3eb49fb4af2b145ada5921bfa46ceb22affe83"}
Sep 11 19:36:28.093140 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:28.093107    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bvhv7" event={"ID":"d537e8db-07f2-4b1d-88cf-be3abd543e71","Type":"ContainerStarted","Data":"2890bac56cce5d8109573bfab98278bd1f16598bc5d8c656503e796866425855"}
Sep 11 19:36:28.093562 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:28.093236    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bvhv7"
Sep 11 19:36:28.110476 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:28.110429    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bvhv7" podStartSLOduration=1.435271613 podStartE2EDuration="4.11041263s" totalImagesPullingTime="2.675141017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:36:28.109173981 +0000 UTC m=+604.726343472" watchObservedRunningTime="2026-09-11 19:36:28.11041263 +0000 UTC m=+604.727582123"
Sep 11 19:36:35.080458 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:35.080425    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6d6bf85947-78cbt"
Sep 11 19:36:59.102739 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:36:59.102641    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bvhv7"
Sep 11 19:37:41.451028 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:37:41.450993    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bd2rp"]
Sep 11 19:37:41.453339 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:37:41.453322    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bd2rp"
Sep 11 19:37:41.455562 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:37:41.455535    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 19:37:41.455701 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:37:41.455535    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 19:37:41.455701 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:37:41.455579    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 19:37:41.455701 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:37:41.455578    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-sjcn9\""
Sep 11 19:37:41.466650 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:37:41.466624    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bd2rp"]
Sep 11 19:37:41.510477 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:37:41.510455    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnlgk\" (UniqueName: \"kubernetes.io/projected/d6c77b1e-ad5d-439f-801d-a40432e32cb6-kube-api-access-lnlgk\") pod \"dns-operator-controller-manager-7b84cd7c9c-bd2rp\" (UID: \"d6c77b1e-ad5d-439f-801d-a40432e32cb6\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bd2rp"
Sep 11 19:37:41.611842 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:37:41.611792    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnlgk\" (UniqueName: \"kubernetes.io/projected/d6c77b1e-ad5d-439f-801d-a40432e32cb6-kube-api-access-lnlgk\") pod \"dns-operator-controller-manager-7b84cd7c9c-bd2rp\" (UID: \"d6c77b1e-ad5d-439f-801d-a40432e32cb6\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bd2rp"
Sep 11 19:37:41.620851 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:37:41.620819    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnlgk\" (UniqueName: \"kubernetes.io/projected/d6c77b1e-ad5d-439f-801d-a40432e32cb6-kube-api-access-lnlgk\") pod \"dns-operator-controller-manager-7b84cd7c9c-bd2rp\" (UID: \"d6c77b1e-ad5d-439f-801d-a40432e32cb6\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bd2rp"
Sep 11 19:37:41.763058 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:37:41.762963    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bd2rp"
Sep 11 19:37:42.035194 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:37:42.035066    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bd2rp"]
Sep 11 19:37:42.039465 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:37:42.039437    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c77b1e_ad5d_439f_801d_a40432e32cb6.slice/crio-747272e7fe2a6ad2029d39849cd0d862ae9696aaf166edb1c7aee7dda25615c6 WatchSource:0}: Error finding container 747272e7fe2a6ad2029d39849cd0d862ae9696aaf166edb1c7aee7dda25615c6: Status 404 returned error can't find the container with id 747272e7fe2a6ad2029d39849cd0d862ae9696aaf166edb1c7aee7dda25615c6
Sep 11 19:37:42.041653 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:37:42.041635    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 19:37:42.404135 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:37:42.404054    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bd2rp" event={"ID":"d6c77b1e-ad5d-439f-801d-a40432e32cb6","Type":"ContainerStarted","Data":"747272e7fe2a6ad2029d39849cd0d862ae9696aaf166edb1c7aee7dda25615c6"}
Sep 11 19:37:44.416152 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:37:44.416125    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bd2rp" event={"ID":"d6c77b1e-ad5d-439f-801d-a40432e32cb6","Type":"ContainerStarted","Data":"87e33552fa9a05dc536a5e3254a63a5b0b00e244156595c3c1150554f0965156"}
Sep 11 19:37:44.416486 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:37:44.416253    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bd2rp"
Sep 11 19:37:44.437069 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:37:44.437015    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bd2rp" podStartSLOduration=1.140899615 podStartE2EDuration="3.436997627s" totalImagesPullingTime="2.296098012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:37:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:37:44.435322609 +0000 UTC m=+681.052492100" watchObservedRunningTime="2026-09-11 19:37:44.436997627 +0000 UTC m=+681.054167117"
Sep 11 19:37:55.423512 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:37:55.423483    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bd2rp"
Sep 11 19:38:03.489460 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:03.489425    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pb52q"]
Sep 11 19:38:03.493213 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:03.493184    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pb52q"
Sep 11 19:38:03.495847 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:03.495821    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5xldn\""
Sep 11 19:38:03.505906 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:03.505882    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pb52q"]
Sep 11 19:38:03.590334 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:03.590304    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbkt6\" (UniqueName: \"kubernetes.io/projected/4c1f8e9a-0473-4b07-9e59-c43df138200f-kube-api-access-tbkt6\") pod \"kuadrant-operator-controller-manager-5f685c496f-pb52q\" (UID: \"4c1f8e9a-0473-4b07-9e59-c43df138200f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pb52q"
Sep 11 19:38:03.590334 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:03.590335    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c1f8e9a-0473-4b07-9e59-c43df138200f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-pb52q\" (UID: \"4c1f8e9a-0473-4b07-9e59-c43df138200f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pb52q"
Sep 11 19:38:03.602695 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:03.602669    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pb52q"]
Sep 11 19:38:03.602894 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:38:03.602876    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-tbkt6], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pb52q" podUID="4c1f8e9a-0473-4b07-9e59-c43df138200f"
Sep 11 19:38:03.690887 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:03.690862    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbkt6\" (UniqueName: \"kubernetes.io/projected/4c1f8e9a-0473-4b07-9e59-c43df138200f-kube-api-access-tbkt6\") pod \"kuadrant-operator-controller-manager-5f685c496f-pb52q\" (UID: \"4c1f8e9a-0473-4b07-9e59-c43df138200f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pb52q"
Sep 11 19:38:03.690991 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:03.690894    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c1f8e9a-0473-4b07-9e59-c43df138200f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-pb52q\" (UID: \"4c1f8e9a-0473-4b07-9e59-c43df138200f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pb52q"
Sep 11 19:38:03.691203 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:03.691188    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c1f8e9a-0473-4b07-9e59-c43df138200f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-pb52q\" (UID: \"4c1f8e9a-0473-4b07-9e59-c43df138200f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pb52q"
Sep 11 19:38:03.704675 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:03.704645    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbkt6\" (UniqueName: \"kubernetes.io/projected/4c1f8e9a-0473-4b07-9e59-c43df138200f-kube-api-access-tbkt6\") pod \"kuadrant-operator-controller-manager-5f685c496f-pb52q\" (UID: \"4c1f8e9a-0473-4b07-9e59-c43df138200f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pb52q"
Sep 11 19:38:04.077598 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.077568    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pb52q"]
Sep 11 19:38:04.088917 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.088890    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pb52q"]
Sep 11 19:38:04.155111 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.155079    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h"]
Sep 11 19:38:04.158369 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.158351    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h"
Sep 11 19:38:04.179027 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.179004    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h"]
Sep 11 19:38:04.296601 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.296568    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4778eeb-c99d-45aa-9653-2c5a81d6280e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nv76h\" (UID: \"b4778eeb-c99d-45aa-9653-2c5a81d6280e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h"
Sep 11 19:38:04.296823 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.296613    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vvd4\" (UniqueName: \"kubernetes.io/projected/b4778eeb-c99d-45aa-9653-2c5a81d6280e-kube-api-access-5vvd4\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nv76h\" (UID: \"b4778eeb-c99d-45aa-9653-2c5a81d6280e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h"
Sep 11 19:38:04.397573 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.397505    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4778eeb-c99d-45aa-9653-2c5a81d6280e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nv76h\" (UID: \"b4778eeb-c99d-45aa-9653-2c5a81d6280e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h"
Sep 11 19:38:04.397573 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.397547    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vvd4\" (UniqueName: \"kubernetes.io/projected/b4778eeb-c99d-45aa-9653-2c5a81d6280e-kube-api-access-5vvd4\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nv76h\" (UID: \"b4778eeb-c99d-45aa-9653-2c5a81d6280e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h"
Sep 11 19:38:04.397889 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.397866    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4778eeb-c99d-45aa-9653-2c5a81d6280e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nv76h\" (UID: \"b4778eeb-c99d-45aa-9653-2c5a81d6280e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h"
Sep 11 19:38:04.413336 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.413313    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vvd4\" (UniqueName: \"kubernetes.io/projected/b4778eeb-c99d-45aa-9653-2c5a81d6280e-kube-api-access-5vvd4\") pod \"kuadrant-operator-controller-manager-76f8f686bf-nv76h\" (UID: \"b4778eeb-c99d-45aa-9653-2c5a81d6280e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h"
Sep 11 19:38:04.468345 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.468321    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h"
Sep 11 19:38:04.498294 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.498269    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pb52q"
Sep 11 19:38:04.501333 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:38:04.501306    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-pb52q\" is forbidden: User \"system:node:ip-10-0-138-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-43.ec2.internal' and this object" podUID="4c1f8e9a-0473-4b07-9e59-c43df138200f" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pb52q"
Sep 11 19:38:04.509901 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.509881    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pb52q"
Sep 11 19:38:04.513188 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:38:04.513161    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-pb52q\" is forbidden: User \"system:node:ip-10-0-138-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-43.ec2.internal' and this object" podUID="4c1f8e9a-0473-4b07-9e59-c43df138200f" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pb52q"
Sep 11 19:38:04.599147 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.599114    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c1f8e9a-0473-4b07-9e59-c43df138200f-extensions-socket-volume\") pod \"4c1f8e9a-0473-4b07-9e59-c43df138200f\" (UID: \"4c1f8e9a-0473-4b07-9e59-c43df138200f\") "
Sep 11 19:38:04.599306 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.599225    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbkt6\" (UniqueName: \"kubernetes.io/projected/4c1f8e9a-0473-4b07-9e59-c43df138200f-kube-api-access-tbkt6\") pod \"4c1f8e9a-0473-4b07-9e59-c43df138200f\" (UID: \"4c1f8e9a-0473-4b07-9e59-c43df138200f\") "
Sep 11 19:38:04.599426 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.599398    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1f8e9a-0473-4b07-9e59-c43df138200f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4c1f8e9a-0473-4b07-9e59-c43df138200f" (UID: "4c1f8e9a-0473-4b07-9e59-c43df138200f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:38:04.601116 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.601095    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1f8e9a-0473-4b07-9e59-c43df138200f-kube-api-access-tbkt6" (OuterVolumeSpecName: "kube-api-access-tbkt6") pod "4c1f8e9a-0473-4b07-9e59-c43df138200f" (UID: "4c1f8e9a-0473-4b07-9e59-c43df138200f"). InnerVolumeSpecName "kube-api-access-tbkt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:38:04.700733 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.700695    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbkt6\" (UniqueName: \"kubernetes.io/projected/4c1f8e9a-0473-4b07-9e59-c43df138200f-kube-api-access-tbkt6\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:38:04.700733 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.700732    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4c1f8e9a-0473-4b07-9e59-c43df138200f-extensions-socket-volume\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:38:04.755955 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:04.755930    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h"]
Sep 11 19:38:04.759818 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:38:04.759790    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4778eeb_c99d_45aa_9653_2c5a81d6280e.slice/crio-0bd41215194882cd919ed59b4ad04cafaf2e04e6a3fb7670d3ac99410eca742b WatchSource:0}: Error finding container 0bd41215194882cd919ed59b4ad04cafaf2e04e6a3fb7670d3ac99410eca742b: Status 404 returned error can't find the container with id 0bd41215194882cd919ed59b4ad04cafaf2e04e6a3fb7670d3ac99410eca742b
Sep 11 19:38:05.505107 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:05.505076    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pb52q"
Sep 11 19:38:05.505540 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:05.505107    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h" event={"ID":"b4778eeb-c99d-45aa-9653-2c5a81d6280e","Type":"ContainerStarted","Data":"0bd41215194882cd919ed59b4ad04cafaf2e04e6a3fb7670d3ac99410eca742b"}
Sep 11 19:38:05.507372 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:38:05.507332    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-pb52q\" is forbidden: User \"system:node:ip-10-0-138-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-43.ec2.internal' and this object" podUID="4c1f8e9a-0473-4b07-9e59-c43df138200f" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pb52q"
Sep 11 19:38:05.514403 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:38:05.514371    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-pb52q\" is forbidden: User \"system:node:ip-10-0-138-43.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-43.ec2.internal' and this object" podUID="4c1f8e9a-0473-4b07-9e59-c43df138200f" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-pb52q"
Sep 11 19:38:05.813770 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:05.813674    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c1f8e9a-0473-4b07-9e59-c43df138200f" path="/var/lib/kubelet/pods/4c1f8e9a-0473-4b07-9e59-c43df138200f/volumes"
Sep 11 19:38:08.523467 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:08.523360    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h" event={"ID":"b4778eeb-c99d-45aa-9653-2c5a81d6280e","Type":"ContainerStarted","Data":"724e300cc56f7c3a1939c66fb378d17bd16911dd9b58205e0431480e5aed4551"}
Sep 11 19:38:08.523467 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:08.523454    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h"
Sep 11 19:38:08.543177 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:08.543131    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h" podStartSLOduration=1.035287191 podStartE2EDuration="4.543115905s" totalImagesPullingTime="3.507828714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:38:08.54171563 +0000 UTC m=+705.158885112" watchObservedRunningTime="2026-09-11 19:38:08.543115905 +0000 UTC m=+705.160285398"
Sep 11 19:38:16.145446 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:16.145413    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h"]
Sep 11 19:38:16.145986 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:16.145597    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h" podUID="b4778eeb-c99d-45aa-9653-2c5a81d6280e" containerName="manager" containerID="cri-o://724e300cc56f7c3a1939c66fb378d17bd16911dd9b58205e0431480e5aed4551" gracePeriod=10
Sep 11 19:38:16.147566 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:16.147540    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h"
Sep 11 19:38:16.387128 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:16.387103    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h"
Sep 11 19:38:16.392765 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:16.392739    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vvd4\" (UniqueName: \"kubernetes.io/projected/b4778eeb-c99d-45aa-9653-2c5a81d6280e-kube-api-access-5vvd4\") pod \"b4778eeb-c99d-45aa-9653-2c5a81d6280e\" (UID: \"b4778eeb-c99d-45aa-9653-2c5a81d6280e\") "
Sep 11 19:38:16.392886 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:16.392836    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4778eeb-c99d-45aa-9653-2c5a81d6280e-extensions-socket-volume\") pod \"b4778eeb-c99d-45aa-9653-2c5a81d6280e\" (UID: \"b4778eeb-c99d-45aa-9653-2c5a81d6280e\") "
Sep 11 19:38:16.393312 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:16.393286    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4778eeb-c99d-45aa-9653-2c5a81d6280e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b4778eeb-c99d-45aa-9653-2c5a81d6280e" (UID: "b4778eeb-c99d-45aa-9653-2c5a81d6280e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:38:16.394892 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:16.394864    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4778eeb-c99d-45aa-9653-2c5a81d6280e-kube-api-access-5vvd4" (OuterVolumeSpecName: "kube-api-access-5vvd4") pod "b4778eeb-c99d-45aa-9653-2c5a81d6280e" (UID: "b4778eeb-c99d-45aa-9653-2c5a81d6280e"). InnerVolumeSpecName "kube-api-access-5vvd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:38:16.493963 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:16.493926    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4778eeb-c99d-45aa-9653-2c5a81d6280e-extensions-socket-volume\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:38:16.493963 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:16.493954    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vvd4\" (UniqueName: \"kubernetes.io/projected/b4778eeb-c99d-45aa-9653-2c5a81d6280e-kube-api-access-5vvd4\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:38:16.561211 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:16.561177    2558 generic.go:356] "Generic (PLEG): container finished" podID="b4778eeb-c99d-45aa-9653-2c5a81d6280e" containerID="724e300cc56f7c3a1939c66fb378d17bd16911dd9b58205e0431480e5aed4551" exitCode=0
Sep 11 19:38:16.561378 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:16.561214    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h"
Sep 11 19:38:16.561378 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:16.561262    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h" event={"ID":"b4778eeb-c99d-45aa-9653-2c5a81d6280e","Type":"ContainerDied","Data":"724e300cc56f7c3a1939c66fb378d17bd16911dd9b58205e0431480e5aed4551"}
Sep 11 19:38:16.561378 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:16.561327    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h" event={"ID":"b4778eeb-c99d-45aa-9653-2c5a81d6280e","Type":"ContainerDied","Data":"0bd41215194882cd919ed59b4ad04cafaf2e04e6a3fb7670d3ac99410eca742b"}
Sep 11 19:38:16.561378 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:16.561358    2558 scope.go:117] "RemoveContainer" containerID="724e300cc56f7c3a1939c66fb378d17bd16911dd9b58205e0431480e5aed4551"
Sep 11 19:38:16.570009 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:16.569991    2558 scope.go:117] "RemoveContainer" containerID="724e300cc56f7c3a1939c66fb378d17bd16911dd9b58205e0431480e5aed4551"
Sep 11 19:38:16.570299 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:38:16.570280    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"724e300cc56f7c3a1939c66fb378d17bd16911dd9b58205e0431480e5aed4551\": container with ID starting with 724e300cc56f7c3a1939c66fb378d17bd16911dd9b58205e0431480e5aed4551 not found: ID does not exist" containerID="724e300cc56f7c3a1939c66fb378d17bd16911dd9b58205e0431480e5aed4551"
Sep 11 19:38:16.570362 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:16.570309    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"724e300cc56f7c3a1939c66fb378d17bd16911dd9b58205e0431480e5aed4551"} err="failed to get container status \"724e300cc56f7c3a1939c66fb378d17bd16911dd9b58205e0431480e5aed4551\": rpc error: code = NotFound desc = could not find container \"724e300cc56f7c3a1939c66fb378d17bd16911dd9b58205e0431480e5aed4551\": container with ID starting with 724e300cc56f7c3a1939c66fb378d17bd16911dd9b58205e0431480e5aed4551 not found: ID does not exist"
Sep 11 19:38:16.589578 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:16.589551    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h"]
Sep 11 19:38:16.600819 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:16.600794    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-nv76h"]
Sep 11 19:38:17.813296 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:17.813260    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4778eeb-c99d-45aa-9653-2c5a81d6280e" path="/var/lib/kubelet/pods/b4778eeb-c99d-45aa-9653-2c5a81d6280e/volumes"
Sep 11 19:38:32.442884 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.442844    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"]
Sep 11 19:38:32.443281 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.443143    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4778eeb-c99d-45aa-9653-2c5a81d6280e" containerName="manager"
Sep 11 19:38:32.443281 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.443153    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4778eeb-c99d-45aa-9653-2c5a81d6280e" containerName="manager"
Sep 11 19:38:32.443281 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.443204    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4778eeb-c99d-45aa-9653-2c5a81d6280e" containerName="manager"
Sep 11 19:38:32.447116 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.447098    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.449659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.449614    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 19:38:32.449816 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.449795    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 19:38:32.449869 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.449853    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 19:38:32.449952 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.449930    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2n29z\""
Sep 11 19:38:32.459016 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.458993    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"]
Sep 11 19:38:32.534561 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.534516    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.534561 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.534560    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.534837 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.534590    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5gf9\" (UniqueName: \"kubernetes.io/projected/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-kube-api-access-c5gf9\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.534837 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.534693    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.534837 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.534757    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.534837 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.534782    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.534837 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.534797    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.534837 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.534834    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.535048 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.534864    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.635767 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.635735    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.635767 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.635780    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.636011 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.635804    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.636011 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.635825    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5gf9\" (UniqueName: \"kubernetes.io/projected/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-kube-api-access-c5gf9\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.636011 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.635851    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.636011 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.635878    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.636011 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.635905    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.636011 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.635928    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.636011 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.635962    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.636394 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.636345    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.636515 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.636397    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.636515 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.636482    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.636629 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.636560    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.636768 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.636750    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.638025 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.638007    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.638511 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.638491    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.643962 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.643936    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.644295 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.644269    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5gf9\" (UniqueName: \"kubernetes.io/projected/c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba-kube-api-access-c5gf9\") pod \"maas-default-gateway-openshift-default-5ccb968c66-pqmwd\" (UID: \"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:32.757278 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:32.757183    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:33.032889 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:33.032859    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"]
Sep 11 19:38:33.036314 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:38:33.036274    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0eb5c20_3a6f_4c5c_b48b_14dab8bd23ba.slice/crio-fcaf9fb2436565b55d8a58630aeb3c0821f8162e03303e7ccfaee3d6059b440b WatchSource:0}: Error finding container fcaf9fb2436565b55d8a58630aeb3c0821f8162e03303e7ccfaee3d6059b440b: Status 404 returned error can't find the container with id fcaf9fb2436565b55d8a58630aeb3c0821f8162e03303e7ccfaee3d6059b440b
Sep 11 19:38:33.640944 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:33.640908    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd" event={"ID":"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba","Type":"ContainerStarted","Data":"fcaf9fb2436565b55d8a58630aeb3c0821f8162e03303e7ccfaee3d6059b440b"}
Sep 11 19:38:35.948273 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:35.948233    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:38:35.948514 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:35.948318    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:38:35.948514 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:35.948346    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:38:36.658265 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:36.658230    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd" event={"ID":"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba","Type":"ContainerStarted","Data":"3431d0af64ec35fffca5bba6aa3ad1ae8832463132192757bfae7a8590f36442"}
Sep 11 19:38:36.683468 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:36.683409    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd" podStartSLOduration=1.774571975 podStartE2EDuration="4.683393324s" totalImagesPullingTime="2.908821349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:38:36.678044347 +0000 UTC m=+733.295213839" watchObservedRunningTime="2026-09-11 19:38:36.683393324 +0000 UTC m=+733.300562818"
Sep 11 19:38:36.758039 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:36.757993    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:37.762648 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:37.762616    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:38.667024 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:38.666985    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:38.668231 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:38.668211    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:38:46.352561 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:46.352527    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v8swm"]
Sep 11 19:38:46.354938 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:46.354918    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-v8swm"
Sep 11 19:38:46.357320 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:46.357289    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 19:38:46.357431 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:46.357338    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-klrmj\""
Sep 11 19:38:46.365831 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:46.365806    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v8swm"]
Sep 11 19:38:46.447238 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:46.447203    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v8swm"]
Sep 11 19:38:46.457923 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:46.457888    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb4db\" (UniqueName: \"kubernetes.io/projected/e4c12426-77d3-4836-a107-9e59e743f53b-kube-api-access-hb4db\") pod \"limitador-limitador-8588749977-v8swm\" (UID: \"e4c12426-77d3-4836-a107-9e59e743f53b\") " pod="kuadrant-system/limitador-limitador-8588749977-v8swm"
Sep 11 19:38:46.458083 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:46.458018    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e4c12426-77d3-4836-a107-9e59e743f53b-config-file\") pod \"limitador-limitador-8588749977-v8swm\" (UID: \"e4c12426-77d3-4836-a107-9e59e743f53b\") " pod="kuadrant-system/limitador-limitador-8588749977-v8swm"
Sep 11 19:38:46.558994 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:46.558955    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e4c12426-77d3-4836-a107-9e59e743f53b-config-file\") pod \"limitador-limitador-8588749977-v8swm\" (UID: \"e4c12426-77d3-4836-a107-9e59e743f53b\") " pod="kuadrant-system/limitador-limitador-8588749977-v8swm"
Sep 11 19:38:46.559162 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:46.559018    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb4db\" (UniqueName: \"kubernetes.io/projected/e4c12426-77d3-4836-a107-9e59e743f53b-kube-api-access-hb4db\") pod \"limitador-limitador-8588749977-v8swm\" (UID: \"e4c12426-77d3-4836-a107-9e59e743f53b\") " pod="kuadrant-system/limitador-limitador-8588749977-v8swm"
Sep 11 19:38:46.559655 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:46.559633    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e4c12426-77d3-4836-a107-9e59e743f53b-config-file\") pod \"limitador-limitador-8588749977-v8swm\" (UID: \"e4c12426-77d3-4836-a107-9e59e743f53b\") " pod="kuadrant-system/limitador-limitador-8588749977-v8swm"
Sep 11 19:38:46.567977 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:46.567953    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb4db\" (UniqueName: \"kubernetes.io/projected/e4c12426-77d3-4836-a107-9e59e743f53b-kube-api-access-hb4db\") pod \"limitador-limitador-8588749977-v8swm\" (UID: \"e4c12426-77d3-4836-a107-9e59e743f53b\") " pod="kuadrant-system/limitador-limitador-8588749977-v8swm"
Sep 11 19:38:46.664698 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:46.664670    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-v8swm"
Sep 11 19:38:46.953275 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:46.953170    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v8swm"]
Sep 11 19:38:46.956025 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:38:46.955995    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c12426_77d3_4836_a107_9e59e743f53b.slice/crio-f09dee677e811f827b52c252740392e9aad81906c30363186d25574ee41e57aa WatchSource:0}: Error finding container f09dee677e811f827b52c252740392e9aad81906c30363186d25574ee41e57aa: Status 404 returned error can't find the container with id f09dee677e811f827b52c252740392e9aad81906c30363186d25574ee41e57aa
Sep 11 19:38:47.336346 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:47.336272    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-4r4tk"]
Sep 11 19:38:47.340479 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:47.340461    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-4r4tk"
Sep 11 19:38:47.342655 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:47.342633    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-v465p\""
Sep 11 19:38:47.348808 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:47.348782    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4r4tk"]
Sep 11 19:38:47.466915 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:47.466880    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdzpj\" (UniqueName: \"kubernetes.io/projected/3d61cad2-9894-4581-a433-0ee58b6357a1-kube-api-access-hdzpj\") pod \"authorino-59c974bd48-4r4tk\" (UID: \"3d61cad2-9894-4581-a433-0ee58b6357a1\") " pod="kuadrant-system/authorino-59c974bd48-4r4tk"
Sep 11 19:38:47.568076 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:47.568047    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdzpj\" (UniqueName: \"kubernetes.io/projected/3d61cad2-9894-4581-a433-0ee58b6357a1-kube-api-access-hdzpj\") pod \"authorino-59c974bd48-4r4tk\" (UID: \"3d61cad2-9894-4581-a433-0ee58b6357a1\") " pod="kuadrant-system/authorino-59c974bd48-4r4tk"
Sep 11 19:38:47.576446 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:47.576411    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdzpj\" (UniqueName: \"kubernetes.io/projected/3d61cad2-9894-4581-a433-0ee58b6357a1-kube-api-access-hdzpj\") pod \"authorino-59c974bd48-4r4tk\" (UID: \"3d61cad2-9894-4581-a433-0ee58b6357a1\") " pod="kuadrant-system/authorino-59c974bd48-4r4tk"
Sep 11 19:38:47.650829 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:47.650750    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-4r4tk"
Sep 11 19:38:47.708413 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:47.708381    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-v8swm" event={"ID":"e4c12426-77d3-4836-a107-9e59e743f53b","Type":"ContainerStarted","Data":"f09dee677e811f827b52c252740392e9aad81906c30363186d25574ee41e57aa"}
Sep 11 19:38:47.920356 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:47.920170    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4r4tk"]
Sep 11 19:38:48.055151 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:38:48.055112    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d61cad2_9894_4581_a433_0ee58b6357a1.slice/crio-7a330598a5005ffe7d401baceb084a2aaf3eeae2804087eeee4e622a5a0d286b WatchSource:0}: Error finding container 7a330598a5005ffe7d401baceb084a2aaf3eeae2804087eeee4e622a5a0d286b: Status 404 returned error can't find the container with id 7a330598a5005ffe7d401baceb084a2aaf3eeae2804087eeee4e622a5a0d286b
Sep 11 19:38:48.714790 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:48.714750    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-4r4tk" event={"ID":"3d61cad2-9894-4581-a433-0ee58b6357a1","Type":"ContainerStarted","Data":"7a330598a5005ffe7d401baceb084a2aaf3eeae2804087eeee4e622a5a0d286b"}
Sep 11 19:38:48.716632 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:48.716597    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-v8swm" event={"ID":"e4c12426-77d3-4836-a107-9e59e743f53b","Type":"ContainerStarted","Data":"d3f5369b9703dfbbe37a06503f24a3a26483a2d926061d13a4ad49c914939703"}
Sep 11 19:38:48.716788 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:48.716756    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-v8swm"
Sep 11 19:38:48.733445 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:48.733391    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-v8swm" podStartSLOduration=1.582297256 podStartE2EDuration="2.733375162s" totalImagesPullingTime="1.151077906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:38:48.732226018 +0000 UTC m=+745.349395509" watchObservedRunningTime="2026-09-11 19:38:48.733375162 +0000 UTC m=+745.350544653"
Sep 11 19:38:49.721888 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:49.721849    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-4r4tk" event={"ID":"3d61cad2-9894-4581-a433-0ee58b6357a1","Type":"ContainerStarted","Data":"e4e3e1528b15a65aaccd32f2ebf12f41eb2905f6a26321f38a76130332a1f5ae"}
Sep 11 19:38:49.738424 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:49.738374    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-4r4tk" podStartSLOduration=1.150457163 podStartE2EDuration="2.738356847s" totalImagesPullingTime="1.587899684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:38:49.736240904 +0000 UTC m=+746.353410395" watchObservedRunningTime="2026-09-11 19:38:49.738356847 +0000 UTC m=+746.355526340"
Sep 11 19:38:59.722970 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:38:59.722936    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-v8swm"
Sep 11 19:39:00.378838 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:00.378803    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v8swm"]
Sep 11 19:39:00.379006 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:00.378975    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-v8swm" podUID="e4c12426-77d3-4836-a107-9e59e743f53b" containerName="limitador" containerID="cri-o://d3f5369b9703dfbbe37a06503f24a3a26483a2d926061d13a4ad49c914939703" gracePeriod=30
Sep 11 19:39:00.770581 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:00.770538    2558 generic.go:356] "Generic (PLEG): container finished" podID="e4c12426-77d3-4836-a107-9e59e743f53b" containerID="d3f5369b9703dfbbe37a06503f24a3a26483a2d926061d13a4ad49c914939703" exitCode=0
Sep 11 19:39:00.770961 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:00.770611    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-v8swm" event={"ID":"e4c12426-77d3-4836-a107-9e59e743f53b","Type":"ContainerDied","Data":"d3f5369b9703dfbbe37a06503f24a3a26483a2d926061d13a4ad49c914939703"}
Sep 11 19:39:01.315547 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:01.315522    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-v8swm"
Sep 11 19:39:01.483758 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:01.483653    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb4db\" (UniqueName: \"kubernetes.io/projected/e4c12426-77d3-4836-a107-9e59e743f53b-kube-api-access-hb4db\") pod \"e4c12426-77d3-4836-a107-9e59e743f53b\" (UID: \"e4c12426-77d3-4836-a107-9e59e743f53b\") "
Sep 11 19:39:01.483758 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:01.483692    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e4c12426-77d3-4836-a107-9e59e743f53b-config-file\") pod \"e4c12426-77d3-4836-a107-9e59e743f53b\" (UID: \"e4c12426-77d3-4836-a107-9e59e743f53b\") "
Sep 11 19:39:01.484087 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:01.484064    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c12426-77d3-4836-a107-9e59e743f53b-config-file" (OuterVolumeSpecName: "config-file") pod "e4c12426-77d3-4836-a107-9e59e743f53b" (UID: "e4c12426-77d3-4836-a107-9e59e743f53b"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:39:01.485761 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:01.485735    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c12426-77d3-4836-a107-9e59e743f53b-kube-api-access-hb4db" (OuterVolumeSpecName: "kube-api-access-hb4db") pod "e4c12426-77d3-4836-a107-9e59e743f53b" (UID: "e4c12426-77d3-4836-a107-9e59e743f53b"). InnerVolumeSpecName "kube-api-access-hb4db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:39:01.584477 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:01.584424    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb4db\" (UniqueName: \"kubernetes.io/projected/e4c12426-77d3-4836-a107-9e59e743f53b-kube-api-access-hb4db\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:39:01.584477 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:01.584471    2558 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e4c12426-77d3-4836-a107-9e59e743f53b-config-file\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:39:01.776215 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:01.776130    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-v8swm" event={"ID":"e4c12426-77d3-4836-a107-9e59e743f53b","Type":"ContainerDied","Data":"f09dee677e811f827b52c252740392e9aad81906c30363186d25574ee41e57aa"}
Sep 11 19:39:01.776641 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:01.776225    2558 scope.go:117] "RemoveContainer" containerID="d3f5369b9703dfbbe37a06503f24a3a26483a2d926061d13a4ad49c914939703"
Sep 11 19:39:01.776641 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:01.776140    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-v8swm"
Sep 11 19:39:01.797393 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:01.797365    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v8swm"]
Sep 11 19:39:01.801317 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:01.801293    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-v8swm"]
Sep 11 19:39:01.812974 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:01.812950    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4c12426-77d3-4836-a107-9e59e743f53b" path="/var/lib/kubelet/pods/e4c12426-77d3-4836-a107-9e59e743f53b/volumes"
Sep 11 19:39:20.491446 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:39:20.491398    2558 controller.go:195] "Failed to update lease" err="Put \"https://a967f5d11ae1742838df16b28457a97d-ce4c780847635e77.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-138-43.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 11 19:39:30.491900 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:39:30.491843    2558 controller.go:195] "Failed to update lease" err="Put \"https://a967f5d11ae1742838df16b28457a97d-ce4c780847635e77.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-138-43.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 11 19:39:40.493061 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:39:40.492996    2558 controller.go:195] "Failed to update lease" err="Put \"https://a967f5d11ae1742838df16b28457a97d-ce4c780847635e77.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-138-43.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Sep 11 19:39:47.488471 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488354    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.488471 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488385    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.488471 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488377    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.488471 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488369    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.488471 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488395    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.488471 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488437    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.488471 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488400    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.488471 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488422    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.488471 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488464    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.489456 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488409    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6h8px\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.489456 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488503    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.489456 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488507    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.489456 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488552    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7kkw4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.489456 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488562    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.489456 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488569    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.489456 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488566    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.489456 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488589    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.489456 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488605    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.489456 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488618    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.489456 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488611    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.489456 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488636    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-sjcn9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.489456 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488650    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.489456 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488653    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.489456 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488666    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.489456 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488672    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.489456 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488685    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-dockercfg-vfkst\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.489456 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488701    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-82bfd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488735    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488702    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kkvg6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488755    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488762    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488752    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488768    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488783    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488791    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488816    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488823    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488829    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490088 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:39:47.488833    2558 controller.go:195] "Failed to update lease" err="Put \"https://a967f5d11ae1742838df16b28457a97d-ce4c780847635e77.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-138-43.ec2.internal?timeout=10s\": http2: client connection lost"
Sep 11 19:39:47.490088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488860    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488860    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488876    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7tun6f5d38rkr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488887    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8q9lb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488878    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488906    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488913    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488922    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488825    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488937    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488943    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488954    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488953    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488977    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488989    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x76hz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.488990    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489008    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489008    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489010    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489028    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489045    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lk6qt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489055    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rm699\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489071    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2rjxh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.490628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489084    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489086    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489096    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489110    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489119    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489127    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8sl4b\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489129    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489130    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489146    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-v465p\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489158    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pfbwm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489164    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489152    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489172    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489196    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489207    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489198    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489198    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489213    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"default-dockercfg-9kmr9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489228    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489215    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489255    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489273    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489287    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489301    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489304    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489314    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nfn9q\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489290    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489330    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489255    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sc9kv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489342    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489347    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489360    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489318    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489375    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489306    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kxtl7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489402    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k4fsc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.491668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489407    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2n29z\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492229 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489331    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ndnhc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492229 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489429    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492229 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489444    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492229 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489451    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492229 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489368    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492229 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489407    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492229 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489383    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492229 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489476    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5cl97\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492229 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489484    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-fg7w7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492229 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489491    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492229 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489455    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492229 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489505    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492229 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489509    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492229 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489527    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2f99f\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492229 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489531    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492229 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489537    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492229 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489499    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492229 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489412    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wnjht\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492746 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489556    2558 reflector.go:571] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492746 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489564    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492746 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489528    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492746 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489274    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p67zb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492746 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489288    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492746 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489587    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492746 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489566    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s6r98\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492746 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489589    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492746 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489599    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mtlf7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:47.492746 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:47.489452    2558 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-7xw7k\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Sep 11 19:39:48.297868 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.297830    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2f99f\""
Sep 11 19:39:48.299892 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.299867    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 19:39:48.300559 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.300543    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kxtl7\""
Sep 11 19:39:48.306920 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.306901    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 19:39:48.325464 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.325444    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vfkst\""
Sep 11 19:39:48.328880 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.328862    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 19:39:48.328880 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.328870    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 19:39:48.329793 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.329779    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 19:39:48.339449 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.339428    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nfn9q\""
Sep 11 19:39:48.344399 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.344384    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 19:39:48.348970 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.348950    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-fg7w7\""
Sep 11 19:39:48.357676 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.357660    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 19:39:48.366747 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.366727    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 19:39:48.383903 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.383886    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lk6qt\""
Sep 11 19:39:48.392572 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.392549    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 19:39:48.394059 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.394045    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x76hz\""
Sep 11 19:39:48.399466 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.399451    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 11 19:39:48.401315 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.401297    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 19:39:48.404631 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.404617    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 19:39:48.409025 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.409012    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 19:39:48.410849 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.410836    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-8sl4b\""
Sep 11 19:39:48.410913 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.410881    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 19:39:48.418342 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.418323    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ndnhc\""
Sep 11 19:39:48.421435 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.421419    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-7xw7k\""
Sep 11 19:39:48.421500 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.421421    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-sjcn9\""
Sep 11 19:39:48.422664 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.422647    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k4fsc\""
Sep 11 19:39:48.423999 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.423982    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 19:39:48.424241 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.424225    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 19:39:48.425737 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.425719    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 19:39:48.442367 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.442349    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p67zb\""
Sep 11 19:39:48.451774 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.451755    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 11 19:39:48.461004 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.460983    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 19:39:48.465583 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.465567    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 19:39:48.476050 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.476031    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pfbwm\""
Sep 11 19:39:48.479594 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.479578    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 11 19:39:48.483078 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.483060    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 19:39:48.500580 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.500551    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 19:39:48.503339 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.503322    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 19:39:48.506250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.506233    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 11 19:39:48.511719 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.511697    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 19:39:48.521474 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.521452    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 19:39:48.522435 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.522418    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 19:39:48.527184 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.527168    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 19:39:48.533571 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.533398    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 19:39:48.533571 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.533555    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 19:39:48.534000 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.533977    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 19:39:48.535433 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.535402    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 19:39:48.535701 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.535683    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 19:39:48.550102 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.550047    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mtlf7\""
Sep 11 19:39:48.553117 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.553100    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 11 19:39:48.554910 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.554890    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 19:39:48.557788 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.557770    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 19:39:48.560745 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.560728    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 19:39:48.560985 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.560972    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 11 19:39:48.562293 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.562277    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 19:39:48.563805 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.563791    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 19:39:48.569359 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.569343    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2n29z\""
Sep 11 19:39:48.584029 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.584003    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 19:39:48.586030 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.586013    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 19:39:48.591662 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.591647    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 19:39:48.607869 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.607849    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 19:39:48.621406 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.621386    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wnjht\""
Sep 11 19:39:48.631768 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.631748    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 19:39:48.643528 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.643510    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 19:39:48.654700 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.654681    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 19:39:48.662581 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.662555    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 19:39:48.670455 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.670436    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 19:39:48.671391 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.671378    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 19:39:48.673065 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.673050    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7tun6f5d38rkr\""
Sep 11 19:39:48.686066 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.686045    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 19:39:48.686882 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.686866    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 19:39:48.688918 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.688903    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 19:39:48.696647 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.696633    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7kkw4\""
Sep 11 19:39:48.702644 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.702629    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 19:39:48.708377 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.708364    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 19:39:48.714901 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.714886    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 19:39:48.716698 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.716685    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 19:39:48.725292 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.725275    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9kmr9\""
Sep 11 19:39:48.733742 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.733726    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 19:39:48.747406 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.747388    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 19:39:48.768769 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.768750    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 19:39:48.768837 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.768750    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kkvg6\""
Sep 11 19:39:48.789444 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.789422    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 19:39:48.795287 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.795270    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 19:39:48.797385 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.797367    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5cf57"]
Sep 11 19:39:48.797695 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.797684    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c12426-77d3-4836-a107-9e59e743f53b" containerName="limitador"
Sep 11 19:39:48.797750 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.797697    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c12426-77d3-4836-a107-9e59e743f53b" containerName="limitador"
Sep 11 19:39:48.797784 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.797768    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c12426-77d3-4836-a107-9e59e743f53b" containerName="limitador"
Sep 11 19:39:48.801858 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.801819    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5cf57"
Sep 11 19:39:48.804751 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.804731    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-plqzf\""
Sep 11 19:39:48.804751 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.804742    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 19:39:48.804902 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.804779    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 11 19:39:48.804902 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.804807    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 19:39:48.815504 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.815478    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 19:39:48.815747 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.815727    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 19:39:48.818082 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.818060    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 19:39:48.821389 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.821372    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 19:39:48.834402 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.834387    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 19:39:48.846852 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.846836    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 19:39:48.847678 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.847665    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 19:39:48.853044 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.853028    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6h8px\""
Sep 11 19:39:48.879063 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.879045    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 19:39:48.899375 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.899353    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5cl97\""
Sep 11 19:39:48.901239 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.901225    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 19:39:48.913326 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.913308    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-82bfd\""
Sep 11 19:39:48.921103 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.921083    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rm699\""
Sep 11 19:39:48.923968 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.923952    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 19:39:48.924903 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.924886    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 19:39:48.924963 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.924938    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 19:39:48.930637 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.930618    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 19:39:48.931604 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.931591    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 19:39:48.936049 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.936034    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 19:39:48.937113 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.937099    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2rjxh\""
Sep 11 19:39:48.938179 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.938164    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 19:39:48.941230 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.941216    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 19:39:48.942643 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.942630    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 19:39:48.965797 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.965779    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sc9kv\""
Sep 11 19:39:48.969987 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.969968    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 19:39:48.970074 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.970008    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 19:39:48.970510 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.970495    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 19:39:48.971750 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.971728    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-v465p\""
Sep 11 19:39:48.980304 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.980287    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 19:39:48.983704 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.983688    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 19:39:48.987802 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.987786    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54znc\" (UniqueName: \"kubernetes.io/projected/fbf3e31c-3f2a-474d-995c-83efc6611826-kube-api-access-54znc\") pod \"postgres-69fb4594cf-5cf57\" (UID: \"fbf3e31c-3f2a-474d-995c-83efc6611826\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5cf57"
Sep 11 19:39:48.987869 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.987846    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fbf3e31c-3f2a-474d-995c-83efc6611826-data\") pod \"postgres-69fb4594cf-5cf57\" (UID: \"fbf3e31c-3f2a-474d-995c-83efc6611826\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5cf57"
Sep 11 19:39:48.995064 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.995049    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s6r98\""
Sep 11 19:39:48.996592 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:48.996577    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8q9lb\""
Sep 11 19:39:49.005771 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:49.005755    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 19:39:49.007019 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:49.007006    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 19:39:49.012668 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:49.012651    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 19:39:49.015794 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:49.015779    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 19:39:49.028797 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:49.028780    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 19:39:49.038927 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:49.038906    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 19:39:49.063108 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:49.063043    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 19:39:49.068347 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:49.068330    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 19:39:49.078737 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:49.078697    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 19:39:49.079527 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:49.079512    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 19:39:49.088613 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:49.088589    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fbf3e31c-3f2a-474d-995c-83efc6611826-data\") pod \"postgres-69fb4594cf-5cf57\" (UID: \"fbf3e31c-3f2a-474d-995c-83efc6611826\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5cf57"
Sep 11 19:39:49.088684 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:49.088636    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54znc\" (UniqueName: \"kubernetes.io/projected/fbf3e31c-3f2a-474d-995c-83efc6611826-kube-api-access-54znc\") pod \"postgres-69fb4594cf-5cf57\" (UID: \"fbf3e31c-3f2a-474d-995c-83efc6611826\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5cf57"
Sep 11 19:39:49.088976 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:49.088960    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fbf3e31c-3f2a-474d-995c-83efc6611826-data\") pod \"postgres-69fb4594cf-5cf57\" (UID: \"fbf3e31c-3f2a-474d-995c-83efc6611826\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5cf57"
Sep 11 19:39:49.096663 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:49.096639    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54znc\" (UniqueName: \"kubernetes.io/projected/fbf3e31c-3f2a-474d-995c-83efc6611826-kube-api-access-54znc\") pod \"postgres-69fb4594cf-5cf57\" (UID: \"fbf3e31c-3f2a-474d-995c-83efc6611826\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-5cf57"
Sep 11 19:39:49.112187 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:49.112163    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5cf57"
Sep 11 19:39:49.381211 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:49.381143    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-5cf57"]
Sep 11 19:39:49.383433 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:39:49.383406    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf3e31c_3f2a_474d_995c_83efc6611826.slice/crio-20f96454c4edd343aacb61f54aff9186787e67b3e1e0e320f7576e6b44c79b66 WatchSource:0}: Error finding container 20f96454c4edd343aacb61f54aff9186787e67b3e1e0e320f7576e6b44c79b66: Status 404 returned error can't find the container with id 20f96454c4edd343aacb61f54aff9186787e67b3e1e0e320f7576e6b44c79b66
Sep 11 19:39:49.987688 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:49.987653    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5cf57" event={"ID":"fbf3e31c-3f2a-474d-995c-83efc6611826","Type":"ContainerStarted","Data":"20f96454c4edd343aacb61f54aff9186787e67b3e1e0e320f7576e6b44c79b66"}
Sep 11 19:39:55.012993 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:55.012951    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5cf57" event={"ID":"fbf3e31c-3f2a-474d-995c-83efc6611826","Type":"ContainerStarted","Data":"5a2d54128d5f9e142fd528150a8430203a4f5a4642208a1f295b49f20d47a00f"}
Sep 11 19:39:55.013358 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:55.013015    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5cf57"
Sep 11 19:39:55.028916 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:39:55.028865    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5cf57" podStartSLOduration=46.396814948 podStartE2EDuration="51.028850946s" totalImagesPullingTime="4.632035998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:39:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:39:55.027585157 +0000 UTC m=+811.644754673" watchObservedRunningTime="2026-09-11 19:39:55.028850946 +0000 UTC m=+811.646020436"
Sep 11 19:40:01.047560 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:01.047518    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-5cf57"
Sep 11 19:40:01.722577 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:01.722530    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-vvctt"]
Sep 11 19:40:01.727662 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:01.727637    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-vvctt"
Sep 11 19:40:01.730079 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:01.730052    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 11 19:40:01.732664 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:01.732637    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-vvctt"]
Sep 11 19:40:01.793593 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:01.793559    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf6ps\" (UniqueName: \"kubernetes.io/projected/58b1f475-6f3e-47a5-ba1b-16298c42ad06-kube-api-access-vf6ps\") pod \"authorino-6d4dfc649f-vvctt\" (UID: \"58b1f475-6f3e-47a5-ba1b-16298c42ad06\") " pod="kuadrant-system/authorino-6d4dfc649f-vvctt"
Sep 11 19:40:01.793786 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:01.793624    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/58b1f475-6f3e-47a5-ba1b-16298c42ad06-tls-cert\") pod \"authorino-6d4dfc649f-vvctt\" (UID: \"58b1f475-6f3e-47a5-ba1b-16298c42ad06\") " pod="kuadrant-system/authorino-6d4dfc649f-vvctt"
Sep 11 19:40:01.794638 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:01.794617    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-vvctt"]
Sep 11 19:40:01.794864 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:40:01.794844    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-vf6ps tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-vvctt" podUID="58b1f475-6f3e-47a5-ba1b-16298c42ad06"
Sep 11 19:40:01.894467 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:01.894428    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/58b1f475-6f3e-47a5-ba1b-16298c42ad06-tls-cert\") pod \"authorino-6d4dfc649f-vvctt\" (UID: \"58b1f475-6f3e-47a5-ba1b-16298c42ad06\") " pod="kuadrant-system/authorino-6d4dfc649f-vvctt"
Sep 11 19:40:01.894650 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:01.894500    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf6ps\" (UniqueName: \"kubernetes.io/projected/58b1f475-6f3e-47a5-ba1b-16298c42ad06-kube-api-access-vf6ps\") pod \"authorino-6d4dfc649f-vvctt\" (UID: \"58b1f475-6f3e-47a5-ba1b-16298c42ad06\") " pod="kuadrant-system/authorino-6d4dfc649f-vvctt"
Sep 11 19:40:01.896938 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:01.896915    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/58b1f475-6f3e-47a5-ba1b-16298c42ad06-tls-cert\") pod \"authorino-6d4dfc649f-vvctt\" (UID: \"58b1f475-6f3e-47a5-ba1b-16298c42ad06\") " pod="kuadrant-system/authorino-6d4dfc649f-vvctt"
Sep 11 19:40:01.902301 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:01.902278    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf6ps\" (UniqueName: \"kubernetes.io/projected/58b1f475-6f3e-47a5-ba1b-16298c42ad06-kube-api-access-vf6ps\") pod \"authorino-6d4dfc649f-vvctt\" (UID: \"58b1f475-6f3e-47a5-ba1b-16298c42ad06\") " pod="kuadrant-system/authorino-6d4dfc649f-vvctt"
Sep 11 19:40:02.043935 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:02.043850    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-vvctt"
Sep 11 19:40:02.051098 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:02.051074    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-vvctt"
Sep 11 19:40:02.096258 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:02.096228    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vf6ps\" (UniqueName: \"kubernetes.io/projected/58b1f475-6f3e-47a5-ba1b-16298c42ad06-kube-api-access-vf6ps\") pod \"58b1f475-6f3e-47a5-ba1b-16298c42ad06\" (UID: \"58b1f475-6f3e-47a5-ba1b-16298c42ad06\") "
Sep 11 19:40:02.096410 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:02.096264    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/58b1f475-6f3e-47a5-ba1b-16298c42ad06-tls-cert\") pod \"58b1f475-6f3e-47a5-ba1b-16298c42ad06\" (UID: \"58b1f475-6f3e-47a5-ba1b-16298c42ad06\") "
Sep 11 19:40:02.098089 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:02.098057    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58b1f475-6f3e-47a5-ba1b-16298c42ad06-kube-api-access-vf6ps" (OuterVolumeSpecName: "kube-api-access-vf6ps") pod "58b1f475-6f3e-47a5-ba1b-16298c42ad06" (UID: "58b1f475-6f3e-47a5-ba1b-16298c42ad06"). InnerVolumeSpecName "kube-api-access-vf6ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:40:02.098219 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:02.098193    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58b1f475-6f3e-47a5-ba1b-16298c42ad06-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "58b1f475-6f3e-47a5-ba1b-16298c42ad06" (UID: "58b1f475-6f3e-47a5-ba1b-16298c42ad06"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:40:02.197806 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:02.197775    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf6ps\" (UniqueName: \"kubernetes.io/projected/58b1f475-6f3e-47a5-ba1b-16298c42ad06-kube-api-access-vf6ps\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:40:02.197806 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:02.197801    2558 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/58b1f475-6f3e-47a5-ba1b-16298c42ad06-tls-cert\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:40:03.051656 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:03.051609    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-vvctt"
Sep 11 19:40:03.084598 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:03.084567    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-vvctt"]
Sep 11 19:40:03.088173 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:03.088143    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-vvctt"]
Sep 11 19:40:03.814498 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:03.814464    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58b1f475-6f3e-47a5-ba1b-16298c42ad06" path="/var/lib/kubelet/pods/58b1f475-6f3e-47a5-ba1b-16298c42ad06/volumes"
Sep 11 19:40:13.844296 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:13.844253    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4r4tk"]
Sep 11 19:40:13.844736 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:13.844453    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-4r4tk" podUID="3d61cad2-9894-4581-a433-0ee58b6357a1" containerName="authorino" containerID="cri-o://e4e3e1528b15a65aaccd32f2ebf12f41eb2905f6a26321f38a76130332a1f5ae" gracePeriod=30
Sep 11 19:40:14.090079 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:14.090050    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-4r4tk"
Sep 11 19:40:14.098543 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:14.098457    2558 generic.go:356] "Generic (PLEG): container finished" podID="3d61cad2-9894-4581-a433-0ee58b6357a1" containerID="e4e3e1528b15a65aaccd32f2ebf12f41eb2905f6a26321f38a76130332a1f5ae" exitCode=0
Sep 11 19:40:14.098543 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:14.098512    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-4r4tk"
Sep 11 19:40:14.098543 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:14.098511    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-4r4tk" event={"ID":"3d61cad2-9894-4581-a433-0ee58b6357a1","Type":"ContainerDied","Data":"e4e3e1528b15a65aaccd32f2ebf12f41eb2905f6a26321f38a76130332a1f5ae"}
Sep 11 19:40:14.098857 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:14.098571    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-4r4tk" event={"ID":"3d61cad2-9894-4581-a433-0ee58b6357a1","Type":"ContainerDied","Data":"7a330598a5005ffe7d401baceb084a2aaf3eeae2804087eeee4e622a5a0d286b"}
Sep 11 19:40:14.098857 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:14.098597    2558 scope.go:117] "RemoveContainer" containerID="e4e3e1528b15a65aaccd32f2ebf12f41eb2905f6a26321f38a76130332a1f5ae"
Sep 11 19:40:14.108179 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:14.108156    2558 scope.go:117] "RemoveContainer" containerID="e4e3e1528b15a65aaccd32f2ebf12f41eb2905f6a26321f38a76130332a1f5ae"
Sep 11 19:40:14.108544 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:40:14.108521    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e3e1528b15a65aaccd32f2ebf12f41eb2905f6a26321f38a76130332a1f5ae\": container with ID starting with e4e3e1528b15a65aaccd32f2ebf12f41eb2905f6a26321f38a76130332a1f5ae not found: ID does not exist" containerID="e4e3e1528b15a65aaccd32f2ebf12f41eb2905f6a26321f38a76130332a1f5ae"
Sep 11 19:40:14.108638 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:14.108552    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e3e1528b15a65aaccd32f2ebf12f41eb2905f6a26321f38a76130332a1f5ae"} err="failed to get container status \"e4e3e1528b15a65aaccd32f2ebf12f41eb2905f6a26321f38a76130332a1f5ae\": rpc error: code = NotFound desc = could not find container \"e4e3e1528b15a65aaccd32f2ebf12f41eb2905f6a26321f38a76130332a1f5ae\": container with ID starting with e4e3e1528b15a65aaccd32f2ebf12f41eb2905f6a26321f38a76130332a1f5ae not found: ID does not exist"
Sep 11 19:40:14.199326 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:14.199295    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdzpj\" (UniqueName: \"kubernetes.io/projected/3d61cad2-9894-4581-a433-0ee58b6357a1-kube-api-access-hdzpj\") pod \"3d61cad2-9894-4581-a433-0ee58b6357a1\" (UID: \"3d61cad2-9894-4581-a433-0ee58b6357a1\") "
Sep 11 19:40:14.201378 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:14.201350    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d61cad2-9894-4581-a433-0ee58b6357a1-kube-api-access-hdzpj" (OuterVolumeSpecName: "kube-api-access-hdzpj") pod "3d61cad2-9894-4581-a433-0ee58b6357a1" (UID: "3d61cad2-9894-4581-a433-0ee58b6357a1"). InnerVolumeSpecName "kube-api-access-hdzpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:40:14.300039 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:14.299998    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdzpj\" (UniqueName: \"kubernetes.io/projected/3d61cad2-9894-4581-a433-0ee58b6357a1-kube-api-access-hdzpj\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:40:14.430332 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:14.430300    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4r4tk"]
Sep 11 19:40:14.450558 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:14.450525    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-4r4tk"]
Sep 11 19:40:15.813078 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:15.813041    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d61cad2-9894-4581-a433-0ee58b6357a1" path="/var/lib/kubelet/pods/3d61cad2-9894-4581-a433-0ee58b6357a1/volumes"
Sep 11 19:40:18.668724 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.668679    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-85f7776c5c-rk6vz"]
Sep 11 19:40:18.669248 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.669227    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d61cad2-9894-4581-a433-0ee58b6357a1" containerName="authorino"
Sep 11 19:40:18.669347 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.669250    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d61cad2-9894-4581-a433-0ee58b6357a1" containerName="authorino"
Sep 11 19:40:18.669347 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.669342    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d61cad2-9894-4581-a433-0ee58b6357a1" containerName="authorino"
Sep 11 19:40:18.673939 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.673912    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85f7776c5c-rk6vz"
Sep 11 19:40:18.678434 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.678412    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 19:40:18.678612 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.678413    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 19:40:18.678612 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.678595    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-km7bf\""
Sep 11 19:40:18.679496 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.679474    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 19:40:18.687460 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.687434    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85f7776c5c-rk6vz"]
Sep 11 19:40:18.735644 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.735601    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f556401-fc0a-4e26-86a8-ad5b08ac2082-webhook-cert\") pod \"maas-controller-85f7776c5c-rk6vz\" (UID: \"6f556401-fc0a-4e26-86a8-ad5b08ac2082\") " pod="opendatahub/maas-controller-85f7776c5c-rk6vz"
Sep 11 19:40:18.735834 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.735655    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-289tm\" (UniqueName: \"kubernetes.io/projected/6f556401-fc0a-4e26-86a8-ad5b08ac2082-kube-api-access-289tm\") pod \"maas-controller-85f7776c5c-rk6vz\" (UID: \"6f556401-fc0a-4e26-86a8-ad5b08ac2082\") " pod="opendatahub/maas-controller-85f7776c5c-rk6vz"
Sep 11 19:40:18.735834 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.735743    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f556401-fc0a-4e26-86a8-ad5b08ac2082-metrics-certs\") pod \"maas-controller-85f7776c5c-rk6vz\" (UID: \"6f556401-fc0a-4e26-86a8-ad5b08ac2082\") " pod="opendatahub/maas-controller-85f7776c5c-rk6vz"
Sep 11 19:40:18.836504 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.836470    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-289tm\" (UniqueName: \"kubernetes.io/projected/6f556401-fc0a-4e26-86a8-ad5b08ac2082-kube-api-access-289tm\") pod \"maas-controller-85f7776c5c-rk6vz\" (UID: \"6f556401-fc0a-4e26-86a8-ad5b08ac2082\") " pod="opendatahub/maas-controller-85f7776c5c-rk6vz"
Sep 11 19:40:18.836703 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.836532    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f556401-fc0a-4e26-86a8-ad5b08ac2082-metrics-certs\") pod \"maas-controller-85f7776c5c-rk6vz\" (UID: \"6f556401-fc0a-4e26-86a8-ad5b08ac2082\") " pod="opendatahub/maas-controller-85f7776c5c-rk6vz"
Sep 11 19:40:18.836703 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.836659    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f556401-fc0a-4e26-86a8-ad5b08ac2082-webhook-cert\") pod \"maas-controller-85f7776c5c-rk6vz\" (UID: \"6f556401-fc0a-4e26-86a8-ad5b08ac2082\") " pod="opendatahub/maas-controller-85f7776c5c-rk6vz"
Sep 11 19:40:18.836856 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:40:18.836787    2558 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 11 19:40:18.836913 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:40:18.836857    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f556401-fc0a-4e26-86a8-ad5b08ac2082-webhook-cert podName:6f556401-fc0a-4e26-86a8-ad5b08ac2082 nodeName:}" failed. No retries permitted until 2026-09-11 19:40:19.336840744 +0000 UTC m=+835.954010224 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/6f556401-fc0a-4e26-86a8-ad5b08ac2082-webhook-cert") pod "maas-controller-85f7776c5c-rk6vz" (UID: "6f556401-fc0a-4e26-86a8-ad5b08ac2082") : secret "maas-controller-webhook-cert" not found
Sep 11 19:40:18.838038 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.838012    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56b74cbc4d-v298d"]
Sep 11 19:40:18.839118 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.839093    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f556401-fc0a-4e26-86a8-ad5b08ac2082-metrics-certs\") pod \"maas-controller-85f7776c5c-rk6vz\" (UID: \"6f556401-fc0a-4e26-86a8-ad5b08ac2082\") " pod="opendatahub/maas-controller-85f7776c5c-rk6vz"
Sep 11 19:40:18.841463 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.841443    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56b74cbc4d-v298d"
Sep 11 19:40:18.847741 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.847698    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-289tm\" (UniqueName: \"kubernetes.io/projected/6f556401-fc0a-4e26-86a8-ad5b08ac2082-kube-api-access-289tm\") pod \"maas-controller-85f7776c5c-rk6vz\" (UID: \"6f556401-fc0a-4e26-86a8-ad5b08ac2082\") " pod="opendatahub/maas-controller-85f7776c5c-rk6vz"
Sep 11 19:40:18.852050 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.852015    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56b74cbc4d-v298d"]
Sep 11 19:40:18.937823 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.937735    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f382032-fd78-40b4-b17f-3e0b530d1444-metrics-certs\") pod \"maas-controller-56b74cbc4d-v298d\" (UID: \"9f382032-fd78-40b4-b17f-3e0b530d1444\") " pod="opendatahub/maas-controller-56b74cbc4d-v298d"
Sep 11 19:40:18.937823 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.937799    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkqtr\" (UniqueName: \"kubernetes.io/projected/9f382032-fd78-40b4-b17f-3e0b530d1444-kube-api-access-wkqtr\") pod \"maas-controller-56b74cbc4d-v298d\" (UID: \"9f382032-fd78-40b4-b17f-3e0b530d1444\") " pod="opendatahub/maas-controller-56b74cbc4d-v298d"
Sep 11 19:40:18.938011 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:18.937886    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f382032-fd78-40b4-b17f-3e0b530d1444-webhook-cert\") pod \"maas-controller-56b74cbc4d-v298d\" (UID: \"9f382032-fd78-40b4-b17f-3e0b530d1444\") " pod="opendatahub/maas-controller-56b74cbc4d-v298d"
Sep 11 19:40:19.038910 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:19.038867    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f382032-fd78-40b4-b17f-3e0b530d1444-webhook-cert\") pod \"maas-controller-56b74cbc4d-v298d\" (UID: \"9f382032-fd78-40b4-b17f-3e0b530d1444\") " pod="opendatahub/maas-controller-56b74cbc4d-v298d"
Sep 11 19:40:19.039045 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:19.038942    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f382032-fd78-40b4-b17f-3e0b530d1444-metrics-certs\") pod \"maas-controller-56b74cbc4d-v298d\" (UID: \"9f382032-fd78-40b4-b17f-3e0b530d1444\") " pod="opendatahub/maas-controller-56b74cbc4d-v298d"
Sep 11 19:40:19.039045 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:19.038979    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkqtr\" (UniqueName: \"kubernetes.io/projected/9f382032-fd78-40b4-b17f-3e0b530d1444-kube-api-access-wkqtr\") pod \"maas-controller-56b74cbc4d-v298d\" (UID: \"9f382032-fd78-40b4-b17f-3e0b530d1444\") " pod="opendatahub/maas-controller-56b74cbc4d-v298d"
Sep 11 19:40:19.041319 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:19.041299    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f382032-fd78-40b4-b17f-3e0b530d1444-metrics-certs\") pod \"maas-controller-56b74cbc4d-v298d\" (UID: \"9f382032-fd78-40b4-b17f-3e0b530d1444\") " pod="opendatahub/maas-controller-56b74cbc4d-v298d"
Sep 11 19:40:19.041396 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:19.041376    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f382032-fd78-40b4-b17f-3e0b530d1444-webhook-cert\") pod \"maas-controller-56b74cbc4d-v298d\" (UID: \"9f382032-fd78-40b4-b17f-3e0b530d1444\") " pod="opendatahub/maas-controller-56b74cbc4d-v298d"
Sep 11 19:40:19.047911 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:19.047880    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkqtr\" (UniqueName: \"kubernetes.io/projected/9f382032-fd78-40b4-b17f-3e0b530d1444-kube-api-access-wkqtr\") pod \"maas-controller-56b74cbc4d-v298d\" (UID: \"9f382032-fd78-40b4-b17f-3e0b530d1444\") " pod="opendatahub/maas-controller-56b74cbc4d-v298d"
Sep 11 19:40:19.162608 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:19.162541    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56b74cbc4d-v298d"
Sep 11 19:40:19.340787 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:19.340745    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f556401-fc0a-4e26-86a8-ad5b08ac2082-webhook-cert\") pod \"maas-controller-85f7776c5c-rk6vz\" (UID: \"6f556401-fc0a-4e26-86a8-ad5b08ac2082\") " pod="opendatahub/maas-controller-85f7776c5c-rk6vz"
Sep 11 19:40:19.343117 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:19.343094    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f556401-fc0a-4e26-86a8-ad5b08ac2082-webhook-cert\") pod \"maas-controller-85f7776c5c-rk6vz\" (UID: \"6f556401-fc0a-4e26-86a8-ad5b08ac2082\") " pod="opendatahub/maas-controller-85f7776c5c-rk6vz"
Sep 11 19:40:19.434609 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:19.434581    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56b74cbc4d-v298d"]
Sep 11 19:40:19.436794 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:40:19.436765    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f382032_fd78_40b4_b17f_3e0b530d1444.slice/crio-0f6050295cd7b31f15a1a97de982272a0b476ea1bdc23f04378355f40ad57a27 WatchSource:0}: Error finding container 0f6050295cd7b31f15a1a97de982272a0b476ea1bdc23f04378355f40ad57a27: Status 404 returned error can't find the container with id 0f6050295cd7b31f15a1a97de982272a0b476ea1bdc23f04378355f40ad57a27
Sep 11 19:40:19.584589 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:19.584492    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85f7776c5c-rk6vz"
Sep 11 19:40:19.856575 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:19.856545    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85f7776c5c-rk6vz"]
Sep 11 19:40:19.858046 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:40:19.858017    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f556401_fc0a_4e26_86a8_ad5b08ac2082.slice/crio-50dc040c6e8636d1773430e688af53414256cf38841289c9a9cff75163a82fff WatchSource:0}: Error finding container 50dc040c6e8636d1773430e688af53414256cf38841289c9a9cff75163a82fff: Status 404 returned error can't find the container with id 50dc040c6e8636d1773430e688af53414256cf38841289c9a9cff75163a82fff
Sep 11 19:40:20.127788 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:20.127672    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56b74cbc4d-v298d" event={"ID":"9f382032-fd78-40b4-b17f-3e0b530d1444","Type":"ContainerStarted","Data":"0f6050295cd7b31f15a1a97de982272a0b476ea1bdc23f04378355f40ad57a27"}
Sep 11 19:40:20.129199 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:20.129141    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85f7776c5c-rk6vz" event={"ID":"6f556401-fc0a-4e26-86a8-ad5b08ac2082","Type":"ContainerStarted","Data":"50dc040c6e8636d1773430e688af53414256cf38841289c9a9cff75163a82fff"}
Sep 11 19:40:23.145234 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:23.145188    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85f7776c5c-rk6vz" event={"ID":"6f556401-fc0a-4e26-86a8-ad5b08ac2082","Type":"ContainerStarted","Data":"33bc22f1b8a0cc56fe03bd9e3a0253bfd35a1b43e1c6d57bb4ecb137b79ff46a"}
Sep 11 19:40:23.145655 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:23.145358    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-85f7776c5c-rk6vz"
Sep 11 19:40:23.146779 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:23.146753    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56b74cbc4d-v298d" event={"ID":"9f382032-fd78-40b4-b17f-3e0b530d1444","Type":"ContainerStarted","Data":"e9b1f0ccfaba82ec3ff415f7c7be7adeffb3885fff509c1f6146b65823626024"}
Sep 11 19:40:23.146894 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:23.146883    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56b74cbc4d-v298d"
Sep 11 19:40:23.164687 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:23.164637    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-85f7776c5c-rk6vz" podStartSLOduration=2.915700733 podStartE2EDuration="5.164624312s" totalImagesPullingTime="2.248923579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:40:23.163939908 +0000 UTC m=+839.781109400" watchObservedRunningTime="2026-09-11 19:40:23.164624312 +0000 UTC m=+839.781793802"
Sep 11 19:40:23.189168 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:23.189119    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-56b74cbc4d-v298d" podStartSLOduration=2.528892802 podStartE2EDuration="5.18910464s" totalImagesPullingTime="2.660211838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:40:23.188468487 +0000 UTC m=+839.805637991" watchObservedRunningTime="2026-09-11 19:40:23.18910464 +0000 UTC m=+839.806274131"
Sep 11 19:40:34.152903 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:34.152859    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56b74cbc4d-v298d"
Sep 11 19:40:34.153453 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:34.153001    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-85f7776c5c-rk6vz"
Sep 11 19:40:34.257135 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:34.257103    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85f7776c5c-rk6vz"]
Sep 11 19:40:34.257334 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:34.257309    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-85f7776c5c-rk6vz" podUID="6f556401-fc0a-4e26-86a8-ad5b08ac2082" containerName="manager" containerID="cri-o://33bc22f1b8a0cc56fe03bd9e3a0253bfd35a1b43e1c6d57bb4ecb137b79ff46a" gracePeriod=30
Sep 11 19:40:34.493523 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:34.493497    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85f7776c5c-rk6vz"
Sep 11 19:40:34.579911 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:34.579867    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f556401-fc0a-4e26-86a8-ad5b08ac2082-webhook-cert\") pod \"6f556401-fc0a-4e26-86a8-ad5b08ac2082\" (UID: \"6f556401-fc0a-4e26-86a8-ad5b08ac2082\") "
Sep 11 19:40:34.580079 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:34.579967    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-289tm\" (UniqueName: \"kubernetes.io/projected/6f556401-fc0a-4e26-86a8-ad5b08ac2082-kube-api-access-289tm\") pod \"6f556401-fc0a-4e26-86a8-ad5b08ac2082\" (UID: \"6f556401-fc0a-4e26-86a8-ad5b08ac2082\") "
Sep 11 19:40:34.580079 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:34.580050    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f556401-fc0a-4e26-86a8-ad5b08ac2082-metrics-certs\") pod \"6f556401-fc0a-4e26-86a8-ad5b08ac2082\" (UID: \"6f556401-fc0a-4e26-86a8-ad5b08ac2082\") "
Sep 11 19:40:34.582192 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:34.582156    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f556401-fc0a-4e26-86a8-ad5b08ac2082-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6f556401-fc0a-4e26-86a8-ad5b08ac2082" (UID: "6f556401-fc0a-4e26-86a8-ad5b08ac2082"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:40:34.582327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:34.582260    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f556401-fc0a-4e26-86a8-ad5b08ac2082-kube-api-access-289tm" (OuterVolumeSpecName: "kube-api-access-289tm") pod "6f556401-fc0a-4e26-86a8-ad5b08ac2082" (UID: "6f556401-fc0a-4e26-86a8-ad5b08ac2082"). InnerVolumeSpecName "kube-api-access-289tm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:40:34.582327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:34.582271    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f556401-fc0a-4e26-86a8-ad5b08ac2082-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "6f556401-fc0a-4e26-86a8-ad5b08ac2082" (UID: "6f556401-fc0a-4e26-86a8-ad5b08ac2082"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:40:34.681302 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:34.681212    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f556401-fc0a-4e26-86a8-ad5b08ac2082-metrics-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:40:34.681302 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:34.681248    2558 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6f556401-fc0a-4e26-86a8-ad5b08ac2082-webhook-cert\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:40:34.681302 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:34.681260    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-289tm\" (UniqueName: \"kubernetes.io/projected/6f556401-fc0a-4e26-86a8-ad5b08ac2082-kube-api-access-289tm\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:40:35.199862 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:35.199831    2558 generic.go:356] "Generic (PLEG): container finished" podID="6f556401-fc0a-4e26-86a8-ad5b08ac2082" containerID="33bc22f1b8a0cc56fe03bd9e3a0253bfd35a1b43e1c6d57bb4ecb137b79ff46a" exitCode=0
Sep 11 19:40:35.200306 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:35.199880    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85f7776c5c-rk6vz"
Sep 11 19:40:35.200306 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:35.199922    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85f7776c5c-rk6vz" event={"ID":"6f556401-fc0a-4e26-86a8-ad5b08ac2082","Type":"ContainerDied","Data":"33bc22f1b8a0cc56fe03bd9e3a0253bfd35a1b43e1c6d57bb4ecb137b79ff46a"}
Sep 11 19:40:35.200306 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:35.199984    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85f7776c5c-rk6vz" event={"ID":"6f556401-fc0a-4e26-86a8-ad5b08ac2082","Type":"ContainerDied","Data":"50dc040c6e8636d1773430e688af53414256cf38841289c9a9cff75163a82fff"}
Sep 11 19:40:35.200306 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:35.200023    2558 scope.go:117] "RemoveContainer" containerID="33bc22f1b8a0cc56fe03bd9e3a0253bfd35a1b43e1c6d57bb4ecb137b79ff46a"
Sep 11 19:40:35.208474 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:35.208456    2558 scope.go:117] "RemoveContainer" containerID="33bc22f1b8a0cc56fe03bd9e3a0253bfd35a1b43e1c6d57bb4ecb137b79ff46a"
Sep 11 19:40:35.208753 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:40:35.208734    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33bc22f1b8a0cc56fe03bd9e3a0253bfd35a1b43e1c6d57bb4ecb137b79ff46a\": container with ID starting with 33bc22f1b8a0cc56fe03bd9e3a0253bfd35a1b43e1c6d57bb4ecb137b79ff46a not found: ID does not exist" containerID="33bc22f1b8a0cc56fe03bd9e3a0253bfd35a1b43e1c6d57bb4ecb137b79ff46a"
Sep 11 19:40:35.208819 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:35.208764    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33bc22f1b8a0cc56fe03bd9e3a0253bfd35a1b43e1c6d57bb4ecb137b79ff46a"} err="failed to get container status \"33bc22f1b8a0cc56fe03bd9e3a0253bfd35a1b43e1c6d57bb4ecb137b79ff46a\": rpc error: code = NotFound desc = could not find container \"33bc22f1b8a0cc56fe03bd9e3a0253bfd35a1b43e1c6d57bb4ecb137b79ff46a\": container with ID starting with 33bc22f1b8a0cc56fe03bd9e3a0253bfd35a1b43e1c6d57bb4ecb137b79ff46a not found: ID does not exist"
Sep 11 19:40:35.220532 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:35.220507    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85f7776c5c-rk6vz"]
Sep 11 19:40:35.224353 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:35.224329    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-85f7776c5c-rk6vz"]
Sep 11 19:40:35.812927 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:35.812893    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f556401-fc0a-4e26-86a8-ad5b08ac2082" path="/var/lib/kubelet/pods/6f556401-fc0a-4e26-86a8-ad5b08ac2082/volumes"
Sep 11 19:40:57.813478 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.813446    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"]
Sep 11 19:40:57.813878 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.813772    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f556401-fc0a-4e26-86a8-ad5b08ac2082" containerName="manager"
Sep 11 19:40:57.813878 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.813783    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f556401-fc0a-4e26-86a8-ad5b08ac2082" containerName="manager"
Sep 11 19:40:57.813878 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.813837    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f556401-fc0a-4e26-86a8-ad5b08ac2082" containerName="manager"
Sep 11 19:40:57.819513 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.819492    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"
Sep 11 19:40:57.822760 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.822739    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 19:40:57.822856 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.822743    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 19:40:57.823745 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.823702    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hvn79\""
Sep 11 19:40:57.824448 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.824427    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"]
Sep 11 19:40:57.871301 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.871269    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5qsg\" (UniqueName: \"kubernetes.io/projected/8224af77-d328-4adb-a8aa-b53aa915e55b-kube-api-access-d5qsg\") pod \"maas-api-ccb8f7bdd-r55jx\" (UID: \"8224af77-d328-4adb-a8aa-b53aa915e55b\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"
Sep 11 19:40:57.871461 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.871319    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8224af77-d328-4adb-a8aa-b53aa915e55b-openshift-service-ca\") pod \"maas-api-ccb8f7bdd-r55jx\" (UID: \"8224af77-d328-4adb-a8aa-b53aa915e55b\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"
Sep 11 19:40:57.871461 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.871353    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8224af77-d328-4adb-a8aa-b53aa915e55b-maas-api-tls\") pod \"maas-api-ccb8f7bdd-r55jx\" (UID: \"8224af77-d328-4adb-a8aa-b53aa915e55b\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"
Sep 11 19:40:57.871461 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.871387    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8224af77-d328-4adb-a8aa-b53aa915e55b-metrics-certs\") pod \"maas-api-ccb8f7bdd-r55jx\" (UID: \"8224af77-d328-4adb-a8aa-b53aa915e55b\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"
Sep 11 19:40:57.972428 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.972388    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8224af77-d328-4adb-a8aa-b53aa915e55b-openshift-service-ca\") pod \"maas-api-ccb8f7bdd-r55jx\" (UID: \"8224af77-d328-4adb-a8aa-b53aa915e55b\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"
Sep 11 19:40:57.972615 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.972436    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8224af77-d328-4adb-a8aa-b53aa915e55b-maas-api-tls\") pod \"maas-api-ccb8f7bdd-r55jx\" (UID: \"8224af77-d328-4adb-a8aa-b53aa915e55b\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"
Sep 11 19:40:57.972615 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.972467    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8224af77-d328-4adb-a8aa-b53aa915e55b-metrics-certs\") pod \"maas-api-ccb8f7bdd-r55jx\" (UID: \"8224af77-d328-4adb-a8aa-b53aa915e55b\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"
Sep 11 19:40:57.972615 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.972539    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qsg\" (UniqueName: \"kubernetes.io/projected/8224af77-d328-4adb-a8aa-b53aa915e55b-kube-api-access-d5qsg\") pod \"maas-api-ccb8f7bdd-r55jx\" (UID: \"8224af77-d328-4adb-a8aa-b53aa915e55b\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"
Sep 11 19:40:57.973160 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.973137    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8224af77-d328-4adb-a8aa-b53aa915e55b-openshift-service-ca\") pod \"maas-api-ccb8f7bdd-r55jx\" (UID: \"8224af77-d328-4adb-a8aa-b53aa915e55b\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"
Sep 11 19:40:57.974983 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.974953    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8224af77-d328-4adb-a8aa-b53aa915e55b-metrics-certs\") pod \"maas-api-ccb8f7bdd-r55jx\" (UID: \"8224af77-d328-4adb-a8aa-b53aa915e55b\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"
Sep 11 19:40:57.975067 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.975038    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8224af77-d328-4adb-a8aa-b53aa915e55b-maas-api-tls\") pod \"maas-api-ccb8f7bdd-r55jx\" (UID: \"8224af77-d328-4adb-a8aa-b53aa915e55b\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"
Sep 11 19:40:57.980744 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:57.980722    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5qsg\" (UniqueName: \"kubernetes.io/projected/8224af77-d328-4adb-a8aa-b53aa915e55b-kube-api-access-d5qsg\") pod \"maas-api-ccb8f7bdd-r55jx\" (UID: \"8224af77-d328-4adb-a8aa-b53aa915e55b\") " pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"
Sep 11 19:40:58.131742 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.131649    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"
Sep 11 19:40:58.382038 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.381955    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-xptvp"]
Sep 11 19:40:58.385965 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.385943    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-xptvp"
Sep 11 19:40:58.388617 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.388594    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6rkt9\""
Sep 11 19:40:58.388617 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.388613    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 19:40:58.397684 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.397662    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-xptvp"]
Sep 11 19:40:58.409995 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.409970    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"]
Sep 11 19:40:58.411872 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:40:58.411843    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8224af77_d328_4adb_a8aa_b53aa915e55b.slice/crio-601f0b74258f75b9c4a83d3276664f6eb024c241152a79ec821425b495ccfa18 WatchSource:0}: Error finding container 601f0b74258f75b9c4a83d3276664f6eb024c241152a79ec821425b495ccfa18: Status 404 returned error can't find the container with id 601f0b74258f75b9c4a83d3276664f6eb024c241152a79ec821425b495ccfa18
Sep 11 19:40:58.441092 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.441062    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-wkf6k"]
Sep 11 19:40:58.444236 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.444221    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wkf6k"
Sep 11 19:40:58.455174 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.455148    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-wkf6k"]
Sep 11 19:40:58.476913 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.476885    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lps89\" (UniqueName: \"kubernetes.io/projected/cb31f235-193f-4244-9c95-06e347d981ee-kube-api-access-lps89\") pod \"payload-processing-5b54b7fbb7-xptvp\" (UID: \"cb31f235-193f-4244-9c95-06e347d981ee\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xptvp"
Sep 11 19:40:58.477056 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.476931    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f7cec5-65d3-4c19-a677-50f6e7969695-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-wkf6k\" (UID: \"33f7cec5-65d3-4c19-a677-50f6e7969695\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wkf6k"
Sep 11 19:40:58.477056 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.476959    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzbk5\" (UniqueName: \"kubernetes.io/projected/33f7cec5-65d3-4c19-a677-50f6e7969695-kube-api-access-xzbk5\") pod \"payload-pre-processing-59fbc47bf8-wkf6k\" (UID: \"33f7cec5-65d3-4c19-a677-50f6e7969695\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wkf6k"
Sep 11 19:40:58.477056 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.477036    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb31f235-193f-4244-9c95-06e347d981ee-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-xptvp\" (UID: \"cb31f235-193f-4244-9c95-06e347d981ee\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xptvp"
Sep 11 19:40:58.578464 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.578423    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzbk5\" (UniqueName: \"kubernetes.io/projected/33f7cec5-65d3-4c19-a677-50f6e7969695-kube-api-access-xzbk5\") pod \"payload-pre-processing-59fbc47bf8-wkf6k\" (UID: \"33f7cec5-65d3-4c19-a677-50f6e7969695\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wkf6k"
Sep 11 19:40:58.578663 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.578490    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb31f235-193f-4244-9c95-06e347d981ee-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-xptvp\" (UID: \"cb31f235-193f-4244-9c95-06e347d981ee\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xptvp"
Sep 11 19:40:58.578663 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.578570    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lps89\" (UniqueName: \"kubernetes.io/projected/cb31f235-193f-4244-9c95-06e347d981ee-kube-api-access-lps89\") pod \"payload-processing-5b54b7fbb7-xptvp\" (UID: \"cb31f235-193f-4244-9c95-06e347d981ee\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xptvp"
Sep 11 19:40:58.578663 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.578622    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f7cec5-65d3-4c19-a677-50f6e7969695-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-wkf6k\" (UID: \"33f7cec5-65d3-4c19-a677-50f6e7969695\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wkf6k"
Sep 11 19:40:58.579237 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.579211    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb31f235-193f-4244-9c95-06e347d981ee-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-xptvp\" (UID: \"cb31f235-193f-4244-9c95-06e347d981ee\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xptvp"
Sep 11 19:40:58.579370 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.579350    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f7cec5-65d3-4c19-a677-50f6e7969695-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-wkf6k\" (UID: \"33f7cec5-65d3-4c19-a677-50f6e7969695\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wkf6k"
Sep 11 19:40:58.590755 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.590730    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzbk5\" (UniqueName: \"kubernetes.io/projected/33f7cec5-65d3-4c19-a677-50f6e7969695-kube-api-access-xzbk5\") pod \"payload-pre-processing-59fbc47bf8-wkf6k\" (UID: \"33f7cec5-65d3-4c19-a677-50f6e7969695\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wkf6k"
Sep 11 19:40:58.590850 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.590766    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lps89\" (UniqueName: \"kubernetes.io/projected/cb31f235-193f-4244-9c95-06e347d981ee-kube-api-access-lps89\") pod \"payload-processing-5b54b7fbb7-xptvp\" (UID: \"cb31f235-193f-4244-9c95-06e347d981ee\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xptvp"
Sep 11 19:40:58.697347 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.697298    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-xptvp"
Sep 11 19:40:58.755247 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.754888    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wkf6k"
Sep 11 19:40:58.992795 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:58.992752    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-xptvp"]
Sep 11 19:40:58.993416 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:40:58.993391    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb31f235_193f_4244_9c95_06e347d981ee.slice/crio-010cc7c988d5b48fa4cf20c249b526784634fd40b760742f9c434860f602451c WatchSource:0}: Error finding container 010cc7c988d5b48fa4cf20c249b526784634fd40b760742f9c434860f602451c: Status 404 returned error can't find the container with id 010cc7c988d5b48fa4cf20c249b526784634fd40b760742f9c434860f602451c
Sep 11 19:40:59.034230 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:59.034200    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-wkf6k"]
Sep 11 19:40:59.055609 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:40:59.055565    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f7cec5_65d3_4c19_a677_50f6e7969695.slice/crio-8472094fd4db4a494ad0da51a2f75d3658ff1bff2c3a2dec314bc43b880445ab WatchSource:0}: Error finding container 8472094fd4db4a494ad0da51a2f75d3658ff1bff2c3a2dec314bc43b880445ab: Status 404 returned error can't find the container with id 8472094fd4db4a494ad0da51a2f75d3658ff1bff2c3a2dec314bc43b880445ab
Sep 11 19:40:59.316159 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:59.316072    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wkf6k" event={"ID":"33f7cec5-65d3-4c19-a677-50f6e7969695","Type":"ContainerStarted","Data":"8472094fd4db4a494ad0da51a2f75d3658ff1bff2c3a2dec314bc43b880445ab"}
Sep 11 19:40:59.317251 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:59.317222    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-xptvp" event={"ID":"cb31f235-193f-4244-9c95-06e347d981ee","Type":"ContainerStarted","Data":"010cc7c988d5b48fa4cf20c249b526784634fd40b760742f9c434860f602451c"}
Sep 11 19:40:59.318214 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:40:59.318193    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx" event={"ID":"8224af77-d328-4adb-a8aa-b53aa915e55b","Type":"ContainerStarted","Data":"601f0b74258f75b9c4a83d3276664f6eb024c241152a79ec821425b495ccfa18"}
Sep 11 19:41:01.334847 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:01.334745    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx" event={"ID":"8224af77-d328-4adb-a8aa-b53aa915e55b","Type":"ContainerStarted","Data":"153b8e1823bc0ebaaff70fc5ac8821003f1dc408ad7ae647a69b603bee4eda0d"}
Sep 11 19:41:01.335310 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:01.334980    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"
Sep 11 19:41:01.369048 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:01.368989    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx" podStartSLOduration=1.792189791 podStartE2EDuration="4.368967332s" totalImagesPullingTime="2.576777541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:40:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:41:01.366036462 +0000 UTC m=+877.983205953" watchObservedRunningTime="2026-09-11 19:41:01.368967332 +0000 UTC m=+877.986136827"
Sep 11 19:41:03.366979 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:03.366937    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wkf6k" event={"ID":"33f7cec5-65d3-4c19-a677-50f6e7969695","Type":"ContainerStarted","Data":"eefed0365d834329479c21aa3621c20e7db504c3cd66dde9b1c281e3153be66e"}
Sep 11 19:41:03.367620 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:03.367601    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wkf6k"
Sep 11 19:41:03.369491 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:03.369463    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-xptvp" event={"ID":"cb31f235-193f-4244-9c95-06e347d981ee","Type":"ContainerStarted","Data":"297ca7131b0b94bf0fb520b24c91e113d86506d07896a12a910a9d1e7e7b7d70"}
Sep 11 19:41:03.369611 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:03.369534    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-xptvp"
Sep 11 19:41:03.387861 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:03.387808    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wkf6k" podStartSLOduration=1.956482756 podStartE2EDuration="5.387792744s" totalImagesPullingTime="3.431309988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:41:03.385854622 +0000 UTC m=+880.003024118" watchObservedRunningTime="2026-09-11 19:41:03.387792744 +0000 UTC m=+880.004962226"
Sep 11 19:41:03.406288 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:03.406221    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-xptvp" podStartSLOduration=1.902395453 podStartE2EDuration="5.406202982s" totalImagesPullingTime="3.503807529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:41:03.405577135 +0000 UTC m=+880.022746625" watchObservedRunningTime="2026-09-11 19:41:03.406202982 +0000 UTC m=+880.023372475"
Sep 11 19:41:07.364758 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:07.364672    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"
Sep 11 19:41:07.390752 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:07.390721    2558 generic.go:356] "Generic (PLEG): container finished" podID="c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba" containerID="3431d0af64ec35fffca5bba6aa3ad1ae8832463132192757bfae7a8590f36442" exitCode=0
Sep 11 19:41:07.390898 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:07.390781    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd" event={"ID":"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba","Type":"ContainerDied","Data":"3431d0af64ec35fffca5bba6aa3ad1ae8832463132192757bfae7a8590f36442"}
Sep 11 19:41:07.391149 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:07.391134    2558 scope.go:117] "RemoveContainer" containerID="3431d0af64ec35fffca5bba6aa3ad1ae8832463132192757bfae7a8590f36442"
Sep 11 19:41:07.392161 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:07.392128    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:41:07.392246 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:07.392192    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:41:07.392246 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:07.392227    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:41:08.397377 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:08.397343    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd" event={"ID":"c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba","Type":"ContainerStarted","Data":"822f4de0c89c5fb20d2f042df61446f76f63dd86d4196d01e93cbddc4421730f"}
Sep 11 19:41:08.667162 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:08.667108    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:41:08.759979 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:08.759474    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:41:08.759979 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:08.759614    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-pqmwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 11 19:41:08.759979 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:08.759662    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd" podUID="c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 11 19:41:09.757639 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:09.757588    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-pqmwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 11 19:41:09.758180 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:09.757663    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd" podUID="c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 11 19:41:10.758448 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:10.758404    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-pqmwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 11 19:41:10.759096 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:10.759060    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd" podUID="c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 11 19:41:11.758313 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:11.758262    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-pqmwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 11 19:41:11.758545 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:11.758354    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd" podUID="c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 11 19:41:12.759092 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:12.758504    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-pqmwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Sep 11 19:41:12.759092 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:12.758572    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd" podUID="c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Sep 11 19:41:13.761326 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:13.761293    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:41:13.762212 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:13.762194    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-pqmwd"
Sep 11 19:41:14.377776 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:14.377747    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-xptvp"
Sep 11 19:41:15.382303 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:15.382268    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-wkf6k"
Sep 11 19:41:27.572867 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.572833    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"]
Sep 11 19:41:27.577562 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.577542    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:27.580262 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.580245    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 19:41:27.581176 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.581158    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 11 19:41:27.581248 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.581202    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 19:41:27.581300 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.581210    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cm7wn\""
Sep 11 19:41:27.593427 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.590350    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"]
Sep 11 19:41:27.735088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.735049    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn6km\" (UniqueName: \"kubernetes.io/projected/cbbffc01-0831-4480-834c-8fb2cc60a333-kube-api-access-nn6km\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz\" (UID: \"cbbffc01-0831-4480-834c-8fb2cc60a333\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:27.735088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.735093    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbbffc01-0831-4480-834c-8fb2cc60a333-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz\" (UID: \"cbbffc01-0831-4480-834c-8fb2cc60a333\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:27.735293 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.735129    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbbffc01-0831-4480-834c-8fb2cc60a333-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz\" (UID: \"cbbffc01-0831-4480-834c-8fb2cc60a333\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:27.735293 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.735176    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbbffc01-0831-4480-834c-8fb2cc60a333-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz\" (UID: \"cbbffc01-0831-4480-834c-8fb2cc60a333\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:27.735293 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.735217    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbbffc01-0831-4480-834c-8fb2cc60a333-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz\" (UID: \"cbbffc01-0831-4480-834c-8fb2cc60a333\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:27.735293 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.735252    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbbffc01-0831-4480-834c-8fb2cc60a333-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz\" (UID: \"cbbffc01-0831-4480-834c-8fb2cc60a333\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:27.836374 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.836294    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbbffc01-0831-4480-834c-8fb2cc60a333-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz\" (UID: \"cbbffc01-0831-4480-834c-8fb2cc60a333\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:27.836374 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.836338    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbbffc01-0831-4480-834c-8fb2cc60a333-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz\" (UID: \"cbbffc01-0831-4480-834c-8fb2cc60a333\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:27.836374 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.836362    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbbffc01-0831-4480-834c-8fb2cc60a333-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz\" (UID: \"cbbffc01-0831-4480-834c-8fb2cc60a333\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:27.836593 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.836405    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbbffc01-0831-4480-834c-8fb2cc60a333-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz\" (UID: \"cbbffc01-0831-4480-834c-8fb2cc60a333\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:27.836593 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.836475    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbbffc01-0831-4480-834c-8fb2cc60a333-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz\" (UID: \"cbbffc01-0831-4480-834c-8fb2cc60a333\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:27.836593 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.836522    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn6km\" (UniqueName: \"kubernetes.io/projected/cbbffc01-0831-4480-834c-8fb2cc60a333-kube-api-access-nn6km\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz\" (UID: \"cbbffc01-0831-4480-834c-8fb2cc60a333\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:27.836812 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.836786    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbbffc01-0831-4480-834c-8fb2cc60a333-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz\" (UID: \"cbbffc01-0831-4480-834c-8fb2cc60a333\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:27.836967 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.836934    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbbffc01-0831-4480-834c-8fb2cc60a333-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz\" (UID: \"cbbffc01-0831-4480-834c-8fb2cc60a333\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:27.837080 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.836943    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbbffc01-0831-4480-834c-8fb2cc60a333-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz\" (UID: \"cbbffc01-0831-4480-834c-8fb2cc60a333\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:27.838759 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.838737    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbbffc01-0831-4480-834c-8fb2cc60a333-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz\" (UID: \"cbbffc01-0831-4480-834c-8fb2cc60a333\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:27.839032 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.839012    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbbffc01-0831-4480-834c-8fb2cc60a333-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz\" (UID: \"cbbffc01-0831-4480-834c-8fb2cc60a333\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:27.844604 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.844583    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn6km\" (UniqueName: \"kubernetes.io/projected/cbbffc01-0831-4480-834c-8fb2cc60a333-kube-api-access-nn6km\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz\" (UID: \"cbbffc01-0831-4480-834c-8fb2cc60a333\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:27.888519 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:27.888490    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:28.162755 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:28.162731    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"]
Sep 11 19:41:28.164338 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:41:28.164309    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbbffc01_0831_4480_834c_8fb2cc60a333.slice/crio-93407bb57b99dcd7de67944ce842ad98f45b01d0a15ea496aa2dc0014acfa026 WatchSource:0}: Error finding container 93407bb57b99dcd7de67944ce842ad98f45b01d0a15ea496aa2dc0014acfa026: Status 404 returned error can't find the container with id 93407bb57b99dcd7de67944ce842ad98f45b01d0a15ea496aa2dc0014acfa026
Sep 11 19:41:28.493223 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:28.493145    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz" event={"ID":"cbbffc01-0831-4480-834c-8fb2cc60a333","Type":"ContainerStarted","Data":"93407bb57b99dcd7de67944ce842ad98f45b01d0a15ea496aa2dc0014acfa026"}
Sep 11 19:41:29.499862 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:29.499768    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz" event={"ID":"cbbffc01-0831-4480-834c-8fb2cc60a333","Type":"ContainerStarted","Data":"262b0f828a235cde6f84f53329d027b7fdb4324f452108d57a82743f175b429e"}
Sep 11 19:41:29.518849 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:29.518800    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz" podStartSLOduration=1.452158971 podStartE2EDuration="2.518784717s" totalImagesPullingTime="1.066625746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:41:29.517436374 +0000 UTC m=+906.134605864" watchObservedRunningTime="2026-09-11 19:41:29.518784717 +0000 UTC m=+906.135954207"
Sep 11 19:41:37.888782 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:37.888729    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:37.889325 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:37.888810    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:37.901542 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:37.901511    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:38.550934 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:38.550898    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz"
Sep 11 19:41:43.868309 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:43.867661    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"]
Sep 11 19:41:43.875844 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:43.875817    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:43.879180 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:43.879159    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 11 19:41:43.891719 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:43.891688    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"]
Sep 11 19:41:43.982590 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:43.982551    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7bf84f6-aff6-414e-aedb-123f5e4022dd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z4zxz\" (UID: \"f7bf84f6-aff6-414e-aedb-123f5e4022dd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:43.982590 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:43.982592    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7bf84f6-aff6-414e-aedb-123f5e4022dd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z4zxz\" (UID: \"f7bf84f6-aff6-414e-aedb-123f5e4022dd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:43.982837 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:43.982761    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7bf84f6-aff6-414e-aedb-123f5e4022dd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z4zxz\" (UID: \"f7bf84f6-aff6-414e-aedb-123f5e4022dd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:43.982837 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:43.982804    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7bf84f6-aff6-414e-aedb-123f5e4022dd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z4zxz\" (UID: \"f7bf84f6-aff6-414e-aedb-123f5e4022dd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:43.982837 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:43.982827    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7bf84f6-aff6-414e-aedb-123f5e4022dd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z4zxz\" (UID: \"f7bf84f6-aff6-414e-aedb-123f5e4022dd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:43.982938 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:43.982865    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg2f4\" (UniqueName: \"kubernetes.io/projected/f7bf84f6-aff6-414e-aedb-123f5e4022dd-kube-api-access-cg2f4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z4zxz\" (UID: \"f7bf84f6-aff6-414e-aedb-123f5e4022dd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:44.083653 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:44.083618    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg2f4\" (UniqueName: \"kubernetes.io/projected/f7bf84f6-aff6-414e-aedb-123f5e4022dd-kube-api-access-cg2f4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z4zxz\" (UID: \"f7bf84f6-aff6-414e-aedb-123f5e4022dd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:44.083877 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:44.083661    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7bf84f6-aff6-414e-aedb-123f5e4022dd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z4zxz\" (UID: \"f7bf84f6-aff6-414e-aedb-123f5e4022dd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:44.083877 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:44.083681    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7bf84f6-aff6-414e-aedb-123f5e4022dd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z4zxz\" (UID: \"f7bf84f6-aff6-414e-aedb-123f5e4022dd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:44.083877 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:44.083801    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7bf84f6-aff6-414e-aedb-123f5e4022dd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z4zxz\" (UID: \"f7bf84f6-aff6-414e-aedb-123f5e4022dd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:44.083877 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:44.083839    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7bf84f6-aff6-414e-aedb-123f5e4022dd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z4zxz\" (UID: \"f7bf84f6-aff6-414e-aedb-123f5e4022dd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:44.083877 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:44.083869    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7bf84f6-aff6-414e-aedb-123f5e4022dd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z4zxz\" (UID: \"f7bf84f6-aff6-414e-aedb-123f5e4022dd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:44.084150 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:44.084103    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7bf84f6-aff6-414e-aedb-123f5e4022dd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z4zxz\" (UID: \"f7bf84f6-aff6-414e-aedb-123f5e4022dd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:44.084206 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:44.084164    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7bf84f6-aff6-414e-aedb-123f5e4022dd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z4zxz\" (UID: \"f7bf84f6-aff6-414e-aedb-123f5e4022dd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:44.084323 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:44.084302    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7bf84f6-aff6-414e-aedb-123f5e4022dd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z4zxz\" (UID: \"f7bf84f6-aff6-414e-aedb-123f5e4022dd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:44.086056 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:44.086034    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7bf84f6-aff6-414e-aedb-123f5e4022dd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z4zxz\" (UID: \"f7bf84f6-aff6-414e-aedb-123f5e4022dd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:44.086413 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:44.086394    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7bf84f6-aff6-414e-aedb-123f5e4022dd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z4zxz\" (UID: \"f7bf84f6-aff6-414e-aedb-123f5e4022dd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:44.101922 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:44.101896    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg2f4\" (UniqueName: \"kubernetes.io/projected/f7bf84f6-aff6-414e-aedb-123f5e4022dd-kube-api-access-cg2f4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-z4zxz\" (UID: \"f7bf84f6-aff6-414e-aedb-123f5e4022dd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:44.185730 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:44.185666    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:44.461760 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:44.461726    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"]
Sep 11 19:41:44.463512 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:41:44.463486    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bf84f6_aff6_414e_aedb_123f5e4022dd.slice/crio-ee87b4672ea40c296c5ac39e950825b7ff08e388e5bf81281353ace23139c7d6 WatchSource:0}: Error finding container ee87b4672ea40c296c5ac39e950825b7ff08e388e5bf81281353ace23139c7d6: Status 404 returned error can't find the container with id ee87b4672ea40c296c5ac39e950825b7ff08e388e5bf81281353ace23139c7d6
Sep 11 19:41:44.567689 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:44.567649    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz" event={"ID":"f7bf84f6-aff6-414e-aedb-123f5e4022dd","Type":"ContainerStarted","Data":"ee87b4672ea40c296c5ac39e950825b7ff08e388e5bf81281353ace23139c7d6"}
Sep 11 19:41:45.574000 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:45.573964    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz" event={"ID":"f7bf84f6-aff6-414e-aedb-123f5e4022dd","Type":"ContainerStarted","Data":"355229762bb5f7abbadd16f97dfa1a641042599aef16f1b799f3c243136ce73f"}
Sep 11 19:41:45.603021 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:45.602964    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz" podStartSLOduration=2.388682713 podStartE2EDuration="2.602947916s" totalImagesPullingTime="214.265203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:41:45.600853698 +0000 UTC m=+922.218023189" watchObservedRunningTime="2026-09-11 19:41:45.602947916 +0000 UTC m=+922.220117407"
Sep 11 19:41:54.186093 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:54.186050    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:54.186556 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:54.186140    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:54.198642 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:54.198609    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:41:54.628571 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:41:54.628478    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-z4zxz"
Sep 11 19:42:11.848031 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:11.847995    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"]
Sep 11 19:42:11.852964 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:11.852943    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:11.855631 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:11.855611    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 11 19:42:11.864173 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:11.864146    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"]
Sep 11 19:42:11.938524 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:11.938487    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eb64051-b17b-41df-a9de-2ae4708c59db-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw\" (UID: \"6eb64051-b17b-41df-a9de-2ae4708c59db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:11.938693 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:11.938535    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eb64051-b17b-41df-a9de-2ae4708c59db-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw\" (UID: \"6eb64051-b17b-41df-a9de-2ae4708c59db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:11.938693 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:11.938658    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eb64051-b17b-41df-a9de-2ae4708c59db-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw\" (UID: \"6eb64051-b17b-41df-a9de-2ae4708c59db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:11.938817 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:11.938692    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xrgd\" (UniqueName: \"kubernetes.io/projected/6eb64051-b17b-41df-a9de-2ae4708c59db-kube-api-access-5xrgd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw\" (UID: \"6eb64051-b17b-41df-a9de-2ae4708c59db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:11.938817 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:11.938802    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eb64051-b17b-41df-a9de-2ae4708c59db-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw\" (UID: \"6eb64051-b17b-41df-a9de-2ae4708c59db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:11.938920 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:11.938902    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eb64051-b17b-41df-a9de-2ae4708c59db-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw\" (UID: \"6eb64051-b17b-41df-a9de-2ae4708c59db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:12.040382 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:12.040346    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eb64051-b17b-41df-a9de-2ae4708c59db-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw\" (UID: \"6eb64051-b17b-41df-a9de-2ae4708c59db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:12.040574 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:12.040411    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eb64051-b17b-41df-a9de-2ae4708c59db-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw\" (UID: \"6eb64051-b17b-41df-a9de-2ae4708c59db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:12.040574 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:12.040430    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xrgd\" (UniqueName: \"kubernetes.io/projected/6eb64051-b17b-41df-a9de-2ae4708c59db-kube-api-access-5xrgd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw\" (UID: \"6eb64051-b17b-41df-a9de-2ae4708c59db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:12.040574 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:12.040471    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eb64051-b17b-41df-a9de-2ae4708c59db-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw\" (UID: \"6eb64051-b17b-41df-a9de-2ae4708c59db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:12.040574 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:12.040488    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eb64051-b17b-41df-a9de-2ae4708c59db-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw\" (UID: \"6eb64051-b17b-41df-a9de-2ae4708c59db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:12.040574 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:12.040514    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eb64051-b17b-41df-a9de-2ae4708c59db-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw\" (UID: \"6eb64051-b17b-41df-a9de-2ae4708c59db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:12.040880 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:12.040811    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6eb64051-b17b-41df-a9de-2ae4708c59db-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw\" (UID: \"6eb64051-b17b-41df-a9de-2ae4708c59db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:12.040880 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:12.040859    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6eb64051-b17b-41df-a9de-2ae4708c59db-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw\" (UID: \"6eb64051-b17b-41df-a9de-2ae4708c59db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:12.040956 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:12.040895    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6eb64051-b17b-41df-a9de-2ae4708c59db-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw\" (UID: \"6eb64051-b17b-41df-a9de-2ae4708c59db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:12.042799 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:12.042776    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6eb64051-b17b-41df-a9de-2ae4708c59db-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw\" (UID: \"6eb64051-b17b-41df-a9de-2ae4708c59db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:12.043072 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:12.043050    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6eb64051-b17b-41df-a9de-2ae4708c59db-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw\" (UID: \"6eb64051-b17b-41df-a9de-2ae4708c59db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:12.049844 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:12.049817    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xrgd\" (UniqueName: \"kubernetes.io/projected/6eb64051-b17b-41df-a9de-2ae4708c59db-kube-api-access-5xrgd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw\" (UID: \"6eb64051-b17b-41df-a9de-2ae4708c59db\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:12.164867 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:12.164830    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:12.440318 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:12.440291    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"]
Sep 11 19:42:12.441777 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:42:12.441745    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb64051_b17b_41df_a9de_2ae4708c59db.slice/crio-879a5509bd85160edd844717201ea0a3c29f63d3bc8459595f4b1dd786153e24 WatchSource:0}: Error finding container 879a5509bd85160edd844717201ea0a3c29f63d3bc8459595f4b1dd786153e24: Status 404 returned error can't find the container with id 879a5509bd85160edd844717201ea0a3c29f63d3bc8459595f4b1dd786153e24
Sep 11 19:42:12.702306 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:12.702279    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw" event={"ID":"6eb64051-b17b-41df-a9de-2ae4708c59db","Type":"ContainerStarted","Data":"377b09c1c6cf6129e6530695d4397de8d64f830eaf992bc0dd39433718d8391c"}
Sep 11 19:42:12.702415 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:12.702338    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw" event={"ID":"6eb64051-b17b-41df-a9de-2ae4708c59db","Type":"ContainerStarted","Data":"879a5509bd85160edd844717201ea0a3c29f63d3bc8459595f4b1dd786153e24"}
Sep 11 19:42:12.723554 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:12.723495    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw" podStartSLOduration=1.5276109450000002 podStartE2EDuration="1.723472496s" totalImagesPullingTime="195.861551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:42:12.721384102 +0000 UTC m=+949.338553594" watchObservedRunningTime="2026-09-11 19:42:12.723472496 +0000 UTC m=+949.340642012"
Sep 11 19:42:22.165950 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:22.165910    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:22.166398 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:22.165976    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:22.178914 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:22.178891    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:22.758231 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:22.758194    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw"
Sep 11 19:42:25.060610 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.060578    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"]
Sep 11 19:42:25.064213 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.064194    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:25.066937 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.066914    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 11 19:42:25.081429 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.081401    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"]
Sep 11 19:42:25.160383 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.160346    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee29cea2-3b01-42f8-b6e1-afa4e9a872c9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lqzgf\" (UID: \"ee29cea2-3b01-42f8-b6e1-afa4e9a872c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:25.160562 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.160407    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee29cea2-3b01-42f8-b6e1-afa4e9a872c9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lqzgf\" (UID: \"ee29cea2-3b01-42f8-b6e1-afa4e9a872c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:25.160562 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.160467    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee29cea2-3b01-42f8-b6e1-afa4e9a872c9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lqzgf\" (UID: \"ee29cea2-3b01-42f8-b6e1-afa4e9a872c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:25.160562 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.160490    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgdx6\" (UniqueName: \"kubernetes.io/projected/ee29cea2-3b01-42f8-b6e1-afa4e9a872c9-kube-api-access-bgdx6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lqzgf\" (UID: \"ee29cea2-3b01-42f8-b6e1-afa4e9a872c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:25.160685 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.160556    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee29cea2-3b01-42f8-b6e1-afa4e9a872c9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lqzgf\" (UID: \"ee29cea2-3b01-42f8-b6e1-afa4e9a872c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:25.160685 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.160589    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee29cea2-3b01-42f8-b6e1-afa4e9a872c9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lqzgf\" (UID: \"ee29cea2-3b01-42f8-b6e1-afa4e9a872c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:25.261376 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.261339    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee29cea2-3b01-42f8-b6e1-afa4e9a872c9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lqzgf\" (UID: \"ee29cea2-3b01-42f8-b6e1-afa4e9a872c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:25.261568 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.261401    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee29cea2-3b01-42f8-b6e1-afa4e9a872c9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lqzgf\" (UID: \"ee29cea2-3b01-42f8-b6e1-afa4e9a872c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:25.261568 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.261429    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgdx6\" (UniqueName: \"kubernetes.io/projected/ee29cea2-3b01-42f8-b6e1-afa4e9a872c9-kube-api-access-bgdx6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lqzgf\" (UID: \"ee29cea2-3b01-42f8-b6e1-afa4e9a872c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:25.261568 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.261460    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee29cea2-3b01-42f8-b6e1-afa4e9a872c9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lqzgf\" (UID: \"ee29cea2-3b01-42f8-b6e1-afa4e9a872c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:25.261568 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.261484    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee29cea2-3b01-42f8-b6e1-afa4e9a872c9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lqzgf\" (UID: \"ee29cea2-3b01-42f8-b6e1-afa4e9a872c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:25.261568 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.261503    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee29cea2-3b01-42f8-b6e1-afa4e9a872c9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lqzgf\" (UID: \"ee29cea2-3b01-42f8-b6e1-afa4e9a872c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:25.261909 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.261883    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee29cea2-3b01-42f8-b6e1-afa4e9a872c9-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lqzgf\" (UID: \"ee29cea2-3b01-42f8-b6e1-afa4e9a872c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:25.261909 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.261896    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee29cea2-3b01-42f8-b6e1-afa4e9a872c9-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lqzgf\" (UID: \"ee29cea2-3b01-42f8-b6e1-afa4e9a872c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:25.262016 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.261954    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee29cea2-3b01-42f8-b6e1-afa4e9a872c9-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lqzgf\" (UID: \"ee29cea2-3b01-42f8-b6e1-afa4e9a872c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:25.263833 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.263810    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee29cea2-3b01-42f8-b6e1-afa4e9a872c9-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lqzgf\" (UID: \"ee29cea2-3b01-42f8-b6e1-afa4e9a872c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:25.264124 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.264105    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee29cea2-3b01-42f8-b6e1-afa4e9a872c9-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lqzgf\" (UID: \"ee29cea2-3b01-42f8-b6e1-afa4e9a872c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:25.269792 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.269767    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgdx6\" (UniqueName: \"kubernetes.io/projected/ee29cea2-3b01-42f8-b6e1-afa4e9a872c9-kube-api-access-bgdx6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-lqzgf\" (UID: \"ee29cea2-3b01-42f8-b6e1-afa4e9a872c9\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:25.375200 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.375104    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:25.677635 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.677610    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"]
Sep 11 19:42:25.679847 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:42:25.679816    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee29cea2_3b01_42f8_b6e1_afa4e9a872c9.slice/crio-ed7d45eef587b2a96a01ac1fcdd12b6abaf9eb75713739cb1ae452e2325c6a67 WatchSource:0}: Error finding container ed7d45eef587b2a96a01ac1fcdd12b6abaf9eb75713739cb1ae452e2325c6a67: Status 404 returned error can't find the container with id ed7d45eef587b2a96a01ac1fcdd12b6abaf9eb75713739cb1ae452e2325c6a67
Sep 11 19:42:25.761395 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:25.761352    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf" event={"ID":"ee29cea2-3b01-42f8-b6e1-afa4e9a872c9","Type":"ContainerStarted","Data":"ed7d45eef587b2a96a01ac1fcdd12b6abaf9eb75713739cb1ae452e2325c6a67"}
Sep 11 19:42:26.767984 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:26.767950    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf" event={"ID":"ee29cea2-3b01-42f8-b6e1-afa4e9a872c9","Type":"ContainerStarted","Data":"31e53bc6fb79ea2ac39c909bcf7bd776045bb16f846764eef2d56bc5707953a2"}
Sep 11 19:42:26.794791 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:26.791268    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf" podStartSLOduration=1.591987675 podStartE2EDuration="1.791248304s" totalImagesPullingTime="199.260629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:42:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:42:26.787219566 +0000 UTC m=+963.404389055" watchObservedRunningTime="2026-09-11 19:42:26.791248304 +0000 UTC m=+963.408417796"
Sep 11 19:42:35.376021 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:35.375976    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:35.376502 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:35.376047    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:35.388954 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:35.388931    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:42:35.827104 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:42:35.827073    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-lqzgf"
Sep 11 19:44:47.534850 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.534815    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"]
Sep 11 19:44:47.538191 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.538173    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.541628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.541592    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ec80ad-openshift-default-dockercfg-55c75\""
Sep 11 19:44:47.557836 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.557807    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"]
Sep 11 19:44:47.585950 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.585907    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-podinfo\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.586130 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.585964    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-workload-socket\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.586130 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.586003    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqc82\" (UniqueName: \"kubernetes.io/projected/e3adf554-075e-45e2-8e5c-0019ce755a1a-kube-api-access-gqc82\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.586130 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.586032    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-credential-socket\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.586130 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.586055    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-envoy\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.586270 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.586156    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-token\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.586270 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.586202    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-data\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.586270 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.586244    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-workload-certs\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.586364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.586303    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3adf554-075e-45e2-8e5c-0019ce755a1a-istiod-ca-cert\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.687397 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.687359    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-envoy\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.687579 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.687431    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-token\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.687579 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.687477    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-data\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.687579 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.687505    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-workload-certs\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.687579 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.687557    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3adf554-075e-45e2-8e5c-0019ce755a1a-istiod-ca-cert\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.687821 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.687631    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-podinfo\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.687821 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.687731    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-workload-socket\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.687821 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.687773    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqc82\" (UniqueName: \"kubernetes.io/projected/e3adf554-075e-45e2-8e5c-0019ce755a1a-kube-api-access-gqc82\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.687821 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.687816    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-credential-socket\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.688015 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.687955    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-workload-certs\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.688585 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.688336    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-workload-socket\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.688585 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.688359    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-data\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.688585 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.688543    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-credential-socket\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.688847 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.688602    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3adf554-075e-45e2-8e5c-0019ce755a1a-istiod-ca-cert\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.690052 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.690024    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-envoy\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.690937 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.690918    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-podinfo\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.701364 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.701271    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"]
Sep 11 19:44:47.704281 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.704225    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-token\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.704953 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.704925    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqc82\" (UniqueName: \"kubernetes.io/projected/e3adf554-075e-45e2-8e5c-0019ce755a1a-kube-api-access-gqc82\") pod \"e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") " pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.706463 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.706442    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.709229 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.709206    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-10cc24-openshift-default-dockercfg-cbvs7\""
Sep 11 19:44:47.718277 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.718246    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"]
Sep 11 19:44:47.789147 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.789068    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-podinfo\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.789147 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.789115    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-data\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.789374 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.789203    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-token\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.789374 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.789347    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-workload-certs\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.789507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.789396    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-credential-socket\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.789507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.789420    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjgkr\" (UniqueName: \"kubernetes.io/projected/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-kube-api-access-qjgkr\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.789507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.789448    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-workload-socket\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.789507 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.789474    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-envoy\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.789723 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.789571    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istiod-ca-cert\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.849135 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.849094    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:47.890155 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.890119    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istiod-ca-cert\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.890344 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.890173    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-podinfo\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.890344 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.890204    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-data\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.890344 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.890243    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-token\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.890344 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.890335    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-workload-certs\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.890563 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.890375    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-credential-socket\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.890563 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.890401    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjgkr\" (UniqueName: \"kubernetes.io/projected/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-kube-api-access-qjgkr\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.890563 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.890430    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-workload-socket\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.890563 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.890457    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-envoy\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.890919 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.890890    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-workload-certs\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.891017 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.890923    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istiod-ca-cert\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.891186 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.891163    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-credential-socket\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.891254 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.891199    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-workload-socket\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.891416 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.891389    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-data\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.893033 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.893012    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-envoy\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.893834 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.893813    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-podinfo\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.912661 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.912625    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-token\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:47.913402 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:47.913373    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjgkr\" (UniqueName: \"kubernetes.io/projected/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-kube-api-access-qjgkr\") pod \"e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") " pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:48.027971 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:48.027931    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:48.131184 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:48.131151    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"]
Sep 11 19:44:48.133004 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:44:48.132974    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3adf554_075e_45e2_8e5c_0019ce755a1a.slice/crio-c809204cfe4cb334dd61f361d8f477e33d8840e12110c675d75781bb75fbfcdf WatchSource:0}: Error finding container c809204cfe4cb334dd61f361d8f477e33d8840e12110c675d75781bb75fbfcdf: Status 404 returned error can't find the container with id c809204cfe4cb334dd61f361d8f477e33d8840e12110c675d75781bb75fbfcdf
Sep 11 19:44:48.135380 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:48.135344    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:44:48.135472 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:48.135410    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:44:48.135472 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:48.135446    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:44:48.307499 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:48.307467    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"]
Sep 11 19:44:48.308998 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:44:48.308968    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod195a2bd1_6a70_4e0a_a728_27fcb3ead58b.slice/crio-cb12df8619c9cde92f31086c9d78b8ae4333363a3e0a16f16ca918258a49af72 WatchSource:0}: Error finding container cb12df8619c9cde92f31086c9d78b8ae4333363a3e0a16f16ca918258a49af72: Status 404 returned error can't find the container with id cb12df8619c9cde92f31086c9d78b8ae4333363a3e0a16f16ca918258a49af72
Sep 11 19:44:48.311415 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:48.311380    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:44:48.311486 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:48.311446    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:44:48.311486 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:48.311481    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:44:48.458649 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:48.458608    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq" event={"ID":"195a2bd1-6a70-4e0a-a728-27fcb3ead58b","Type":"ContainerStarted","Data":"02e8b243217eccca6b797952f9013e5239aa9821207978fe2687e28b22756580"}
Sep 11 19:44:48.458986 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:48.458723    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq" event={"ID":"195a2bd1-6a70-4e0a-a728-27fcb3ead58b","Type":"ContainerStarted","Data":"cb12df8619c9cde92f31086c9d78b8ae4333363a3e0a16f16ca918258a49af72"}
Sep 11 19:44:48.460588 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:48.460555    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8" event={"ID":"e3adf554-075e-45e2-8e5c-0019ce755a1a","Type":"ContainerStarted","Data":"bcff301226a83253444b84d95dd932bddca541d1abd2c8cc3e7f1f65f9bfa193"}
Sep 11 19:44:48.460739 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:48.460620    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8" event={"ID":"e3adf554-075e-45e2-8e5c-0019ce755a1a","Type":"ContainerStarted","Data":"c809204cfe4cb334dd61f361d8f477e33d8840e12110c675d75781bb75fbfcdf"}
Sep 11 19:44:48.480461 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:48.480396    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq" podStartSLOduration=1.48037543 podStartE2EDuration="1.48037543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:44:48.479863756 +0000 UTC m=+1105.097033261" watchObservedRunningTime="2026-09-11 19:44:48.48037543 +0000 UTC m=+1105.097544918"
Sep 11 19:44:48.518797 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:48.518732    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8" podStartSLOduration=1.518698688 podStartE2EDuration="1.518698688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:44:48.518381409 +0000 UTC m=+1105.135550901" watchObservedRunningTime="2026-09-11 19:44:48.518698688 +0000 UTC m=+1105.135868178"
Sep 11 19:44:48.849406 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:48.849362    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:49.028904 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:49.028860    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:49.034141 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:49.034114    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:49.466544 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:49.466504    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:49.467289 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:49.467269    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:44:49.855917 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:49.855842    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:50.471379 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:50.471343    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:50.472330 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:50.472307    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:44:52.154081 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:52.154043    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"]
Sep 11 19:44:52.157566 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:52.157533    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"
Sep 11 19:44:52.162934 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:52.162909    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ec80ad-serving-cert\""
Sep 11 19:44:52.185372 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:52.184883    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"]
Sep 11 19:44:52.235480 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:52.235440    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-metrics-certs\") pod \"maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw\" (UID: \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"
Sep 11 19:44:52.235700 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:52.235587    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-maas-api-tls\") pod \"maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw\" (UID: \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"
Sep 11 19:44:52.235700 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:52.235659    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zrt6\" (UniqueName: \"kubernetes.io/projected/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-kube-api-access-8zrt6\") pod \"maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw\" (UID: \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"
Sep 11 19:44:52.235700 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:52.235692    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw\" (UID: \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"
Sep 11 19:44:52.336587 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:52.336545    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-maas-api-tls\") pod \"maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw\" (UID: \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"
Sep 11 19:44:52.336815 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:52.336610    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zrt6\" (UniqueName: \"kubernetes.io/projected/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-kube-api-access-8zrt6\") pod \"maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw\" (UID: \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"
Sep 11 19:44:52.336815 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:52.336634    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw\" (UID: \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"
Sep 11 19:44:52.336815 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:52.336656    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-metrics-certs\") pod \"maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw\" (UID: \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"
Sep 11 19:44:52.336815 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:44:52.336747    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-serving-cert: secret "maas-api-e2e-inf-a-ec80ad-serving-cert" not found
Sep 11 19:44:52.337034 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:44:52.336828    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-maas-api-tls podName:6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb nodeName:}" failed. No retries permitted until 2026-09-11 19:44:52.836810333 +0000 UTC m=+1109.453979819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-maas-api-tls") pod "maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw" (UID: "6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb") : secret "maas-api-e2e-inf-a-ec80ad-serving-cert" not found
Sep 11 19:44:52.337421 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:52.337399    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw\" (UID: \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"
Sep 11 19:44:52.339349 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:52.339325    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-metrics-certs\") pod \"maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw\" (UID: \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"
Sep 11 19:44:52.346950 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:52.346925    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zrt6\" (UniqueName: \"kubernetes.io/projected/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-kube-api-access-8zrt6\") pod \"maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw\" (UID: \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"
Sep 11 19:44:52.842925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:52.842889    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-maas-api-tls\") pod \"maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw\" (UID: \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"
Sep 11 19:44:52.845375 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:52.845348    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-maas-api-tls\") pod \"maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw\" (UID: \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"
Sep 11 19:44:53.070261 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:53.070226    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"
Sep 11 19:44:53.367933 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:53.367908    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"]
Sep 11 19:44:53.368262 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:44:53.368083    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6368fdd9_5b9c_4d0d_9f42_ddfba3f052bb.slice/crio-3928d43a3fe6e5b93e63278b274c3e53d44b72623b685200c0f77f8da3c8a1e5 WatchSource:0}: Error finding container 3928d43a3fe6e5b93e63278b274c3e53d44b72623b685200c0f77f8da3c8a1e5: Status 404 returned error can't find the container with id 3928d43a3fe6e5b93e63278b274c3e53d44b72623b685200c0f77f8da3c8a1e5
Sep 11 19:44:53.369554 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:53.369533    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 19:44:53.486624 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:53.486591    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw" event={"ID":"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb","Type":"ContainerStarted","Data":"3928d43a3fe6e5b93e63278b274c3e53d44b72623b685200c0f77f8da3c8a1e5"}
Sep 11 19:44:54.309250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.309217    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"]
Sep 11 19:44:54.313092 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.313068    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"
Sep 11 19:44:54.315824 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.315795    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-c7766c-serving-cert\""
Sep 11 19:44:54.330387 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.330358    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"]
Sep 11 19:44:54.359016 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.358985    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f96l8\" (UniqueName: \"kubernetes.io/projected/a1086d50-c963-4705-92da-51a1c144de7b-kube-api-access-f96l8\") pod \"maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp\" (UID: \"a1086d50-c963-4705-92da-51a1c144de7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"
Sep 11 19:44:54.359181 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.359095    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1086d50-c963-4705-92da-51a1c144de7b-metrics-certs\") pod \"maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp\" (UID: \"a1086d50-c963-4705-92da-51a1c144de7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"
Sep 11 19:44:54.359235 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.359209    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1086d50-c963-4705-92da-51a1c144de7b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp\" (UID: \"a1086d50-c963-4705-92da-51a1c144de7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"
Sep 11 19:44:54.359272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.359254    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1086d50-c963-4705-92da-51a1c144de7b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp\" (UID: \"a1086d50-c963-4705-92da-51a1c144de7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"
Sep 11 19:44:54.460221 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.460188    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1086d50-c963-4705-92da-51a1c144de7b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp\" (UID: \"a1086d50-c963-4705-92da-51a1c144de7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"
Sep 11 19:44:54.460732 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.460237    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1086d50-c963-4705-92da-51a1c144de7b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp\" (UID: \"a1086d50-c963-4705-92da-51a1c144de7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"
Sep 11 19:44:54.460732 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.460278    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f96l8\" (UniqueName: \"kubernetes.io/projected/a1086d50-c963-4705-92da-51a1c144de7b-kube-api-access-f96l8\") pod \"maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp\" (UID: \"a1086d50-c963-4705-92da-51a1c144de7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"
Sep 11 19:44:54.460732 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.460367    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1086d50-c963-4705-92da-51a1c144de7b-metrics-certs\") pod \"maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp\" (UID: \"a1086d50-c963-4705-92da-51a1c144de7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"
Sep 11 19:44:54.460974 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.460941    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1086d50-c963-4705-92da-51a1c144de7b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp\" (UID: \"a1086d50-c963-4705-92da-51a1c144de7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"
Sep 11 19:44:54.462722 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.462691    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1086d50-c963-4705-92da-51a1c144de7b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp\" (UID: \"a1086d50-c963-4705-92da-51a1c144de7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"
Sep 11 19:44:54.462895 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.462877    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1086d50-c963-4705-92da-51a1c144de7b-metrics-certs\") pod \"maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp\" (UID: \"a1086d50-c963-4705-92da-51a1c144de7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"
Sep 11 19:44:54.471085 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.471057    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f96l8\" (UniqueName: \"kubernetes.io/projected/a1086d50-c963-4705-92da-51a1c144de7b-kube-api-access-f96l8\") pod \"maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp\" (UID: \"a1086d50-c963-4705-92da-51a1c144de7b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"
Sep 11 19:44:54.493087 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.493049    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw" event={"ID":"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb","Type":"ContainerStarted","Data":"3cf070cd00bb0d2c9c84d85c8d49e774ab60d3997871e146e0c4bab0cf455d41"}
Sep 11 19:44:54.493235 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.493161    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"
Sep 11 19:44:54.518904 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.518848    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw" podStartSLOduration=2.096113713 podStartE2EDuration="2.518830632s" totalImagesPullingTime="422.716919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:44:54.515590138 +0000 UTC m=+1111.132759625" watchObservedRunningTime="2026-09-11 19:44:54.518830632 +0000 UTC m=+1111.136000122"
Sep 11 19:44:54.625480 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.625391    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"
Sep 11 19:44:54.907846 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:54.907819    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"]
Sep 11 19:44:54.909895 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:44:54.909866    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1086d50_c963_4705_92da_51a1c144de7b.slice/crio-dcc0cc16a403f99cfc715c1f1bf493a746853c976120eb3e252e8642feedd81e WatchSource:0}: Error finding container dcc0cc16a403f99cfc715c1f1bf493a746853c976120eb3e252e8642feedd81e: Status 404 returned error can't find the container with id dcc0cc16a403f99cfc715c1f1bf493a746853c976120eb3e252e8642feedd81e
Sep 11 19:44:55.499636 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:55.499602    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp" event={"ID":"a1086d50-c963-4705-92da-51a1c144de7b","Type":"ContainerStarted","Data":"470d86a243a924640afdf4686d06a4c78d049333618f4cc87a4d27aee906648e"}
Sep 11 19:44:55.500077 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:55.499659    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp" event={"ID":"a1086d50-c963-4705-92da-51a1c144de7b","Type":"ContainerStarted","Data":"dcc0cc16a403f99cfc715c1f1bf493a746853c976120eb3e252e8642feedd81e"}
Sep 11 19:44:55.500077 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:55.499769    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"
Sep 11 19:44:55.530919 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:55.530860    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp" podStartSLOduration=1.15136225 podStartE2EDuration="1.530836558s" totalImagesPullingTime="379.474308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:44:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:44:55.530458115 +0000 UTC m=+1112.147627608" watchObservedRunningTime="2026-09-11 19:44:55.530836558 +0000 UTC m=+1112.148006051"
Sep 11 19:44:56.403683 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.403649    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj"]
Sep 11 19:44:56.407263 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.407241    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj"
Sep 11 19:44:56.409633 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.409607    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-10cc24\""
Sep 11 19:44:56.421326 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.421294    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj"]
Sep 11 19:44:56.463128 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.463092    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn"]
Sep 11 19:44:56.466673 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.466655    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn"
Sep 11 19:44:56.474721 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.474686    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-10cc24-dockercfg-l2hnp\""
Sep 11 19:44:56.481106 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.481051    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tdkx\" (UniqueName: \"kubernetes.io/projected/8e865ae0-9dab-49b1-ad48-732afc99bfff-kube-api-access-8tdkx\") pod \"payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj\" (UID: \"8e865ae0-9dab-49b1-ad48-732afc99bfff\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj"
Sep 11 19:44:56.481237 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.481155    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e865ae0-9dab-49b1-ad48-732afc99bfff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj\" (UID: \"8e865ae0-9dab-49b1-ad48-732afc99bfff\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj"
Sep 11 19:44:56.483500 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.483475    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn"]
Sep 11 19:44:56.582202 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.582162    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tdkx\" (UniqueName: \"kubernetes.io/projected/8e865ae0-9dab-49b1-ad48-732afc99bfff-kube-api-access-8tdkx\") pod \"payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj\" (UID: \"8e865ae0-9dab-49b1-ad48-732afc99bfff\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj"
Sep 11 19:44:56.582682 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.582240    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b95b4ef-0d68-4e4a-a326-fd35bf643b09-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn\" (UID: \"6b95b4ef-0d68-4e4a-a326-fd35bf643b09\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn"
Sep 11 19:44:56.582682 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.582272    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f59v4\" (UniqueName: \"kubernetes.io/projected/6b95b4ef-0d68-4e4a-a326-fd35bf643b09-kube-api-access-f59v4\") pod \"payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn\" (UID: \"6b95b4ef-0d68-4e4a-a326-fd35bf643b09\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn"
Sep 11 19:44:56.582682 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.582334    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e865ae0-9dab-49b1-ad48-732afc99bfff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj\" (UID: \"8e865ae0-9dab-49b1-ad48-732afc99bfff\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj"
Sep 11 19:44:56.583175 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.583156    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e865ae0-9dab-49b1-ad48-732afc99bfff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj\" (UID: \"8e865ae0-9dab-49b1-ad48-732afc99bfff\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj"
Sep 11 19:44:56.591367 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.591341    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tdkx\" (UniqueName: \"kubernetes.io/projected/8e865ae0-9dab-49b1-ad48-732afc99bfff-kube-api-access-8tdkx\") pod \"payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj\" (UID: \"8e865ae0-9dab-49b1-ad48-732afc99bfff\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj"
Sep 11 19:44:56.683874 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.683836    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b95b4ef-0d68-4e4a-a326-fd35bf643b09-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn\" (UID: \"6b95b4ef-0d68-4e4a-a326-fd35bf643b09\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn"
Sep 11 19:44:56.684067 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.683878    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f59v4\" (UniqueName: \"kubernetes.io/projected/6b95b4ef-0d68-4e4a-a326-fd35bf643b09-kube-api-access-f59v4\") pod \"payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn\" (UID: \"6b95b4ef-0d68-4e4a-a326-fd35bf643b09\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn"
Sep 11 19:44:56.684613 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.684584    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b95b4ef-0d68-4e4a-a326-fd35bf643b09-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn\" (UID: \"6b95b4ef-0d68-4e4a-a326-fd35bf643b09\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn"
Sep 11 19:44:56.693346 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.693319    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f59v4\" (UniqueName: \"kubernetes.io/projected/6b95b4ef-0d68-4e4a-a326-fd35bf643b09-kube-api-access-f59v4\") pod \"payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn\" (UID: \"6b95b4ef-0d68-4e4a-a326-fd35bf643b09\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn"
Sep 11 19:44:56.718223 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.718189    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj"
Sep 11 19:44:56.777073 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.777038    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn"
Sep 11 19:44:56.996558 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:56.996532    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj"]
Sep 11 19:44:56.998413 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:44:56.998386    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e865ae0_9dab_49b1_ad48_732afc99bfff.slice/crio-da37fa707c5dc42ceb355159b74be7674b0dab4a53d1ad3e7b38ff7092aee0ca WatchSource:0}: Error finding container da37fa707c5dc42ceb355159b74be7674b0dab4a53d1ad3e7b38ff7092aee0ca: Status 404 returned error can't find the container with id da37fa707c5dc42ceb355159b74be7674b0dab4a53d1ad3e7b38ff7092aee0ca
Sep 11 19:44:57.060887 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:57.060860    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn"]
Sep 11 19:44:57.062521 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:44:57.062495    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b95b4ef_0d68_4e4a_a326_fd35bf643b09.slice/crio-4e9d17051e16cf4f9fc7944d27be41cc8148a499d361145d2a4d177f45e79956 WatchSource:0}: Error finding container 4e9d17051e16cf4f9fc7944d27be41cc8148a499d361145d2a4d177f45e79956: Status 404 returned error can't find the container with id 4e9d17051e16cf4f9fc7944d27be41cc8148a499d361145d2a4d177f45e79956
Sep 11 19:44:57.512146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:57.512108    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj" event={"ID":"8e865ae0-9dab-49b1-ad48-732afc99bfff","Type":"ContainerStarted","Data":"5ab2dd185aae09ca63d4b25e87896564f6d1abf44e6a6c70ba2cd3461e84d0ef"}
Sep 11 19:44:57.512338 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:57.512164    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj" event={"ID":"8e865ae0-9dab-49b1-ad48-732afc99bfff","Type":"ContainerStarted","Data":"da37fa707c5dc42ceb355159b74be7674b0dab4a53d1ad3e7b38ff7092aee0ca"}
Sep 11 19:44:57.512338 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:57.512191    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj"
Sep 11 19:44:57.514145 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:57.514109    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn" event={"ID":"6b95b4ef-0d68-4e4a-a326-fd35bf643b09","Type":"ContainerStarted","Data":"669b867329fc39388c96e7586727651681cb4d45bf1e65ed7667b14dd61fd364"}
Sep 11 19:44:57.514283 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:57.514171    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn"
Sep 11 19:44:57.514283 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:57.514193    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn" event={"ID":"6b95b4ef-0d68-4e4a-a326-fd35bf643b09","Type":"ContainerStarted","Data":"4e9d17051e16cf4f9fc7944d27be41cc8148a499d361145d2a4d177f45e79956"}
Sep 11 19:44:57.539100 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:57.539042    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj" podStartSLOduration=1.539019739 podStartE2EDuration="1.539019739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:44:57.535253033 +0000 UTC m=+1114.152422525" watchObservedRunningTime="2026-09-11 19:44:57.539019739 +0000 UTC m=+1114.156189230"
Sep 11 19:44:57.559066 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:44:57.558995    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn" podStartSLOduration=1.558972155 podStartE2EDuration="1.558972155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:44:57.557028552 +0000 UTC m=+1114.174198039" watchObservedRunningTime="2026-09-11 19:44:57.558972155 +0000 UTC m=+1114.176141647"
Sep 11 19:45:00.506469 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:00.506423    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"
Sep 11 19:45:01.511865 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:01.511836    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"
Sep 11 19:45:06.772732 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:06.772210    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"]
Sep 11 19:45:06.797287 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:06.797253    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"]
Sep 11 19:45:06.797856 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:06.797836    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"
Sep 11 19:45:06.803089 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:06.803066    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4c47b5e2-serving-cert\""
Sep 11 19:45:06.911545 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:06.911499    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32810fb6-b1a4-433f-b633-4f4c17f085ea-metrics-certs\") pod \"maas-api-e2e-ait-4c47b5e2-776f766d67-xq255\" (UID: \"32810fb6-b1a4-433f-b633-4f4c17f085ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"
Sep 11 19:45:06.911792 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:06.911573    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32810fb6-b1a4-433f-b633-4f4c17f085ea-maas-api-tls\") pod \"maas-api-e2e-ait-4c47b5e2-776f766d67-xq255\" (UID: \"32810fb6-b1a4-433f-b633-4f4c17f085ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"
Sep 11 19:45:06.911792 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:06.911635    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32810fb6-b1a4-433f-b633-4f4c17f085ea-openshift-service-ca\") pod \"maas-api-e2e-ait-4c47b5e2-776f766d67-xq255\" (UID: \"32810fb6-b1a4-433f-b633-4f4c17f085ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"
Sep 11 19:45:06.911936 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:06.911791    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vqx8\" (UniqueName: \"kubernetes.io/projected/32810fb6-b1a4-433f-b633-4f4c17f085ea-kube-api-access-2vqx8\") pod \"maas-api-e2e-ait-4c47b5e2-776f766d67-xq255\" (UID: \"32810fb6-b1a4-433f-b633-4f4c17f085ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"
Sep 11 19:45:07.012562 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:07.012519    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vqx8\" (UniqueName: \"kubernetes.io/projected/32810fb6-b1a4-433f-b633-4f4c17f085ea-kube-api-access-2vqx8\") pod \"maas-api-e2e-ait-4c47b5e2-776f766d67-xq255\" (UID: \"32810fb6-b1a4-433f-b633-4f4c17f085ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"
Sep 11 19:45:07.012776 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:07.012611    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32810fb6-b1a4-433f-b633-4f4c17f085ea-metrics-certs\") pod \"maas-api-e2e-ait-4c47b5e2-776f766d67-xq255\" (UID: \"32810fb6-b1a4-433f-b633-4f4c17f085ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"
Sep 11 19:45:07.012776 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:07.012701    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32810fb6-b1a4-433f-b633-4f4c17f085ea-maas-api-tls\") pod \"maas-api-e2e-ait-4c47b5e2-776f766d67-xq255\" (UID: \"32810fb6-b1a4-433f-b633-4f4c17f085ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"
Sep 11 19:45:07.012776 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:07.012771    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32810fb6-b1a4-433f-b633-4f4c17f085ea-openshift-service-ca\") pod \"maas-api-e2e-ait-4c47b5e2-776f766d67-xq255\" (UID: \"32810fb6-b1a4-433f-b633-4f4c17f085ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"
Sep 11 19:45:07.012970 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:45:07.012946    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-serving-cert: secret "maas-api-e2e-ait-4c47b5e2-serving-cert" not found
Sep 11 19:45:07.013044 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:45:07.013028    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32810fb6-b1a4-433f-b633-4f4c17f085ea-maas-api-tls podName:32810fb6-b1a4-433f-b633-4f4c17f085ea nodeName:}" failed. No retries permitted until 2026-09-11 19:45:07.513005622 +0000 UTC m=+1124.130175111 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32810fb6-b1a4-433f-b633-4f4c17f085ea-maas-api-tls") pod "maas-api-e2e-ait-4c47b5e2-776f766d67-xq255" (UID: "32810fb6-b1a4-433f-b633-4f4c17f085ea") : secret "maas-api-e2e-ait-4c47b5e2-serving-cert" not found
Sep 11 19:45:07.013632 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:07.013607    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32810fb6-b1a4-433f-b633-4f4c17f085ea-openshift-service-ca\") pod \"maas-api-e2e-ait-4c47b5e2-776f766d67-xq255\" (UID: \"32810fb6-b1a4-433f-b633-4f4c17f085ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"
Sep 11 19:45:07.015217 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:07.015145    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32810fb6-b1a4-433f-b633-4f4c17f085ea-metrics-certs\") pod \"maas-api-e2e-ait-4c47b5e2-776f766d67-xq255\" (UID: \"32810fb6-b1a4-433f-b633-4f4c17f085ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"
Sep 11 19:45:07.021929 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:07.021896    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vqx8\" (UniqueName: \"kubernetes.io/projected/32810fb6-b1a4-433f-b633-4f4c17f085ea-kube-api-access-2vqx8\") pod \"maas-api-e2e-ait-4c47b5e2-776f766d67-xq255\" (UID: \"32810fb6-b1a4-433f-b633-4f4c17f085ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"
Sep 11 19:45:07.519253 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:07.519154    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32810fb6-b1a4-433f-b633-4f4c17f085ea-maas-api-tls\") pod \"maas-api-e2e-ait-4c47b5e2-776f766d67-xq255\" (UID: \"32810fb6-b1a4-433f-b633-4f4c17f085ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"
Sep 11 19:45:07.521750 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:07.521697    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32810fb6-b1a4-433f-b633-4f4c17f085ea-maas-api-tls\") pod \"maas-api-e2e-ait-4c47b5e2-776f766d67-xq255\" (UID: \"32810fb6-b1a4-433f-b633-4f4c17f085ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"
Sep 11 19:45:07.714956 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:07.714909    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"
Sep 11 19:45:07.844284 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:07.844225    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn"]
Sep 11 19:45:07.851228 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:07.851195    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn"
Sep 11 19:45:07.858282 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:07.858096    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4c47b5e2\""
Sep 11 19:45:07.861302 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:07.860068    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4c47b5e2-dockercfg-lj2ws\""
Sep 11 19:45:07.868973 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:07.868926    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn"]
Sep 11 19:45:08.018122 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:08.018086    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"]
Sep 11 19:45:08.019683 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:45:08.019651    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32810fb6_b1a4_433f_b633_4f4c17f085ea.slice/crio-555e10232b95c4195cae6704fefcace6c1b77eb04571fb41893f400894f718b6 WatchSource:0}: Error finding container 555e10232b95c4195cae6704fefcace6c1b77eb04571fb41893f400894f718b6: Status 404 returned error can't find the container with id 555e10232b95c4195cae6704fefcace6c1b77eb04571fb41893f400894f718b6
Sep 11 19:45:08.025035 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:08.025010    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vh6m\" (UniqueName: \"kubernetes.io/projected/be735088-af15-41c7-b352-1630b1f37c82-kube-api-access-9vh6m\") pod \"payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn\" (UID: \"be735088-af15-41c7-b352-1630b1f37c82\") " pod="openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn"
Sep 11 19:45:08.025245 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:08.025217    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be735088-af15-41c7-b352-1630b1f37c82-plugins-config-volume\") pod \"payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn\" (UID: \"be735088-af15-41c7-b352-1630b1f37c82\") " pod="openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn"
Sep 11 19:45:08.125991 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:08.125886    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be735088-af15-41c7-b352-1630b1f37c82-plugins-config-volume\") pod \"payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn\" (UID: \"be735088-af15-41c7-b352-1630b1f37c82\") " pod="openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn"
Sep 11 19:45:08.126167 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:08.125989    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vh6m\" (UniqueName: \"kubernetes.io/projected/be735088-af15-41c7-b352-1630b1f37c82-kube-api-access-9vh6m\") pod \"payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn\" (UID: \"be735088-af15-41c7-b352-1630b1f37c82\") " pod="openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn"
Sep 11 19:45:08.126929 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:08.126904    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be735088-af15-41c7-b352-1630b1f37c82-plugins-config-volume\") pod \"payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn\" (UID: \"be735088-af15-41c7-b352-1630b1f37c82\") " pod="openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn"
Sep 11 19:45:08.145657 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:08.145618    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vh6m\" (UniqueName: \"kubernetes.io/projected/be735088-af15-41c7-b352-1630b1f37c82-kube-api-access-9vh6m\") pod \"payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn\" (UID: \"be735088-af15-41c7-b352-1630b1f37c82\") " pod="openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn"
Sep 11 19:45:08.173335 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:08.173295    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn"
Sep 11 19:45:08.472795 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:45:08.472757    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe735088_af15_41c7_b352_1630b1f37c82.slice/crio-a9473c8f8986e4c49d3a52af5e39b297010336e5a319bb8496612367cd9732b1 WatchSource:0}: Error finding container a9473c8f8986e4c49d3a52af5e39b297010336e5a319bb8496612367cd9732b1: Status 404 returned error can't find the container with id a9473c8f8986e4c49d3a52af5e39b297010336e5a319bb8496612367cd9732b1
Sep 11 19:45:08.473301 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:08.473275    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn"]
Sep 11 19:45:08.523192 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:08.523146    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn"
Sep 11 19:45:08.524283 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:08.524256    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj"
Sep 11 19:45:08.606689 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:08.606513    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn" event={"ID":"be735088-af15-41c7-b352-1630b1f37c82","Type":"ContainerStarted","Data":"35969ae95bc66602f117b951db32f66a3fc92d58338b4fba8da799f62d1b713a"}
Sep 11 19:45:08.606689 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:08.606596    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn" event={"ID":"be735088-af15-41c7-b352-1630b1f37c82","Type":"ContainerStarted","Data":"a9473c8f8986e4c49d3a52af5e39b297010336e5a319bb8496612367cd9732b1"}
Sep 11 19:45:08.606689 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:08.606653    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn"
Sep 11 19:45:08.609350 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:08.609244    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255" event={"ID":"32810fb6-b1a4-433f-b633-4f4c17f085ea","Type":"ContainerStarted","Data":"958700900d3f891cba8b8efe007df1e59408479ffae20fb87be0bb1d6c83d3cf"}
Sep 11 19:45:08.609350 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:08.609306    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255" event={"ID":"32810fb6-b1a4-433f-b633-4f4c17f085ea","Type":"ContainerStarted","Data":"555e10232b95c4195cae6704fefcace6c1b77eb04571fb41893f400894f718b6"}
Sep 11 19:45:08.609560 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:08.609395    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"
Sep 11 19:45:08.630491 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:08.630154    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn" podStartSLOduration=1.630131577 podStartE2EDuration="1.630131577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:45:08.626619235 +0000 UTC m=+1125.243788726" watchObservedRunningTime="2026-09-11 19:45:08.630131577 +0000 UTC m=+1125.247301097"
Sep 11 19:45:08.649764 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:08.649679    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255" podStartSLOduration=2.254131973 podStartE2EDuration="2.649653028s" totalImagesPullingTime="395.521055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:45:08.645521591 +0000 UTC m=+1125.262691082" watchObservedRunningTime="2026-09-11 19:45:08.649653028 +0000 UTC m=+1125.266822520"
Sep 11 19:45:10.622732 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:10.622684    2558 generic.go:356] "Generic (PLEG): container finished" podID="195a2bd1-6a70-4e0a-a728-27fcb3ead58b" containerID="02e8b243217eccca6b797952f9013e5239aa9821207978fe2687e28b22756580" exitCode=0
Sep 11 19:45:10.623310 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:10.622761    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq" event={"ID":"195a2bd1-6a70-4e0a-a728-27fcb3ead58b","Type":"ContainerDied","Data":"02e8b243217eccca6b797952f9013e5239aa9821207978fe2687e28b22756580"}
Sep 11 19:45:10.623310 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:10.623302    2558 scope.go:117] "RemoveContainer" containerID="02e8b243217eccca6b797952f9013e5239aa9821207978fe2687e28b22756580"
Sep 11 19:45:10.624367 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:10.624345    2558 generic.go:356] "Generic (PLEG): container finished" podID="e3adf554-075e-45e2-8e5c-0019ce755a1a" containerID="bcff301226a83253444b84d95dd932bddca541d1abd2c8cc3e7f1f65f9bfa193" exitCode=0
Sep 11 19:45:10.624514 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:10.624378    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8" event={"ID":"e3adf554-075e-45e2-8e5c-0019ce755a1a","Type":"ContainerDied","Data":"bcff301226a83253444b84d95dd932bddca541d1abd2c8cc3e7f1f65f9bfa193"}
Sep 11 19:45:10.624514 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:10.624476    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:45:10.624644 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:10.624545    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:45:10.624644 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:10.624576    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:45:10.624842 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:10.624824    2558 scope.go:117] "RemoveContainer" containerID="bcff301226a83253444b84d95dd932bddca541d1abd2c8cc3e7f1f65f9bfa193"
Sep 11 19:45:10.625994 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:10.625959    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:45:10.626053 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:10.626029    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:45:10.626095 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:10.626062    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:45:11.633288 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:11.633247    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq" event={"ID":"195a2bd1-6a70-4e0a-a728-27fcb3ead58b","Type":"ContainerStarted","Data":"b8a66eaa68e39e048d81941ef66dcafcdf4351ecbd20835f7c2df706959036b5"}
Sep 11 19:45:11.636423 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:11.635766    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8" event={"ID":"e3adf554-075e-45e2-8e5c-0019ce755a1a","Type":"ContainerStarted","Data":"8323aa4f9f3fa53993063e6d78a0ece1091d91d815f36572e1fab07a34555b4d"}
Sep 11 19:45:11.850379 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:11.850343    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:45:11.850669 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:11.850388    2558 patch_prober.go:28] interesting pod/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 11 19:45:11.850863 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:11.850835    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8" podUID="e3adf554-075e-45e2-8e5c-0019ce755a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 11 19:45:12.028634 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:12.028588    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:45:12.028837 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:12.028776    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 11 19:45:12.028837 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:12.028825    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq" podUID="195a2bd1-6a70-4e0a-a728-27fcb3ead58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 11 19:45:12.850439 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:12.850399    2558 patch_prober.go:28] interesting pod/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 11 19:45:12.851075 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:12.851034    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8" podUID="e3adf554-075e-45e2-8e5c-0019ce755a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 11 19:45:13.028429 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:13.028395    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 11 19:45:13.028677 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:13.028651    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq" podUID="195a2bd1-6a70-4e0a-a728-27fcb3ead58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 11 19:45:13.850055 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:13.850001    2558 patch_prober.go:28] interesting pod/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 11 19:45:13.850396 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:13.850341    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8" podUID="e3adf554-075e-45e2-8e5c-0019ce755a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 11 19:45:14.028452 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:14.028420    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 11 19:45:14.028917 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:14.028473    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq" podUID="195a2bd1-6a70-4e0a-a728-27fcb3ead58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 11 19:45:14.621728 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:14.621681    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"
Sep 11 19:45:14.850206 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:14.850165    2558 patch_prober.go:28] interesting pod/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 11 19:45:14.850533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:14.850499    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8" podUID="e3adf554-075e-45e2-8e5c-0019ce755a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 11 19:45:15.028600 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:15.028555    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 11 19:45:15.028991 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:15.028634    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq" podUID="195a2bd1-6a70-4e0a-a728-27fcb3ead58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 11 19:45:15.850194 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:15.850145    2558 patch_prober.go:28] interesting pod/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 11 19:45:15.850516 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:15.850468    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8" podUID="e3adf554-075e-45e2-8e5c-0019ce755a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 11 19:45:16.029113 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:16.029073    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 11 19:45:16.029560 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:16.029139    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq" podUID="195a2bd1-6a70-4e0a-a728-27fcb3ead58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 11 19:45:16.852203 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:16.852158    2558 patch_prober.go:28] interesting pod/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Sep 11 19:45:16.852497 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:16.852464    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8" podUID="e3adf554-075e-45e2-8e5c-0019ce755a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Sep 11 19:45:17.028439 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:17.028401    2558 patch_prober.go:28] interesting pod/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 11 19:45:17.028697 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:17.028670    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq" podUID="195a2bd1-6a70-4e0a-a728-27fcb3ead58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 11 19:45:17.854356 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:17.854316    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:45:17.854863 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:17.854844    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:45:17.855516 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:17.855497    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:45:18.033558 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:18.033524    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:45:18.033958 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:18.033936    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:45:18.034786 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:18.034763    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:45:19.617874 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:19.617837    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn"
Sep 11 19:45:28.019437 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.019396    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"]
Sep 11 19:45:28.019915 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.019614    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255" podUID="32810fb6-b1a4-433f-b633-4f4c17f085ea" containerName="maas-api" containerID="cri-o://958700900d3f891cba8b8efe007df1e59408479ffae20fb87be0bb1d6c83d3cf" gracePeriod=30
Sep 11 19:45:28.080810 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.080770    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn"]
Sep 11 19:45:28.081089 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.081035    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn" podUID="be735088-af15-41c7-b352-1630b1f37c82" containerName="payload-processing" containerID="cri-o://35969ae95bc66602f117b951db32f66a3fc92d58338b4fba8da799f62d1b713a" gracePeriod=30
Sep 11 19:45:28.344109 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.344079    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"
Sep 11 19:45:28.379244 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.379217    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn"
Sep 11 19:45:28.456672 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.456636    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32810fb6-b1a4-433f-b633-4f4c17f085ea-openshift-service-ca\") pod \"32810fb6-b1a4-433f-b633-4f4c17f085ea\" (UID: \"32810fb6-b1a4-433f-b633-4f4c17f085ea\") "
Sep 11 19:45:28.456854 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.456775    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vqx8\" (UniqueName: \"kubernetes.io/projected/32810fb6-b1a4-433f-b633-4f4c17f085ea-kube-api-access-2vqx8\") pod \"32810fb6-b1a4-433f-b633-4f4c17f085ea\" (UID: \"32810fb6-b1a4-433f-b633-4f4c17f085ea\") "
Sep 11 19:45:28.456913 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.456875    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32810fb6-b1a4-433f-b633-4f4c17f085ea-maas-api-tls\") pod \"32810fb6-b1a4-433f-b633-4f4c17f085ea\" (UID: \"32810fb6-b1a4-433f-b633-4f4c17f085ea\") "
Sep 11 19:45:28.456961 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.456935    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vh6m\" (UniqueName: \"kubernetes.io/projected/be735088-af15-41c7-b352-1630b1f37c82-kube-api-access-9vh6m\") pod \"be735088-af15-41c7-b352-1630b1f37c82\" (UID: \"be735088-af15-41c7-b352-1630b1f37c82\") "
Sep 11 19:45:28.457014 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.456976    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be735088-af15-41c7-b352-1630b1f37c82-plugins-config-volume\") pod \"be735088-af15-41c7-b352-1630b1f37c82\" (UID: \"be735088-af15-41c7-b352-1630b1f37c82\") "
Sep 11 19:45:28.457074 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.457032    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32810fb6-b1a4-433f-b633-4f4c17f085ea-metrics-certs\") pod \"32810fb6-b1a4-433f-b633-4f4c17f085ea\" (UID: \"32810fb6-b1a4-433f-b633-4f4c17f085ea\") "
Sep 11 19:45:28.457074 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.457044    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32810fb6-b1a4-433f-b633-4f4c17f085ea-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "32810fb6-b1a4-433f-b633-4f4c17f085ea" (UID: "32810fb6-b1a4-433f-b633-4f4c17f085ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:45:28.457407 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.457382    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32810fb6-b1a4-433f-b633-4f4c17f085ea-openshift-service-ca\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:45:28.457514 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.457441    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be735088-af15-41c7-b352-1630b1f37c82-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "be735088-af15-41c7-b352-1630b1f37c82" (UID: "be735088-af15-41c7-b352-1630b1f37c82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:45:28.459196 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.459159    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32810fb6-b1a4-433f-b633-4f4c17f085ea-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "32810fb6-b1a4-433f-b633-4f4c17f085ea" (UID: "32810fb6-b1a4-433f-b633-4f4c17f085ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:45:28.459336 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.459251    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be735088-af15-41c7-b352-1630b1f37c82-kube-api-access-9vh6m" (OuterVolumeSpecName: "kube-api-access-9vh6m") pod "be735088-af15-41c7-b352-1630b1f37c82" (UID: "be735088-af15-41c7-b352-1630b1f37c82"). InnerVolumeSpecName "kube-api-access-9vh6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:45:28.459558 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.459531    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32810fb6-b1a4-433f-b633-4f4c17f085ea-kube-api-access-2vqx8" (OuterVolumeSpecName: "kube-api-access-2vqx8") pod "32810fb6-b1a4-433f-b633-4f4c17f085ea" (UID: "32810fb6-b1a4-433f-b633-4f4c17f085ea"). InnerVolumeSpecName "kube-api-access-2vqx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:45:28.459697 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.459677    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32810fb6-b1a4-433f-b633-4f4c17f085ea-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "32810fb6-b1a4-433f-b633-4f4c17f085ea" (UID: "32810fb6-b1a4-433f-b633-4f4c17f085ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:45:28.558122 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.558027    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vqx8\" (UniqueName: \"kubernetes.io/projected/32810fb6-b1a4-433f-b633-4f4c17f085ea-kube-api-access-2vqx8\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:45:28.558122 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.558060    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32810fb6-b1a4-433f-b633-4f4c17f085ea-maas-api-tls\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:45:28.558122 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.558073    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vh6m\" (UniqueName: \"kubernetes.io/projected/be735088-af15-41c7-b352-1630b1f37c82-kube-api-access-9vh6m\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:45:28.558122 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.558086    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be735088-af15-41c7-b352-1630b1f37c82-plugins-config-volume\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:45:28.558122 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.558098    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32810fb6-b1a4-433f-b633-4f4c17f085ea-metrics-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:45:28.748644 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.748610    2558 generic.go:356] "Generic (PLEG): container finished" podID="32810fb6-b1a4-433f-b633-4f4c17f085ea" containerID="958700900d3f891cba8b8efe007df1e59408479ffae20fb87be0bb1d6c83d3cf" exitCode=0
Sep 11 19:45:28.748846 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.748654    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"
Sep 11 19:45:28.748846 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.748699    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255" event={"ID":"32810fb6-b1a4-433f-b633-4f4c17f085ea","Type":"ContainerDied","Data":"958700900d3f891cba8b8efe007df1e59408479ffae20fb87be0bb1d6c83d3cf"}
Sep 11 19:45:28.748846 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.748789    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255" event={"ID":"32810fb6-b1a4-433f-b633-4f4c17f085ea","Type":"ContainerDied","Data":"555e10232b95c4195cae6704fefcace6c1b77eb04571fb41893f400894f718b6"}
Sep 11 19:45:28.748846 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.748812    2558 scope.go:117] "RemoveContainer" containerID="958700900d3f891cba8b8efe007df1e59408479ffae20fb87be0bb1d6c83d3cf"
Sep 11 19:45:28.754165 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.754109    2558 generic.go:356] "Generic (PLEG): container finished" podID="be735088-af15-41c7-b352-1630b1f37c82" containerID="35969ae95bc66602f117b951db32f66a3fc92d58338b4fba8da799f62d1b713a" exitCode=0
Sep 11 19:45:28.754165 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.754162    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn"
Sep 11 19:45:28.754386 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.754160    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn" event={"ID":"be735088-af15-41c7-b352-1630b1f37c82","Type":"ContainerDied","Data":"35969ae95bc66602f117b951db32f66a3fc92d58338b4fba8da799f62d1b713a"}
Sep 11 19:45:28.754386 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.754311    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn" event={"ID":"be735088-af15-41c7-b352-1630b1f37c82","Type":"ContainerDied","Data":"a9473c8f8986e4c49d3a52af5e39b297010336e5a319bb8496612367cd9732b1"}
Sep 11 19:45:28.760830 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.760697    2558 scope.go:117] "RemoveContainer" containerID="958700900d3f891cba8b8efe007df1e59408479ffae20fb87be0bb1d6c83d3cf"
Sep 11 19:45:28.761148 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:45:28.761121    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"958700900d3f891cba8b8efe007df1e59408479ffae20fb87be0bb1d6c83d3cf\": container with ID starting with 958700900d3f891cba8b8efe007df1e59408479ffae20fb87be0bb1d6c83d3cf not found: ID does not exist" containerID="958700900d3f891cba8b8efe007df1e59408479ffae20fb87be0bb1d6c83d3cf"
Sep 11 19:45:28.761232 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.761161    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"958700900d3f891cba8b8efe007df1e59408479ffae20fb87be0bb1d6c83d3cf"} err="failed to get container status \"958700900d3f891cba8b8efe007df1e59408479ffae20fb87be0bb1d6c83d3cf\": rpc error: code = NotFound desc = could not find container \"958700900d3f891cba8b8efe007df1e59408479ffae20fb87be0bb1d6c83d3cf\": container with ID starting with 958700900d3f891cba8b8efe007df1e59408479ffae20fb87be0bb1d6c83d3cf not found: ID does not exist"
Sep 11 19:45:28.761232 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.761185    2558 scope.go:117] "RemoveContainer" containerID="35969ae95bc66602f117b951db32f66a3fc92d58338b4fba8da799f62d1b713a"
Sep 11 19:45:28.773694 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.773671    2558 scope.go:117] "RemoveContainer" containerID="35969ae95bc66602f117b951db32f66a3fc92d58338b4fba8da799f62d1b713a"
Sep 11 19:45:28.774106 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:45:28.774082    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35969ae95bc66602f117b951db32f66a3fc92d58338b4fba8da799f62d1b713a\": container with ID starting with 35969ae95bc66602f117b951db32f66a3fc92d58338b4fba8da799f62d1b713a not found: ID does not exist" containerID="35969ae95bc66602f117b951db32f66a3fc92d58338b4fba8da799f62d1b713a"
Sep 11 19:45:28.774219 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.774110    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35969ae95bc66602f117b951db32f66a3fc92d58338b4fba8da799f62d1b713a"} err="failed to get container status \"35969ae95bc66602f117b951db32f66a3fc92d58338b4fba8da799f62d1b713a\": rpc error: code = NotFound desc = could not find container \"35969ae95bc66602f117b951db32f66a3fc92d58338b4fba8da799f62d1b713a\": container with ID starting with 35969ae95bc66602f117b951db32f66a3fc92d58338b4fba8da799f62d1b713a not found: ID does not exist"
Sep 11 19:45:28.780481 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.780451    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"]
Sep 11 19:45:28.784032 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.784005    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4c47b5e2-776f766d67-xq255"]
Sep 11 19:45:28.800173 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.800144    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn"]
Sep 11 19:45:28.805328 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:28.805303    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4c47b5e2-d86d96bc9-4cdmn"]
Sep 11 19:45:29.613250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.613208    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"]
Sep 11 19:45:29.613656 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.613643    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32810fb6-b1a4-433f-b633-4f4c17f085ea" containerName="maas-api"
Sep 11 19:45:29.613776 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.613658    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="32810fb6-b1a4-433f-b633-4f4c17f085ea" containerName="maas-api"
Sep 11 19:45:29.613776 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.613674    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be735088-af15-41c7-b352-1630b1f37c82" containerName="payload-processing"
Sep 11 19:45:29.613776 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.613679    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="be735088-af15-41c7-b352-1630b1f37c82" containerName="payload-processing"
Sep 11 19:45:29.613776 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.613760    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="be735088-af15-41c7-b352-1630b1f37c82" containerName="payload-processing"
Sep 11 19:45:29.613776 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.613773    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="32810fb6-b1a4-433f-b633-4f4c17f085ea" containerName="maas-api"
Sep 11 19:45:29.616808 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.616789    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.620415 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.620396    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-2816752e-gw-openshift-default-dockercfg-89t2j\""
Sep 11 19:45:29.637431 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.637393    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"]
Sep 11 19:45:29.772227 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.772189    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-workload-socket\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.772408 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.772251    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-envoy\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.772408 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.772310    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-data\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.772408 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.772349    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istiod-ca-cert\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.772408 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.772378    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-podinfo\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.772408 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.772402    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-credential-socket\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.772585 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.772463    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-workload-certs\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.772585 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.772511    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-token\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.772585 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.772538    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wngln\" (UniqueName: \"kubernetes.io/projected/51bd8f7e-fbf8-40db-8273-7c264b04ede5-kube-api-access-wngln\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.814789 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.814756    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32810fb6-b1a4-433f-b633-4f4c17f085ea" path="/var/lib/kubelet/pods/32810fb6-b1a4-433f-b633-4f4c17f085ea/volumes"
Sep 11 19:45:29.815171 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.815158    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be735088-af15-41c7-b352-1630b1f37c82" path="/var/lib/kubelet/pods/be735088-af15-41c7-b352-1630b1f37c82/volumes"
Sep 11 19:45:29.874213 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.874127    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istiod-ca-cert\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.874213 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.874175    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-podinfo\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.874213 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.874207    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-credential-socket\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.874483 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.874235    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-workload-certs\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.874483 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.874278    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-token\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.874483 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.874317    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wngln\" (UniqueName: \"kubernetes.io/projected/51bd8f7e-fbf8-40db-8273-7c264b04ede5-kube-api-access-wngln\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.874483 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.874400    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-workload-socket\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.874483 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.874446    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-envoy\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.874483 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.874471    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-data\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.874818 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.874696    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-workload-certs\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.874818 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.874796    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-credential-socket\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.874911 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.874821    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-data\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.875016 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.874990    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-workload-socket\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.875016 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.875006    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istiod-ca-cert\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.876701 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.876675    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-envoy\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.876922 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.876906    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-podinfo\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.887592 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.887562    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wngln\" (UniqueName: \"kubernetes.io/projected/51bd8f7e-fbf8-40db-8273-7c264b04ede5-kube-api-access-wngln\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.888095 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.888073    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-token\") pod \"e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") " pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:29.927281 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:29.927246    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:30.208227 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:30.208195    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"]
Sep 11 19:45:30.210153 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:45:30.210107    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51bd8f7e_fbf8_40db_8273_7c264b04ede5.slice/crio-3c0c2354c3e5a12ef6547e20a58d32ff3bed5b5ca6b78935a1b1f5f9fdc798bf WatchSource:0}: Error finding container 3c0c2354c3e5a12ef6547e20a58d32ff3bed5b5ca6b78935a1b1f5f9fdc798bf: Status 404 returned error can't find the container with id 3c0c2354c3e5a12ef6547e20a58d32ff3bed5b5ca6b78935a1b1f5f9fdc798bf
Sep 11 19:45:30.212587 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:30.212554    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:45:30.212681 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:30.212637    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:45:30.212768 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:30.212680    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:45:30.768382 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:30.768344    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl" event={"ID":"51bd8f7e-fbf8-40db-8273-7c264b04ede5","Type":"ContainerStarted","Data":"6b845fa28f2c32eca751e25c9a728e0465e4a04c3fdf8472b334e2ec38ac5d21"}
Sep 11 19:45:30.768889 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:30.768410    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl" event={"ID":"51bd8f7e-fbf8-40db-8273-7c264b04ede5","Type":"ContainerStarted","Data":"3c0c2354c3e5a12ef6547e20a58d32ff3bed5b5ca6b78935a1b1f5f9fdc798bf"}
Sep 11 19:45:30.794395 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:30.794338    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl" podStartSLOduration=1.7943192030000001 podStartE2EDuration="1.794319203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:45:30.791565629 +0000 UTC m=+1147.408735120" watchObservedRunningTime="2026-09-11 19:45:30.794319203 +0000 UTC m=+1147.411488694"
Sep 11 19:45:30.928252 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:30.928218    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:31.933321 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:31.933291    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:32.779079 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:32.779037    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:32.780123 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:32.780104    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:35.726388 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:35.726352    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"]
Sep 11 19:45:35.737905 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:35.737874    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"
Sep 11 19:45:35.743633 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:35.743600    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-2816752e-serving-cert\""
Sep 11 19:45:35.744451 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:35.744423    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"]
Sep 11 19:45:35.839423 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:35.839388    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tl5d\" (UniqueName: \"kubernetes.io/projected/6e340538-6a90-4450-acf7-066b02ef4bb5-kube-api-access-8tl5d\") pod \"maas-api-e2e-migrate-2816752e-65765594c8-jdn7r\" (UID: \"6e340538-6a90-4450-acf7-066b02ef4bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"
Sep 11 19:45:35.839423 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:35.839431    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e340538-6a90-4450-acf7-066b02ef4bb5-maas-api-tls\") pod \"maas-api-e2e-migrate-2816752e-65765594c8-jdn7r\" (UID: \"6e340538-6a90-4450-acf7-066b02ef4bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"
Sep 11 19:45:35.839665 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:35.839535    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e340538-6a90-4450-acf7-066b02ef4bb5-openshift-service-ca\") pod \"maas-api-e2e-migrate-2816752e-65765594c8-jdn7r\" (UID: \"6e340538-6a90-4450-acf7-066b02ef4bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"
Sep 11 19:45:35.839665 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:35.839569    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e340538-6a90-4450-acf7-066b02ef4bb5-metrics-certs\") pod \"maas-api-e2e-migrate-2816752e-65765594c8-jdn7r\" (UID: \"6e340538-6a90-4450-acf7-066b02ef4bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"
Sep 11 19:45:35.940812 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:35.940772    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e340538-6a90-4450-acf7-066b02ef4bb5-openshift-service-ca\") pod \"maas-api-e2e-migrate-2816752e-65765594c8-jdn7r\" (UID: \"6e340538-6a90-4450-acf7-066b02ef4bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"
Sep 11 19:45:35.940812 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:35.940816    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e340538-6a90-4450-acf7-066b02ef4bb5-metrics-certs\") pod \"maas-api-e2e-migrate-2816752e-65765594c8-jdn7r\" (UID: \"6e340538-6a90-4450-acf7-066b02ef4bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"
Sep 11 19:45:35.941054 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:35.940922    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tl5d\" (UniqueName: \"kubernetes.io/projected/6e340538-6a90-4450-acf7-066b02ef4bb5-kube-api-access-8tl5d\") pod \"maas-api-e2e-migrate-2816752e-65765594c8-jdn7r\" (UID: \"6e340538-6a90-4450-acf7-066b02ef4bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"
Sep 11 19:45:35.941054 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:35.940940    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e340538-6a90-4450-acf7-066b02ef4bb5-maas-api-tls\") pod \"maas-api-e2e-migrate-2816752e-65765594c8-jdn7r\" (UID: \"6e340538-6a90-4450-acf7-066b02ef4bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"
Sep 11 19:45:35.941054 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:45:35.941052    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-serving-cert: secret "maas-api-e2e-migrate-2816752e-serving-cert" not found
Sep 11 19:45:35.941225 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:45:35.941114    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e340538-6a90-4450-acf7-066b02ef4bb5-maas-api-tls podName:6e340538-6a90-4450-acf7-066b02ef4bb5 nodeName:}" failed. No retries permitted until 2026-09-11 19:45:36.441093657 +0000 UTC m=+1153.058263126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e340538-6a90-4450-acf7-066b02ef4bb5-maas-api-tls") pod "maas-api-e2e-migrate-2816752e-65765594c8-jdn7r" (UID: "6e340538-6a90-4450-acf7-066b02ef4bb5") : secret "maas-api-e2e-migrate-2816752e-serving-cert" not found
Sep 11 19:45:35.941588 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:35.941562    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e340538-6a90-4450-acf7-066b02ef4bb5-openshift-service-ca\") pod \"maas-api-e2e-migrate-2816752e-65765594c8-jdn7r\" (UID: \"6e340538-6a90-4450-acf7-066b02ef4bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"
Sep 11 19:45:35.943574 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:35.943543    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e340538-6a90-4450-acf7-066b02ef4bb5-metrics-certs\") pod \"maas-api-e2e-migrate-2816752e-65765594c8-jdn7r\" (UID: \"6e340538-6a90-4450-acf7-066b02ef4bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"
Sep 11 19:45:35.953315 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:35.953290    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tl5d\" (UniqueName: \"kubernetes.io/projected/6e340538-6a90-4450-acf7-066b02ef4bb5-kube-api-access-8tl5d\") pod \"maas-api-e2e-migrate-2816752e-65765594c8-jdn7r\" (UID: \"6e340538-6a90-4450-acf7-066b02ef4bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"
Sep 11 19:45:36.445454 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:36.445405    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e340538-6a90-4450-acf7-066b02ef4bb5-maas-api-tls\") pod \"maas-api-e2e-migrate-2816752e-65765594c8-jdn7r\" (UID: \"6e340538-6a90-4450-acf7-066b02ef4bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"
Sep 11 19:45:36.445663 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:45:36.445561    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-serving-cert: secret "maas-api-e2e-migrate-2816752e-serving-cert" not found
Sep 11 19:45:36.445663 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:45:36.445631    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e340538-6a90-4450-acf7-066b02ef4bb5-maas-api-tls podName:6e340538-6a90-4450-acf7-066b02ef4bb5 nodeName:}" failed. No retries permitted until 2026-09-11 19:45:37.445615008 +0000 UTC m=+1154.062784478 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e340538-6a90-4450-acf7-066b02ef4bb5-maas-api-tls") pod "maas-api-e2e-migrate-2816752e-65765594c8-jdn7r" (UID: "6e340538-6a90-4450-acf7-066b02ef4bb5") : secret "maas-api-e2e-migrate-2816752e-serving-cert" not found
Sep 11 19:45:37.456335 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:37.456294    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e340538-6a90-4450-acf7-066b02ef4bb5-maas-api-tls\") pod \"maas-api-e2e-migrate-2816752e-65765594c8-jdn7r\" (UID: \"6e340538-6a90-4450-acf7-066b02ef4bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"
Sep 11 19:45:37.458843 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:37.458818    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e340538-6a90-4450-acf7-066b02ef4bb5-maas-api-tls\") pod \"maas-api-e2e-migrate-2816752e-65765594c8-jdn7r\" (UID: \"6e340538-6a90-4450-acf7-066b02ef4bb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"
Sep 11 19:45:37.553516 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:37.553473    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"
Sep 11 19:45:37.832328 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:37.832293    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"]
Sep 11 19:45:37.833113 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:45:37.833089    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e340538_6a90_4450_acf7_066b02ef4bb5.slice/crio-065ae8d11da8b8b4bd5f89afb39928be794217a88abf7eefc73bcfa8303d0124 WatchSource:0}: Error finding container 065ae8d11da8b8b4bd5f89afb39928be794217a88abf7eefc73bcfa8303d0124: Status 404 returned error can't find the container with id 065ae8d11da8b8b4bd5f89afb39928be794217a88abf7eefc73bcfa8303d0124
Sep 11 19:45:38.814477 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:38.814440    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r" event={"ID":"6e340538-6a90-4450-acf7-066b02ef4bb5","Type":"ContainerStarted","Data":"56b914722acbdecb1d14c3973fa84060c4836b4e7e21d41bce7ed904db23252b"}
Sep 11 19:45:38.814874 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:38.814497    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"
Sep 11 19:45:38.814874 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:38.814514    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r" event={"ID":"6e340538-6a90-4450-acf7-066b02ef4bb5","Type":"ContainerStarted","Data":"065ae8d11da8b8b4bd5f89afb39928be794217a88abf7eefc73bcfa8303d0124"}
Sep 11 19:45:38.844922 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:38.844867    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r" podStartSLOduration=3.398555774 podStartE2EDuration="3.84485245s" totalImagesPullingTime="446.296676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:45:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:45:38.842481958 +0000 UTC m=+1155.459651451" watchObservedRunningTime="2026-09-11 19:45:38.84485245 +0000 UTC m=+1155.462021941"
Sep 11 19:45:44.824948 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:44.824917    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"
Sep 11 19:45:51.893273 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:51.893243    2558 generic.go:356] "Generic (PLEG): container finished" podID="51bd8f7e-fbf8-40db-8273-7c264b04ede5" containerID="6b845fa28f2c32eca751e25c9a728e0465e4a04c3fdf8472b334e2ec38ac5d21" exitCode=0
Sep 11 19:45:51.893681 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:51.893295    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl" event={"ID":"51bd8f7e-fbf8-40db-8273-7c264b04ede5","Type":"ContainerDied","Data":"6b845fa28f2c32eca751e25c9a728e0465e4a04c3fdf8472b334e2ec38ac5d21"}
Sep 11 19:45:51.893681 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:51.893673    2558 scope.go:117] "RemoveContainer" containerID="6b845fa28f2c32eca751e25c9a728e0465e4a04c3fdf8472b334e2ec38ac5d21"
Sep 11 19:45:51.894842 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:51.894809    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:45:51.894916 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:51.894900    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:45:51.894953 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:51.894942    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:45:52.900610 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:52.900569    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl" event={"ID":"51bd8f7e-fbf8-40db-8273-7c264b04ede5","Type":"ContainerStarted","Data":"e983b7cdd460fe90d3651838cd257c7d87001a30e71aeab808c7556d41f8718d"}
Sep 11 19:45:52.927541 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:52.927494    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:52.928059 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:52.927886    2558 patch_prober.go:28] interesting pod/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 11 19:45:52.928059 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:52.927951    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl" podUID="51bd8f7e-fbf8-40db-8273-7c264b04ede5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 11 19:45:53.928344 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:53.928303    2558 patch_prober.go:28] interesting pod/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 11 19:45:53.928746 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:53.928366    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl" podUID="51bd8f7e-fbf8-40db-8273-7c264b04ede5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 11 19:45:54.927929 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:54.927892    2558 patch_prober.go:28] interesting pod/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 11 19:45:54.928115 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:54.927963    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl" podUID="51bd8f7e-fbf8-40db-8273-7c264b04ede5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 11 19:45:55.928644 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:55.928604    2558 patch_prober.go:28] interesting pod/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 11 19:45:55.929054 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:55.928672    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl" podUID="51bd8f7e-fbf8-40db-8273-7c264b04ede5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 11 19:45:56.927684 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:56.927644    2558 patch_prober.go:28] interesting pod/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 11 19:45:56.927894 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:56.927743    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl" podUID="51bd8f7e-fbf8-40db-8273-7c264b04ede5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 11 19:45:57.928602 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:57.928486    2558 patch_prober.go:28] interesting pod/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Sep 11 19:45:57.928602 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:57.928565    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl" podUID="51bd8f7e-fbf8-40db-8273-7c264b04ede5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Sep 11 19:45:58.932465 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:58.932430    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:58.932940 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:58.932922    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:45:58.933626 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:45:58.933606    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:46:14.302972 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:14.302927    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"]
Sep 11 19:46:14.303448 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:14.303218    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r" podUID="6e340538-6a90-4450-acf7-066b02ef4bb5" containerName="maas-api" containerID="cri-o://56b914722acbdecb1d14c3973fa84060c4836b4e7e21d41bce7ed904db23252b" gracePeriod=30
Sep 11 19:46:14.555745 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:14.555666    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"
Sep 11 19:46:14.659694 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:14.659656    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e340538-6a90-4450-acf7-066b02ef4bb5-metrics-certs\") pod \"6e340538-6a90-4450-acf7-066b02ef4bb5\" (UID: \"6e340538-6a90-4450-acf7-066b02ef4bb5\") "
Sep 11 19:46:14.659899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:14.659831    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e340538-6a90-4450-acf7-066b02ef4bb5-openshift-service-ca\") pod \"6e340538-6a90-4450-acf7-066b02ef4bb5\" (UID: \"6e340538-6a90-4450-acf7-066b02ef4bb5\") "
Sep 11 19:46:14.659899 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:14.659858    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tl5d\" (UniqueName: \"kubernetes.io/projected/6e340538-6a90-4450-acf7-066b02ef4bb5-kube-api-access-8tl5d\") pod \"6e340538-6a90-4450-acf7-066b02ef4bb5\" (UID: \"6e340538-6a90-4450-acf7-066b02ef4bb5\") "
Sep 11 19:46:14.660040 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:14.659926    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e340538-6a90-4450-acf7-066b02ef4bb5-maas-api-tls\") pod \"6e340538-6a90-4450-acf7-066b02ef4bb5\" (UID: \"6e340538-6a90-4450-acf7-066b02ef4bb5\") "
Sep 11 19:46:14.660278 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:14.660211    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e340538-6a90-4450-acf7-066b02ef4bb5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6e340538-6a90-4450-acf7-066b02ef4bb5" (UID: "6e340538-6a90-4450-acf7-066b02ef4bb5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:46:14.660394 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:14.660372    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e340538-6a90-4450-acf7-066b02ef4bb5-openshift-service-ca\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:14.662171 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:14.662137    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e340538-6a90-4450-acf7-066b02ef4bb5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6e340538-6a90-4450-acf7-066b02ef4bb5" (UID: "6e340538-6a90-4450-acf7-066b02ef4bb5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:46:14.662281 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:14.662196    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e340538-6a90-4450-acf7-066b02ef4bb5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6e340538-6a90-4450-acf7-066b02ef4bb5" (UID: "6e340538-6a90-4450-acf7-066b02ef4bb5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:46:14.662281 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:14.662213    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e340538-6a90-4450-acf7-066b02ef4bb5-kube-api-access-8tl5d" (OuterVolumeSpecName: "kube-api-access-8tl5d") pod "6e340538-6a90-4450-acf7-066b02ef4bb5" (UID: "6e340538-6a90-4450-acf7-066b02ef4bb5"). InnerVolumeSpecName "kube-api-access-8tl5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:46:14.749050 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:14.749017    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"]
Sep 11 19:46:14.749268 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:14.749221    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl" podUID="51bd8f7e-fbf8-40db-8273-7c264b04ede5" containerName="istio-proxy" containerID="cri-o://e983b7cdd460fe90d3651838cd257c7d87001a30e71aeab808c7556d41f8718d" gracePeriod=30
Sep 11 19:46:14.761831 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:14.761790    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tl5d\" (UniqueName: \"kubernetes.io/projected/6e340538-6a90-4450-acf7-066b02ef4bb5-kube-api-access-8tl5d\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:14.761831 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:14.761831    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e340538-6a90-4450-acf7-066b02ef4bb5-maas-api-tls\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:14.762001 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:14.761841    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e340538-6a90-4450-acf7-066b02ef4bb5-metrics-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:15.020651 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.020616    2558 generic.go:356] "Generic (PLEG): container finished" podID="6e340538-6a90-4450-acf7-066b02ef4bb5" containerID="56b914722acbdecb1d14c3973fa84060c4836b4e7e21d41bce7ed904db23252b" exitCode=0
Sep 11 19:46:15.020869 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.020668    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"
Sep 11 19:46:15.020869 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.020718    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r" event={"ID":"6e340538-6a90-4450-acf7-066b02ef4bb5","Type":"ContainerDied","Data":"56b914722acbdecb1d14c3973fa84060c4836b4e7e21d41bce7ed904db23252b"}
Sep 11 19:46:15.020869 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.020780    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r" event={"ID":"6e340538-6a90-4450-acf7-066b02ef4bb5","Type":"ContainerDied","Data":"065ae8d11da8b8b4bd5f89afb39928be794217a88abf7eefc73bcfa8303d0124"}
Sep 11 19:46:15.020869 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.020806    2558 scope.go:117] "RemoveContainer" containerID="56b914722acbdecb1d14c3973fa84060c4836b4e7e21d41bce7ed904db23252b"
Sep 11 19:46:15.031250 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.031228    2558 scope.go:117] "RemoveContainer" containerID="56b914722acbdecb1d14c3973fa84060c4836b4e7e21d41bce7ed904db23252b"
Sep 11 19:46:15.031538 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:46:15.031518    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b914722acbdecb1d14c3973fa84060c4836b4e7e21d41bce7ed904db23252b\": container with ID starting with 56b914722acbdecb1d14c3973fa84060c4836b4e7e21d41bce7ed904db23252b not found: ID does not exist" containerID="56b914722acbdecb1d14c3973fa84060c4836b4e7e21d41bce7ed904db23252b"
Sep 11 19:46:15.031588 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.031549    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b914722acbdecb1d14c3973fa84060c4836b4e7e21d41bce7ed904db23252b"} err="failed to get container status \"56b914722acbdecb1d14c3973fa84060c4836b4e7e21d41bce7ed904db23252b\": rpc error: code = NotFound desc = could not find container \"56b914722acbdecb1d14c3973fa84060c4836b4e7e21d41bce7ed904db23252b\": container with ID starting with 56b914722acbdecb1d14c3973fa84060c4836b4e7e21d41bce7ed904db23252b not found: ID does not exist"
Sep 11 19:46:15.045882 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.045852    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"]
Sep 11 19:46:15.051873 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.051845    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2816752e-65765594c8-jdn7r"]
Sep 11 19:46:15.499425 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.499388    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"]
Sep 11 19:46:15.499874 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.499856    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e340538-6a90-4450-acf7-066b02ef4bb5" containerName="maas-api"
Sep 11 19:46:15.499874 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.499873    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e340538-6a90-4450-acf7-066b02ef4bb5" containerName="maas-api"
Sep 11 19:46:15.500020 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.499953    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e340538-6a90-4450-acf7-066b02ef4bb5" containerName="maas-api"
Sep 11 19:46:15.502672 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.502653    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.505027 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.505010    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8901a783-openshift-default-dockercfg-5k84w\""
Sep 11 19:46:15.517305 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.517282    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"]
Sep 11 19:46:15.570836 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.570796    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-data\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.570836 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.570834    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f2b635-0daa-41bc-854f-5dc46247ae47-istiod-ca-cert\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.571050 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.570858    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-envoy\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.571050 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.570940    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-token\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.571050 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.571018    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t5zz\" (UniqueName: \"kubernetes.io/projected/b8f2b635-0daa-41bc-854f-5dc46247ae47-kube-api-access-6t5zz\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.571050 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.571049    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-workload-certs\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.571186 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.571072    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-workload-socket\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.571186 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.571106    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-podinfo\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.571186 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.571122    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-credential-socket\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.672410 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.672365    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-podinfo\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.672633 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.672428    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-credential-socket\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.672633 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.672486    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-data\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.672633 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.672513    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f2b635-0daa-41bc-854f-5dc46247ae47-istiod-ca-cert\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.672633 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.672545    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-envoy\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.672633 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.672597    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-token\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.672970 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.672750    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t5zz\" (UniqueName: \"kubernetes.io/projected/b8f2b635-0daa-41bc-854f-5dc46247ae47-kube-api-access-6t5zz\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.672970 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.672802    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-workload-certs\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.672970 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.672844    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-workload-socket\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.672970 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.672943    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-credential-socket\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.673184 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.673135    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-workload-socket\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.673521 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.673489    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-data\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.673857 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.673829    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-workload-certs\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.674303 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.674270    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f2b635-0daa-41bc-854f-5dc46247ae47-istiod-ca-cert\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.675780 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.675754    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-podinfo\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.675976 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.675957    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-envoy\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.682508 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.682482    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-token\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.683397 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.683376    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t5zz\" (UniqueName: \"kubernetes.io/projected/b8f2b635-0daa-41bc-854f-5dc46247ae47-kube-api-access-6t5zz\") pod \"e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") " pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.813327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.813243    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:15.814512 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:15.814486    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e340538-6a90-4450-acf7-066b02ef4bb5" path="/var/lib/kubelet/pods/6e340538-6a90-4450-acf7-066b02ef4bb5/volumes"
Sep 11 19:46:16.092254 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:16.092230    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"]
Sep 11 19:46:16.094995 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:46:16.094963    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f2b635_0daa_41bc_854f_5dc46247ae47.slice/crio-ad63caa6f108b3c366731bf79a394051d754becac1aa300da1a3e6496202b93c WatchSource:0}: Error finding container ad63caa6f108b3c366731bf79a394051d754becac1aa300da1a3e6496202b93c: Status 404 returned error can't find the container with id ad63caa6f108b3c366731bf79a394051d754becac1aa300da1a3e6496202b93c
Sep 11 19:46:16.097192 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:16.097159    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:46:16.097262 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:16.097229    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:46:16.097300 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:16.097261    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:46:17.035443 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:17.035406    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg" event={"ID":"b8f2b635-0daa-41bc-854f-5dc46247ae47","Type":"ContainerStarted","Data":"bf76ab94a4f88c50e13969a8d83f57a6225b681526b0dd33fec6f72518826106"}
Sep 11 19:46:17.035981 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:17.035472    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg" event={"ID":"b8f2b635-0daa-41bc-854f-5dc46247ae47","Type":"ContainerStarted","Data":"ad63caa6f108b3c366731bf79a394051d754becac1aa300da1a3e6496202b93c"}
Sep 11 19:46:17.059078 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:17.059024    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg" podStartSLOduration=2.059007105 podStartE2EDuration="2.059007105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:46:17.056770796 +0000 UTC m=+1193.673940305" watchObservedRunningTime="2026-09-11 19:46:17.059007105 +0000 UTC m=+1193.676176595"
Sep 11 19:46:17.814755 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:17.814724    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:17.819045 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:17.819013    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:18.040553 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:18.040523    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:18.041484 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:18.041464    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:20.015252 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.015182    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:46:20.054367 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.054330    2558 generic.go:356] "Generic (PLEG): container finished" podID="51bd8f7e-fbf8-40db-8273-7c264b04ede5" containerID="e983b7cdd460fe90d3651838cd257c7d87001a30e71aeab808c7556d41f8718d" exitCode=0
Sep 11 19:46:20.054551 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.054380    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"
Sep 11 19:46:20.054551 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.054410    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl" event={"ID":"51bd8f7e-fbf8-40db-8273-7c264b04ede5","Type":"ContainerDied","Data":"e983b7cdd460fe90d3651838cd257c7d87001a30e71aeab808c7556d41f8718d"}
Sep 11 19:46:20.054551 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.054463    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl" event={"ID":"51bd8f7e-fbf8-40db-8273-7c264b04ede5","Type":"ContainerDied","Data":"3c0c2354c3e5a12ef6547e20a58d32ff3bed5b5ca6b78935a1b1f5f9fdc798bf"}
Sep 11 19:46:20.054551 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.054488    2558 scope.go:117] "RemoveContainer" containerID="e983b7cdd460fe90d3651838cd257c7d87001a30e71aeab808c7556d41f8718d"
Sep 11 19:46:20.063933 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.063902    2558 scope.go:117] "RemoveContainer" containerID="6b845fa28f2c32eca751e25c9a728e0465e4a04c3fdf8472b334e2ec38ac5d21"
Sep 11 19:46:20.073176 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.073151    2558 scope.go:117] "RemoveContainer" containerID="e983b7cdd460fe90d3651838cd257c7d87001a30e71aeab808c7556d41f8718d"
Sep 11 19:46:20.073513 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:46:20.073485    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e983b7cdd460fe90d3651838cd257c7d87001a30e71aeab808c7556d41f8718d\": container with ID starting with e983b7cdd460fe90d3651838cd257c7d87001a30e71aeab808c7556d41f8718d not found: ID does not exist" containerID="e983b7cdd460fe90d3651838cd257c7d87001a30e71aeab808c7556d41f8718d"
Sep 11 19:46:20.073609 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.073526    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e983b7cdd460fe90d3651838cd257c7d87001a30e71aeab808c7556d41f8718d"} err="failed to get container status \"e983b7cdd460fe90d3651838cd257c7d87001a30e71aeab808c7556d41f8718d\": rpc error: code = NotFound desc = could not find container \"e983b7cdd460fe90d3651838cd257c7d87001a30e71aeab808c7556d41f8718d\": container with ID starting with e983b7cdd460fe90d3651838cd257c7d87001a30e71aeab808c7556d41f8718d not found: ID does not exist"
Sep 11 19:46:20.073609 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.073552    2558 scope.go:117] "RemoveContainer" containerID="6b845fa28f2c32eca751e25c9a728e0465e4a04c3fdf8472b334e2ec38ac5d21"
Sep 11 19:46:20.073892 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:46:20.073869    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b845fa28f2c32eca751e25c9a728e0465e4a04c3fdf8472b334e2ec38ac5d21\": container with ID starting with 6b845fa28f2c32eca751e25c9a728e0465e4a04c3fdf8472b334e2ec38ac5d21 not found: ID does not exist" containerID="6b845fa28f2c32eca751e25c9a728e0465e4a04c3fdf8472b334e2ec38ac5d21"
Sep 11 19:46:20.073958 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.073904    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b845fa28f2c32eca751e25c9a728e0465e4a04c3fdf8472b334e2ec38ac5d21"} err="failed to get container status \"6b845fa28f2c32eca751e25c9a728e0465e4a04c3fdf8472b334e2ec38ac5d21\": rpc error: code = NotFound desc = could not find container \"6b845fa28f2c32eca751e25c9a728e0465e4a04c3fdf8472b334e2ec38ac5d21\": container with ID starting with 6b845fa28f2c32eca751e25c9a728e0465e4a04c3fdf8472b334e2ec38ac5d21 not found: ID does not exist"
Sep 11 19:46:20.124109 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.124073    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-credential-socket\") pod \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") "
Sep 11 19:46:20.124109 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.124112    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-token\") pod \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") "
Sep 11 19:46:20.124314 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.124142    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-data\") pod \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") "
Sep 11 19:46:20.124314 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.124207    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-podinfo\") pod \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") "
Sep 11 19:46:20.124314 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.124274    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-workload-certs\") pod \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") "
Sep 11 19:46:20.124432 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.124343    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-workload-socket\") pod \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") "
Sep 11 19:46:20.124432 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.124392    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wngln\" (UniqueName: \"kubernetes.io/projected/51bd8f7e-fbf8-40db-8273-7c264b04ede5-kube-api-access-wngln\") pod \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") "
Sep 11 19:46:20.124540 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.124442    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-envoy\") pod \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") "
Sep 11 19:46:20.124540 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.124467    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istiod-ca-cert\") pod \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\" (UID: \"51bd8f7e-fbf8-40db-8273-7c264b04ede5\") "
Sep 11 19:46:20.124540 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.124438    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "51bd8f7e-fbf8-40db-8273-7c264b04ede5" (UID: "51bd8f7e-fbf8-40db-8273-7c264b04ede5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:46:20.124691 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.124619    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "51bd8f7e-fbf8-40db-8273-7c264b04ede5" (UID: "51bd8f7e-fbf8-40db-8273-7c264b04ede5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:46:20.124691 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.124495    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-data" (OuterVolumeSpecName: "istio-data") pod "51bd8f7e-fbf8-40db-8273-7c264b04ede5" (UID: "51bd8f7e-fbf8-40db-8273-7c264b04ede5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:46:20.124691 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.124642    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "51bd8f7e-fbf8-40db-8273-7c264b04ede5" (UID: "51bd8f7e-fbf8-40db-8273-7c264b04ede5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:46:20.124926 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.124904    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "51bd8f7e-fbf8-40db-8273-7c264b04ede5" (UID: "51bd8f7e-fbf8-40db-8273-7c264b04ede5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:46:20.125089 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.125070    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-workload-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:20.125156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.125093    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-workload-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:20.125156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.125106    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istiod-ca-cert\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:20.125156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.125120    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-credential-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:20.125156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.125133    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-data\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:20.126576 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.126540    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51bd8f7e-fbf8-40db-8273-7c264b04ede5-kube-api-access-wngln" (OuterVolumeSpecName: "kube-api-access-wngln") pod "51bd8f7e-fbf8-40db-8273-7c264b04ede5" (UID: "51bd8f7e-fbf8-40db-8273-7c264b04ede5"). InnerVolumeSpecName "kube-api-access-wngln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:46:20.126760 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.126644    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "51bd8f7e-fbf8-40db-8273-7c264b04ede5" (UID: "51bd8f7e-fbf8-40db-8273-7c264b04ede5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:46:20.126988 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.126964    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "51bd8f7e-fbf8-40db-8273-7c264b04ede5" (UID: "51bd8f7e-fbf8-40db-8273-7c264b04ede5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:46:20.127082 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.127034    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-token" (OuterVolumeSpecName: "istio-token") pod "51bd8f7e-fbf8-40db-8273-7c264b04ede5" (UID: "51bd8f7e-fbf8-40db-8273-7c264b04ede5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:46:20.226257 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.226217    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wngln\" (UniqueName: \"kubernetes.io/projected/51bd8f7e-fbf8-40db-8273-7c264b04ede5-kube-api-access-wngln\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:20.226257 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.226251    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-envoy\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:20.226257 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.226261    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-token\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:20.226502 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.226270    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51bd8f7e-fbf8-40db-8273-7c264b04ede5-istio-podinfo\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:20.378577 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.378534    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"]
Sep 11 19:46:20.384101 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.384066    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-2816752e-gw-openshift-default-5c88466fb-g4pjl"]
Sep 11 19:46:20.422670 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.422632    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"]
Sep 11 19:46:20.423502 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.423404    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51bd8f7e-fbf8-40db-8273-7c264b04ede5" containerName="istio-proxy"
Sep 11 19:46:20.423502 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.423434    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="51bd8f7e-fbf8-40db-8273-7c264b04ede5" containerName="istio-proxy"
Sep 11 19:46:20.423687 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.423593    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="51bd8f7e-fbf8-40db-8273-7c264b04ede5" containerName="istio-proxy"
Sep 11 19:46:20.423687 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.423613    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="51bd8f7e-fbf8-40db-8273-7c264b04ede5" containerName="istio-proxy"
Sep 11 19:46:20.426478 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.426457    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"
Sep 11 19:46:20.428743 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.428695    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8901a783-serving-cert\""
Sep 11 19:46:20.440757 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.440686    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"]
Sep 11 19:46:20.528852 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.528818    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56zj8\" (UniqueName: \"kubernetes.io/projected/8fc8039c-e9f8-4022-b1bd-b22b6541c207-kube-api-access-56zj8\") pod \"maas-api-e2e-ait-8901a783-677f864fbb-nnbnf\" (UID: \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"
Sep 11 19:46:20.529031 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.528956    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fc8039c-e9f8-4022-b1bd-b22b6541c207-metrics-certs\") pod \"maas-api-e2e-ait-8901a783-677f864fbb-nnbnf\" (UID: \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"
Sep 11 19:46:20.529031 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.528993    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fc8039c-e9f8-4022-b1bd-b22b6541c207-openshift-service-ca\") pod \"maas-api-e2e-ait-8901a783-677f864fbb-nnbnf\" (UID: \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"
Sep 11 19:46:20.529104 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.529056    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fc8039c-e9f8-4022-b1bd-b22b6541c207-maas-api-tls\") pod \"maas-api-e2e-ait-8901a783-677f864fbb-nnbnf\" (UID: \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"
Sep 11 19:46:20.630025 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.629926    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fc8039c-e9f8-4022-b1bd-b22b6541c207-maas-api-tls\") pod \"maas-api-e2e-ait-8901a783-677f864fbb-nnbnf\" (UID: \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"
Sep 11 19:46:20.630025 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.629999    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56zj8\" (UniqueName: \"kubernetes.io/projected/8fc8039c-e9f8-4022-b1bd-b22b6541c207-kube-api-access-56zj8\") pod \"maas-api-e2e-ait-8901a783-677f864fbb-nnbnf\" (UID: \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"
Sep 11 19:46:20.630273 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:46:20.630077    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-serving-cert: secret "maas-api-e2e-ait-8901a783-serving-cert" not found
Sep 11 19:46:20.630273 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:46:20.630142    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fc8039c-e9f8-4022-b1bd-b22b6541c207-maas-api-tls podName:8fc8039c-e9f8-4022-b1bd-b22b6541c207 nodeName:}" failed. No retries permitted until 2026-09-11 19:46:21.130126502 +0000 UTC m=+1197.747295975 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8fc8039c-e9f8-4022-b1bd-b22b6541c207-maas-api-tls") pod "maas-api-e2e-ait-8901a783-677f864fbb-nnbnf" (UID: "8fc8039c-e9f8-4022-b1bd-b22b6541c207") : secret "maas-api-e2e-ait-8901a783-serving-cert" not found
Sep 11 19:46:20.630273 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.630087    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fc8039c-e9f8-4022-b1bd-b22b6541c207-metrics-certs\") pod \"maas-api-e2e-ait-8901a783-677f864fbb-nnbnf\" (UID: \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"
Sep 11 19:46:20.630273 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.630215    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fc8039c-e9f8-4022-b1bd-b22b6541c207-openshift-service-ca\") pod \"maas-api-e2e-ait-8901a783-677f864fbb-nnbnf\" (UID: \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"
Sep 11 19:46:20.630868 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.630850    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fc8039c-e9f8-4022-b1bd-b22b6541c207-openshift-service-ca\") pod \"maas-api-e2e-ait-8901a783-677f864fbb-nnbnf\" (UID: \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"
Sep 11 19:46:20.632495 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.632476    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fc8039c-e9f8-4022-b1bd-b22b6541c207-metrics-certs\") pod \"maas-api-e2e-ait-8901a783-677f864fbb-nnbnf\" (UID: \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"
Sep 11 19:46:20.641959 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:20.641938    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56zj8\" (UniqueName: \"kubernetes.io/projected/8fc8039c-e9f8-4022-b1bd-b22b6541c207-kube-api-access-56zj8\") pod \"maas-api-e2e-ait-8901a783-677f864fbb-nnbnf\" (UID: \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"
Sep 11 19:46:21.135325 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:21.135285    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fc8039c-e9f8-4022-b1bd-b22b6541c207-maas-api-tls\") pod \"maas-api-e2e-ait-8901a783-677f864fbb-nnbnf\" (UID: \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"
Sep 11 19:46:21.137835 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:21.137804    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fc8039c-e9f8-4022-b1bd-b22b6541c207-maas-api-tls\") pod \"maas-api-e2e-ait-8901a783-677f864fbb-nnbnf\" (UID: \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"
Sep 11 19:46:21.337762 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:21.337723    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"
Sep 11 19:46:21.618526 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:21.618494    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"]
Sep 11 19:46:21.620467 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:46:21.620439    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc8039c_e9f8_4022_b1bd_b22b6541c207.slice/crio-71bc8672025632d08f8165cd0d0428f719f2c18ea962bd769ec550065155e919 WatchSource:0}: Error finding container 71bc8672025632d08f8165cd0d0428f719f2c18ea962bd769ec550065155e919: Status 404 returned error can't find the container with id 71bc8672025632d08f8165cd0d0428f719f2c18ea962bd769ec550065155e919
Sep 11 19:46:21.814536 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:21.814501    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51bd8f7e-fbf8-40db-8273-7c264b04ede5" path="/var/lib/kubelet/pods/51bd8f7e-fbf8-40db-8273-7c264b04ede5/volumes"
Sep 11 19:46:22.069184 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:22.069083    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf" event={"ID":"8fc8039c-e9f8-4022-b1bd-b22b6541c207","Type":"ContainerStarted","Data":"6fe73a52e3db7ad005195c70689cbc4f94ad50559fe3a1882999d23dff1289a2"}
Sep 11 19:46:22.069184 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:22.069146    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"
Sep 11 19:46:22.069184 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:22.069164    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf" event={"ID":"8fc8039c-e9f8-4022-b1bd-b22b6541c207","Type":"ContainerStarted","Data":"71bc8672025632d08f8165cd0d0428f719f2c18ea962bd769ec550065155e919"}
Sep 11 19:46:22.091370 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:22.091314    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf" podStartSLOduration=1.749839218 podStartE2EDuration="2.091298442s" totalImagesPullingTime="341.459224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:46:22.090632297 +0000 UTC m=+1198.707801789" watchObservedRunningTime="2026-09-11 19:46:22.091298442 +0000 UTC m=+1198.708467933"
Sep 11 19:46:28.081139 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:28.081097    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"
Sep 11 19:46:41.175448 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:41.175412    2558 generic.go:356] "Generic (PLEG): container finished" podID="b8f2b635-0daa-41bc-854f-5dc46247ae47" containerID="bf76ab94a4f88c50e13969a8d83f57a6225b681526b0dd33fec6f72518826106" exitCode=0
Sep 11 19:46:41.176004 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:41.175490    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg" event={"ID":"b8f2b635-0daa-41bc-854f-5dc46247ae47","Type":"ContainerDied","Data":"bf76ab94a4f88c50e13969a8d83f57a6225b681526b0dd33fec6f72518826106"}
Sep 11 19:46:41.176151 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:41.176133    2558 scope.go:117] "RemoveContainer" containerID="bf76ab94a4f88c50e13969a8d83f57a6225b681526b0dd33fec6f72518826106"
Sep 11 19:46:41.177377 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:41.177344    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:46:41.177442 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:41.177419    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:46:41.177481 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:41.177453    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:46:42.182808 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:42.182775    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg" event={"ID":"b8f2b635-0daa-41bc-854f-5dc46247ae47","Type":"ContainerStarted","Data":"c6f81862bb1043331bbc0ec710d051c40e836d4dd5f6dd95c02bc142990da829"}
Sep 11 19:46:42.813562 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:42.813522    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:42.813798 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:42.813755    2558 patch_prober.go:28] interesting pod/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 11 19:46:42.813864 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:42.813825    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg" podUID="b8f2b635-0daa-41bc-854f-5dc46247ae47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 11 19:46:43.813858 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:43.813820    2558 patch_prober.go:28] interesting pod/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 11 19:46:43.814372 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:43.813908    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg" podUID="b8f2b635-0daa-41bc-854f-5dc46247ae47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 11 19:46:44.814079 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:44.814042    2558 patch_prober.go:28] interesting pod/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 11 19:46:44.814470 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:44.814095    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg" podUID="b8f2b635-0daa-41bc-854f-5dc46247ae47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 11 19:46:45.814011 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:45.813977    2558 patch_prober.go:28] interesting pod/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 11 19:46:45.814212 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:45.814031    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg" podUID="b8f2b635-0daa-41bc-854f-5dc46247ae47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 11 19:46:46.814005 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:46.813967    2558 patch_prober.go:28] interesting pod/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body=
Sep 11 19:46:46.814258 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:46.814034    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg" podUID="b8f2b635-0daa-41bc-854f-5dc46247ae47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused"
Sep 11 19:46:47.823815 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:47.823775    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:47.824501 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:47.824472    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:47.825550 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:47.825529    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:51.849277 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:51.849237    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"]
Sep 11 19:46:51.849750 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:51.849492    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf" podUID="8fc8039c-e9f8-4022-b1bd-b22b6541c207" containerName="maas-api" containerID="cri-o://6fe73a52e3db7ad005195c70689cbc4f94ad50559fe3a1882999d23dff1289a2" gracePeriod=30
Sep 11 19:46:52.113867 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.113807    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"
Sep 11 19:46:52.171581 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.171539    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fc8039c-e9f8-4022-b1bd-b22b6541c207-metrics-certs\") pod \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\" (UID: \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\") "
Sep 11 19:46:52.171803 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.171642    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fc8039c-e9f8-4022-b1bd-b22b6541c207-maas-api-tls\") pod \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\" (UID: \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\") "
Sep 11 19:46:52.171803 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.171680    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56zj8\" (UniqueName: \"kubernetes.io/projected/8fc8039c-e9f8-4022-b1bd-b22b6541c207-kube-api-access-56zj8\") pod \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\" (UID: \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\") "
Sep 11 19:46:52.171803 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.171774    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fc8039c-e9f8-4022-b1bd-b22b6541c207-openshift-service-ca\") pod \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\" (UID: \"8fc8039c-e9f8-4022-b1bd-b22b6541c207\") "
Sep 11 19:46:52.172532 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.172480    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc8039c-e9f8-4022-b1bd-b22b6541c207-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8fc8039c-e9f8-4022-b1bd-b22b6541c207" (UID: "8fc8039c-e9f8-4022-b1bd-b22b6541c207"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:46:52.174264 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.174223    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc8039c-e9f8-4022-b1bd-b22b6541c207-kube-api-access-56zj8" (OuterVolumeSpecName: "kube-api-access-56zj8") pod "8fc8039c-e9f8-4022-b1bd-b22b6541c207" (UID: "8fc8039c-e9f8-4022-b1bd-b22b6541c207"). InnerVolumeSpecName "kube-api-access-56zj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:46:52.174386 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.174273    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc8039c-e9f8-4022-b1bd-b22b6541c207-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8fc8039c-e9f8-4022-b1bd-b22b6541c207" (UID: "8fc8039c-e9f8-4022-b1bd-b22b6541c207"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:46:52.174534 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.174506    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc8039c-e9f8-4022-b1bd-b22b6541c207-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8fc8039c-e9f8-4022-b1bd-b22b6541c207" (UID: "8fc8039c-e9f8-4022-b1bd-b22b6541c207"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:46:52.241222 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.241187    2558 generic.go:356] "Generic (PLEG): container finished" podID="8fc8039c-e9f8-4022-b1bd-b22b6541c207" containerID="6fe73a52e3db7ad005195c70689cbc4f94ad50559fe3a1882999d23dff1289a2" exitCode=0
Sep 11 19:46:52.241406 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.241238    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"
Sep 11 19:46:52.241406 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.241233    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf" event={"ID":"8fc8039c-e9f8-4022-b1bd-b22b6541c207","Type":"ContainerDied","Data":"6fe73a52e3db7ad005195c70689cbc4f94ad50559fe3a1882999d23dff1289a2"}
Sep 11 19:46:52.241406 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.241360    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf" event={"ID":"8fc8039c-e9f8-4022-b1bd-b22b6541c207","Type":"ContainerDied","Data":"71bc8672025632d08f8165cd0d0428f719f2c18ea962bd769ec550065155e919"}
Sep 11 19:46:52.241406 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.241385    2558 scope.go:117] "RemoveContainer" containerID="6fe73a52e3db7ad005195c70689cbc4f94ad50559fe3a1882999d23dff1289a2"
Sep 11 19:46:52.253645 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.253624    2558 scope.go:117] "RemoveContainer" containerID="6fe73a52e3db7ad005195c70689cbc4f94ad50559fe3a1882999d23dff1289a2"
Sep 11 19:46:52.254116 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:46:52.254096    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fe73a52e3db7ad005195c70689cbc4f94ad50559fe3a1882999d23dff1289a2\": container with ID starting with 6fe73a52e3db7ad005195c70689cbc4f94ad50559fe3a1882999d23dff1289a2 not found: ID does not exist" containerID="6fe73a52e3db7ad005195c70689cbc4f94ad50559fe3a1882999d23dff1289a2"
Sep 11 19:46:52.254198 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.254129    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe73a52e3db7ad005195c70689cbc4f94ad50559fe3a1882999d23dff1289a2"} err="failed to get container status \"6fe73a52e3db7ad005195c70689cbc4f94ad50559fe3a1882999d23dff1289a2\": rpc error: code = NotFound desc = could not find container \"6fe73a52e3db7ad005195c70689cbc4f94ad50559fe3a1882999d23dff1289a2\": container with ID starting with 6fe73a52e3db7ad005195c70689cbc4f94ad50559fe3a1882999d23dff1289a2 not found: ID does not exist"
Sep 11 19:46:52.273385 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.273341    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8fc8039c-e9f8-4022-b1bd-b22b6541c207-maas-api-tls\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:52.273385 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.273380    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56zj8\" (UniqueName: \"kubernetes.io/projected/8fc8039c-e9f8-4022-b1bd-b22b6541c207-kube-api-access-56zj8\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:52.273385 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.273397    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fc8039c-e9f8-4022-b1bd-b22b6541c207-openshift-service-ca\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:52.273854 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.273413    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8fc8039c-e9f8-4022-b1bd-b22b6541c207-metrics-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:52.294150 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.294107    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"]
Sep 11 19:46:52.302496 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:52.302459    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8901a783-677f864fbb-nnbnf"]
Sep 11 19:46:53.372731 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:53.372675    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"]
Sep 11 19:46:53.373142 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:53.372958    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg" podUID="b8f2b635-0daa-41bc-854f-5dc46247ae47" containerName="istio-proxy" containerID="cri-o://c6f81862bb1043331bbc0ec710d051c40e836d4dd5f6dd95c02bc142990da829" gracePeriod=30
Sep 11 19:46:53.814546 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:53.814507    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fc8039c-e9f8-4022-b1bd-b22b6541c207" path="/var/lib/kubelet/pods/8fc8039c-e9f8-4022-b1bd-b22b6541c207/volumes"
Sep 11 19:46:54.175817 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.175782    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"]
Sep 11 19:46:54.176275 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.176256    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51bd8f7e-fbf8-40db-8273-7c264b04ede5" containerName="istio-proxy"
Sep 11 19:46:54.176327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.176274    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="51bd8f7e-fbf8-40db-8273-7c264b04ede5" containerName="istio-proxy"
Sep 11 19:46:54.176327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.176295    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc8039c-e9f8-4022-b1bd-b22b6541c207" containerName="maas-api"
Sep 11 19:46:54.176327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.176301    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc8039c-e9f8-4022-b1bd-b22b6541c207" containerName="maas-api"
Sep 11 19:46:54.176421 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.176369    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fc8039c-e9f8-4022-b1bd-b22b6541c207" containerName="maas-api"
Sep 11 19:46:54.180696 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.180678    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.183529 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.183502    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-0d707fde-openshift-default-dockercfg-2t55m\""
Sep 11 19:46:54.195191 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.195161    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"]
Sep 11 19:46:54.296432 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.296397    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-envoy\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.296625 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.296462    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-token\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.296625 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.296489    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-credential-socket\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.296625 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.296510    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-workload-certs\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.296625 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.296600    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-podinfo\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.296871 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.296723    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-workload-socket\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.296871 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.296760    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istiod-ca-cert\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.296871 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.296792    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4msb\" (UniqueName: \"kubernetes.io/projected/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-kube-api-access-k4msb\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.296871 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.296820    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-data\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.397867 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.397826    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-workload-socket\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.397867 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.397872    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istiod-ca-cert\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.398407 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.397968    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4msb\" (UniqueName: \"kubernetes.io/projected/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-kube-api-access-k4msb\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.398407 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.397990    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-data\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.398407 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.398011    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-envoy\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.398407 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.398042    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-token\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.398407 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.398063    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-credential-socket\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.398407 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.398089    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-workload-certs\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.398407 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.398133    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-podinfo\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.398407 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.398297    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-workload-socket\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.398880 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.398614    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-credential-socket\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.398880 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.398653    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-data\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.398880 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.398689    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-workload-certs\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.399011 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.398903    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istiod-ca-cert\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.400605 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.400580    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-envoy\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.400845 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.400828    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-podinfo\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.411457 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.411429    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-token\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.411977 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.411957    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4msb\" (UniqueName: \"kubernetes.io/projected/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-kube-api-access-k4msb\") pod \"e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") " pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.492056 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.491965    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:54.782010 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.781980    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"]
Sep 11 19:46:54.782542 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:46:54.782521    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0113daa_a4a1_4bb1_9ad5_835ac98f201d.slice/crio-73a49f9c63c199274f40cc328b74b2b1e213dce45c0033be099a49251edc3679 WatchSource:0}: Error finding container 73a49f9c63c199274f40cc328b74b2b1e213dce45c0033be099a49251edc3679: Status 404 returned error can't find the container with id 73a49f9c63c199274f40cc328b74b2b1e213dce45c0033be099a49251edc3679
Sep 11 19:46:54.784897 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.784862    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:46:54.784975 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.784930    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:46:54.784975 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:54.784960    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:46:55.266724 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:55.266668    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x" event={"ID":"d0113daa-a4a1-4bb1-9ad5-835ac98f201d","Type":"ContainerStarted","Data":"3a450071d6b64d0583c7945461ceb53256a1bc60244772d1cd69935e5496bc2b"}
Sep 11 19:46:55.266897 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:55.266795    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x" event={"ID":"d0113daa-a4a1-4bb1-9ad5-835ac98f201d","Type":"ContainerStarted","Data":"73a49f9c63c199274f40cc328b74b2b1e213dce45c0033be099a49251edc3679"}
Sep 11 19:46:55.295298 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:55.295238    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x" podStartSLOduration=1.295218872 podStartE2EDuration="1.295218872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:46:55.291336099 +0000 UTC m=+1231.908505591" watchObservedRunningTime="2026-09-11 19:46:55.295218872 +0000 UTC m=+1231.912388363"
Sep 11 19:46:55.492176 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:55.492138    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:55.498462 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:55.498432    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:56.272669 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:56.272635    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:56.273807 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:56.273787    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:46:58.630518 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.630485    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:58.745761 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.745724    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t5zz\" (UniqueName: \"kubernetes.io/projected/b8f2b635-0daa-41bc-854f-5dc46247ae47-kube-api-access-6t5zz\") pod \"b8f2b635-0daa-41bc-854f-5dc46247ae47\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") "
Sep 11 19:46:58.745944 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.745789    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-envoy\") pod \"b8f2b635-0daa-41bc-854f-5dc46247ae47\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") "
Sep 11 19:46:58.745944 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.745845    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-token\") pod \"b8f2b635-0daa-41bc-854f-5dc46247ae47\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") "
Sep 11 19:46:58.745944 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.745865    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f2b635-0daa-41bc-854f-5dc46247ae47-istiod-ca-cert\") pod \"b8f2b635-0daa-41bc-854f-5dc46247ae47\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") "
Sep 11 19:46:58.745944 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.745900    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-data\") pod \"b8f2b635-0daa-41bc-854f-5dc46247ae47\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") "
Sep 11 19:46:58.745944 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.745937    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-podinfo\") pod \"b8f2b635-0daa-41bc-854f-5dc46247ae47\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") "
Sep 11 19:46:58.746230 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.745972    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-workload-certs\") pod \"b8f2b635-0daa-41bc-854f-5dc46247ae47\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") "
Sep 11 19:46:58.746230 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.746014    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-credential-socket\") pod \"b8f2b635-0daa-41bc-854f-5dc46247ae47\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") "
Sep 11 19:46:58.746230 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.746041    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-workload-socket\") pod \"b8f2b635-0daa-41bc-854f-5dc46247ae47\" (UID: \"b8f2b635-0daa-41bc-854f-5dc46247ae47\") "
Sep 11 19:46:58.746393 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.746222    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-data" (OuterVolumeSpecName: "istio-data") pod "b8f2b635-0daa-41bc-854f-5dc46247ae47" (UID: "b8f2b635-0daa-41bc-854f-5dc46247ae47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:46:58.746393 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.746319    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8f2b635-0daa-41bc-854f-5dc46247ae47-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b8f2b635-0daa-41bc-854f-5dc46247ae47" (UID: "b8f2b635-0daa-41bc-854f-5dc46247ae47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:46:58.746500 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.746417    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b8f2b635-0daa-41bc-854f-5dc46247ae47" (UID: "b8f2b635-0daa-41bc-854f-5dc46247ae47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:46:58.746500 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.746431    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b8f2b635-0daa-41bc-854f-5dc46247ae47" (UID: "b8f2b635-0daa-41bc-854f-5dc46247ae47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:46:58.746622 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.746607    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b8f2b635-0daa-41bc-854f-5dc46247ae47" (UID: "b8f2b635-0daa-41bc-854f-5dc46247ae47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:46:58.746778 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.746744    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8f2b635-0daa-41bc-854f-5dc46247ae47-istiod-ca-cert\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:58.746864 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.746785    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-data\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:58.746864 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.746800    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-workload-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:58.746864 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.746814    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-workload-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:58.748163 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.748135    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b8f2b635-0daa-41bc-854f-5dc46247ae47" (UID: "b8f2b635-0daa-41bc-854f-5dc46247ae47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:46:58.748285 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.748265    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f2b635-0daa-41bc-854f-5dc46247ae47-kube-api-access-6t5zz" (OuterVolumeSpecName: "kube-api-access-6t5zz") pod "b8f2b635-0daa-41bc-854f-5dc46247ae47" (UID: "b8f2b635-0daa-41bc-854f-5dc46247ae47"). InnerVolumeSpecName "kube-api-access-6t5zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:46:58.748343 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.748300    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-token" (OuterVolumeSpecName: "istio-token") pod "b8f2b635-0daa-41bc-854f-5dc46247ae47" (UID: "b8f2b635-0daa-41bc-854f-5dc46247ae47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:46:58.748419 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.748403    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b8f2b635-0daa-41bc-854f-5dc46247ae47" (UID: "b8f2b635-0daa-41bc-854f-5dc46247ae47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:46:58.847923 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.847888    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-podinfo\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:58.847923 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.847918    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-credential-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:58.847923 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.847927    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t5zz\" (UniqueName: \"kubernetes.io/projected/b8f2b635-0daa-41bc-854f-5dc46247ae47-kube-api-access-6t5zz\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:58.848173 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.847939    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-envoy\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:58.848173 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.847947    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8f2b635-0daa-41bc-854f-5dc46247ae47-istio-token\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:46:58.947632 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.947578    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"]
Sep 11 19:46:58.948321 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.948246    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8f2b635-0daa-41bc-854f-5dc46247ae47" containerName="istio-proxy"
Sep 11 19:46:58.948321 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.948275    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8f2b635-0daa-41bc-854f-5dc46247ae47" containerName="istio-proxy"
Sep 11 19:46:58.948321 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.948309    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8f2b635-0daa-41bc-854f-5dc46247ae47" containerName="istio-proxy"
Sep 11 19:46:58.948321 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.948318    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8f2b635-0daa-41bc-854f-5dc46247ae47" containerName="istio-proxy"
Sep 11 19:46:58.948686 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.948466    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8f2b635-0daa-41bc-854f-5dc46247ae47" containerName="istio-proxy"
Sep 11 19:46:58.948686 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.948475    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8f2b635-0daa-41bc-854f-5dc46247ae47" containerName="istio-proxy"
Sep 11 19:46:58.951962 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.951941    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"
Sep 11 19:46:58.954252 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.954232    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-0d707fde-serving-cert\""
Sep 11 19:46:58.964188 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:58.964161    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"]
Sep 11 19:46:59.050527 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.050424    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-metrics-certs\") pod \"maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz\" (UID: \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"
Sep 11 19:46:59.050527 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.050500    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d7zk\" (UniqueName: \"kubernetes.io/projected/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-kube-api-access-6d7zk\") pod \"maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz\" (UID: \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"
Sep 11 19:46:59.050771 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.050539    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-openshift-service-ca\") pod \"maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz\" (UID: \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"
Sep 11 19:46:59.050771 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.050631    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-maas-api-tls\") pod \"maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz\" (UID: \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"
Sep 11 19:46:59.151484 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.151442    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-metrics-certs\") pod \"maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz\" (UID: \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"
Sep 11 19:46:59.151747 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.151498    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d7zk\" (UniqueName: \"kubernetes.io/projected/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-kube-api-access-6d7zk\") pod \"maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz\" (UID: \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"
Sep 11 19:46:59.151747 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.151526    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-openshift-service-ca\") pod \"maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz\" (UID: \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"
Sep 11 19:46:59.151747 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.151548    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-maas-api-tls\") pod \"maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz\" (UID: \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"
Sep 11 19:46:59.151747 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:46:59.151681    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-serving-cert: secret "maas-api-e2e-derive-0d707fde-serving-cert" not found
Sep 11 19:46:59.151981 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:46:59.151789    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-maas-api-tls podName:6f7639e1-1c93-432c-bd37-a5c28b2b15a7 nodeName:}" failed. No retries permitted until 2026-09-11 19:46:59.651769388 +0000 UTC m=+1236.268938871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-maas-api-tls") pod "maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz" (UID: "6f7639e1-1c93-432c-bd37-a5c28b2b15a7") : secret "maas-api-e2e-derive-0d707fde-serving-cert" not found
Sep 11 19:46:59.152322 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.152301    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-openshift-service-ca\") pod \"maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz\" (UID: \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"
Sep 11 19:46:59.154009 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.153985    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-metrics-certs\") pod \"maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz\" (UID: \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"
Sep 11 19:46:59.162760 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.162740    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d7zk\" (UniqueName: \"kubernetes.io/projected/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-kube-api-access-6d7zk\") pod \"maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz\" (UID: \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"
Sep 11 19:46:59.290703 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.290657    2558 generic.go:356] "Generic (PLEG): container finished" podID="b8f2b635-0daa-41bc-854f-5dc46247ae47" containerID="c6f81862bb1043331bbc0ec710d051c40e836d4dd5f6dd95c02bc142990da829" exitCode=0
Sep 11 19:46:59.290917 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.290748    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg" event={"ID":"b8f2b635-0daa-41bc-854f-5dc46247ae47","Type":"ContainerDied","Data":"c6f81862bb1043331bbc0ec710d051c40e836d4dd5f6dd95c02bc142990da829"}
Sep 11 19:46:59.290917 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.290834    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg" event={"ID":"b8f2b635-0daa-41bc-854f-5dc46247ae47","Type":"ContainerDied","Data":"ad63caa6f108b3c366731bf79a394051d754becac1aa300da1a3e6496202b93c"}
Sep 11 19:46:59.290917 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.290759    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"
Sep 11 19:46:59.290917 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.290857    2558 scope.go:117] "RemoveContainer" containerID="c6f81862bb1043331bbc0ec710d051c40e836d4dd5f6dd95c02bc142990da829"
Sep 11 19:46:59.300303 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.300285    2558 scope.go:117] "RemoveContainer" containerID="bf76ab94a4f88c50e13969a8d83f57a6225b681526b0dd33fec6f72518826106"
Sep 11 19:46:59.308850 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.308828    2558 scope.go:117] "RemoveContainer" containerID="c6f81862bb1043331bbc0ec710d051c40e836d4dd5f6dd95c02bc142990da829"
Sep 11 19:46:59.309143 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:46:59.309119    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6f81862bb1043331bbc0ec710d051c40e836d4dd5f6dd95c02bc142990da829\": container with ID starting with c6f81862bb1043331bbc0ec710d051c40e836d4dd5f6dd95c02bc142990da829 not found: ID does not exist" containerID="c6f81862bb1043331bbc0ec710d051c40e836d4dd5f6dd95c02bc142990da829"
Sep 11 19:46:59.309251 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.309150    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6f81862bb1043331bbc0ec710d051c40e836d4dd5f6dd95c02bc142990da829"} err="failed to get container status \"c6f81862bb1043331bbc0ec710d051c40e836d4dd5f6dd95c02bc142990da829\": rpc error: code = NotFound desc = could not find container \"c6f81862bb1043331bbc0ec710d051c40e836d4dd5f6dd95c02bc142990da829\": container with ID starting with c6f81862bb1043331bbc0ec710d051c40e836d4dd5f6dd95c02bc142990da829 not found: ID does not exist"
Sep 11 19:46:59.309251 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.309170    2558 scope.go:117] "RemoveContainer" containerID="bf76ab94a4f88c50e13969a8d83f57a6225b681526b0dd33fec6f72518826106"
Sep 11 19:46:59.309434 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:46:59.309417    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf76ab94a4f88c50e13969a8d83f57a6225b681526b0dd33fec6f72518826106\": container with ID starting with bf76ab94a4f88c50e13969a8d83f57a6225b681526b0dd33fec6f72518826106 not found: ID does not exist" containerID="bf76ab94a4f88c50e13969a8d83f57a6225b681526b0dd33fec6f72518826106"
Sep 11 19:46:59.309486 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.309440    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf76ab94a4f88c50e13969a8d83f57a6225b681526b0dd33fec6f72518826106"} err="failed to get container status \"bf76ab94a4f88c50e13969a8d83f57a6225b681526b0dd33fec6f72518826106\": rpc error: code = NotFound desc = could not find container \"bf76ab94a4f88c50e13969a8d83f57a6225b681526b0dd33fec6f72518826106\": container with ID starting with bf76ab94a4f88c50e13969a8d83f57a6225b681526b0dd33fec6f72518826106 not found: ID does not exist"
Sep 11 19:46:59.316231 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.316208    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"]
Sep 11 19:46:59.326938 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.326912    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8901a783-openshift-default-b5f6b549d-x4fgg"]
Sep 11 19:46:59.657287 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.657198    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-maas-api-tls\") pod \"maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz\" (UID: \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"
Sep 11 19:46:59.659652 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.659623    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-maas-api-tls\") pod \"maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz\" (UID: \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"
Sep 11 19:46:59.814666 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.814621    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8f2b635-0daa-41bc-854f-5dc46247ae47" path="/var/lib/kubelet/pods/b8f2b635-0daa-41bc-854f-5dc46247ae47/volumes"
Sep 11 19:46:59.866103 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:46:59.866067    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"
Sep 11 19:47:00.161147 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:00.161118    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"]
Sep 11 19:47:00.162873 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:47:00.162847    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7639e1_1c93_432c_bd37_a5c28b2b15a7.slice/crio-14d30c1e27c481ce63a9d5da47aa85fce72232aa711c9b18abdbffa8493caf87 WatchSource:0}: Error finding container 14d30c1e27c481ce63a9d5da47aa85fce72232aa711c9b18abdbffa8493caf87: Status 404 returned error can't find the container with id 14d30c1e27c481ce63a9d5da47aa85fce72232aa711c9b18abdbffa8493caf87
Sep 11 19:47:00.297758 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:00.297658    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz" event={"ID":"6f7639e1-1c93-432c-bd37-a5c28b2b15a7","Type":"ContainerStarted","Data":"14d30c1e27c481ce63a9d5da47aa85fce72232aa711c9b18abdbffa8493caf87"}
Sep 11 19:47:01.305972 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:01.305937    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz" event={"ID":"6f7639e1-1c93-432c-bd37-a5c28b2b15a7","Type":"ContainerStarted","Data":"8c7e7649c8315d161c0f983f900e765129ddca3312300348a848712d24d10e61"}
Sep 11 19:47:01.306391 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:01.306005    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"
Sep 11 19:47:01.345014 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:01.344950    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz" podStartSLOduration=2.987619806 podStartE2EDuration="3.344932169s" totalImagesPullingTime="357.312363ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:47:01.344348366 +0000 UTC m=+1237.961517857" watchObservedRunningTime="2026-09-11 19:47:01.344932169 +0000 UTC m=+1237.962101659"
Sep 11 19:47:07.317651 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:07.317606    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"
Sep 11 19:47:18.402175 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:18.402099    2558 generic.go:356] "Generic (PLEG): container finished" podID="d0113daa-a4a1-4bb1-9ad5-835ac98f201d" containerID="3a450071d6b64d0583c7945461ceb53256a1bc60244772d1cd69935e5496bc2b" exitCode=0
Sep 11 19:47:18.402590 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:18.402170    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x" event={"ID":"d0113daa-a4a1-4bb1-9ad5-835ac98f201d","Type":"ContainerDied","Data":"3a450071d6b64d0583c7945461ceb53256a1bc60244772d1cd69935e5496bc2b"}
Sep 11 19:47:18.402652 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:18.402606    2558 scope.go:117] "RemoveContainer" containerID="3a450071d6b64d0583c7945461ceb53256a1bc60244772d1cd69935e5496bc2b"
Sep 11 19:47:18.403818 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:18.403783    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:47:18.403901 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:18.403856    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:47:18.403901 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:18.403888    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:47:19.409609 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:19.409570    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x" event={"ID":"d0113daa-a4a1-4bb1-9ad5-835ac98f201d","Type":"ContainerStarted","Data":"349e8a0cd8ac8ad94589abcd94b387e81495482792f81df76463471ee10db13d"}
Sep 11 19:47:19.493167 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:19.493127    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:47:19.493353 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:19.493321    2558 patch_prober.go:28] interesting pod/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 19:47:19.493416 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:19.493397    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x" podUID="d0113daa-a4a1-4bb1-9ad5-835ac98f201d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 19:47:20.492582 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:20.492545    2558 patch_prober.go:28] interesting pod/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 19:47:20.492999 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:20.492614    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x" podUID="d0113daa-a4a1-4bb1-9ad5-835ac98f201d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 19:47:21.492482 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:21.492444    2558 patch_prober.go:28] interesting pod/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 19:47:21.492662 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:21.492500    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x" podUID="d0113daa-a4a1-4bb1-9ad5-835ac98f201d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 19:47:22.493367 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:22.493329    2558 patch_prober.go:28] interesting pod/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 19:47:22.493789 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:22.493399    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x" podUID="d0113daa-a4a1-4bb1-9ad5-835ac98f201d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 19:47:23.493014 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:23.492976    2558 patch_prober.go:28] interesting pod/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 19:47:23.493321 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:23.493044    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x" podUID="d0113daa-a4a1-4bb1-9ad5-835ac98f201d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 19:47:24.492536 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:24.492493    2558 patch_prober.go:28] interesting pod/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body=
Sep 11 19:47:24.492942 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:24.492565    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x" podUID="d0113daa-a4a1-4bb1-9ad5-835ac98f201d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused"
Sep 11 19:47:25.497264 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:25.497231    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:47:25.497669 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:25.497645    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:47:25.498463 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:25.498437    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:47:29.042514 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.041473    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"]
Sep 11 19:47:29.042514 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.041830    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz" podUID="6f7639e1-1c93-432c-bd37-a5c28b2b15a7" containerName="maas-api" containerID="cri-o://8c7e7649c8315d161c0f983f900e765129ddca3312300348a848712d24d10e61" gracePeriod=30
Sep 11 19:47:29.315491 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.315456    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"
Sep 11 19:47:29.379595 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.379559    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d7zk\" (UniqueName: \"kubernetes.io/projected/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-kube-api-access-6d7zk\") pod \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\" (UID: \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\") "
Sep 11 19:47:29.379795 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.379607    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-openshift-service-ca\") pod \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\" (UID: \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\") "
Sep 11 19:47:29.379795 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.379654    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-maas-api-tls\") pod \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\" (UID: \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\") "
Sep 11 19:47:29.379927 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.379754    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-metrics-certs\") pod \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\" (UID: \"6f7639e1-1c93-432c-bd37-a5c28b2b15a7\") "
Sep 11 19:47:29.380102 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.380068    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6f7639e1-1c93-432c-bd37-a5c28b2b15a7" (UID: "6f7639e1-1c93-432c-bd37-a5c28b2b15a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:47:29.380243 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.380220    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-openshift-service-ca\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:47:29.382001 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.381969    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6f7639e1-1c93-432c-bd37-a5c28b2b15a7" (UID: "6f7639e1-1c93-432c-bd37-a5c28b2b15a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:47:29.382001 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.381984    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-kube-api-access-6d7zk" (OuterVolumeSpecName: "kube-api-access-6d7zk") pod "6f7639e1-1c93-432c-bd37-a5c28b2b15a7" (UID: "6f7639e1-1c93-432c-bd37-a5c28b2b15a7"). InnerVolumeSpecName "kube-api-access-6d7zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:47:29.382150 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.381989    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6f7639e1-1c93-432c-bd37-a5c28b2b15a7" (UID: "6f7639e1-1c93-432c-bd37-a5c28b2b15a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:47:29.466658 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.466615    2558 generic.go:356] "Generic (PLEG): container finished" podID="6f7639e1-1c93-432c-bd37-a5c28b2b15a7" containerID="8c7e7649c8315d161c0f983f900e765129ddca3312300348a848712d24d10e61" exitCode=0
Sep 11 19:47:29.466854 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.466664    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"
Sep 11 19:47:29.466854 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.466697    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz" event={"ID":"6f7639e1-1c93-432c-bd37-a5c28b2b15a7","Type":"ContainerDied","Data":"8c7e7649c8315d161c0f983f900e765129ddca3312300348a848712d24d10e61"}
Sep 11 19:47:29.466854 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.466785    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz" event={"ID":"6f7639e1-1c93-432c-bd37-a5c28b2b15a7","Type":"ContainerDied","Data":"14d30c1e27c481ce63a9d5da47aa85fce72232aa711c9b18abdbffa8493caf87"}
Sep 11 19:47:29.466854 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.466813    2558 scope.go:117] "RemoveContainer" containerID="8c7e7649c8315d161c0f983f900e765129ddca3312300348a848712d24d10e61"
Sep 11 19:47:29.477329 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.477305    2558 scope.go:117] "RemoveContainer" containerID="8c7e7649c8315d161c0f983f900e765129ddca3312300348a848712d24d10e61"
Sep 11 19:47:29.477755 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:47:29.477688    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c7e7649c8315d161c0f983f900e765129ddca3312300348a848712d24d10e61\": container with ID starting with 8c7e7649c8315d161c0f983f900e765129ddca3312300348a848712d24d10e61 not found: ID does not exist" containerID="8c7e7649c8315d161c0f983f900e765129ddca3312300348a848712d24d10e61"
Sep 11 19:47:29.477923 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.477745    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7e7649c8315d161c0f983f900e765129ddca3312300348a848712d24d10e61"} err="failed to get container status \"8c7e7649c8315d161c0f983f900e765129ddca3312300348a848712d24d10e61\": rpc error: code = NotFound desc = could not find container \"8c7e7649c8315d161c0f983f900e765129ddca3312300348a848712d24d10e61\": container with ID starting with 8c7e7649c8315d161c0f983f900e765129ddca3312300348a848712d24d10e61 not found: ID does not exist"
Sep 11 19:47:29.481965 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.481872    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d7zk\" (UniqueName: \"kubernetes.io/projected/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-kube-api-access-6d7zk\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:47:29.481965 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.481926    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-maas-api-tls\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:47:29.481965 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.481942    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6f7639e1-1c93-432c-bd37-a5c28b2b15a7-metrics-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:47:29.494913 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.494846    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"]
Sep 11 19:47:29.500322 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.500284    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-0d707fde-548f454bb7-lzdcz"]
Sep 11 19:47:29.620128 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.620028    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"]
Sep 11 19:47:29.620312 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.620280    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x" podUID="d0113daa-a4a1-4bb1-9ad5-835ac98f201d" containerName="istio-proxy" containerID="cri-o://349e8a0cd8ac8ad94589abcd94b387e81495482792f81df76463471ee10db13d" gracePeriod=30
Sep 11 19:47:29.814975 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:29.814944    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f7639e1-1c93-432c-bd37-a5c28b2b15a7" path="/var/lib/kubelet/pods/6f7639e1-1c93-432c-bd37-a5c28b2b15a7/volumes"
Sep 11 19:47:34.891547 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:34.891478    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:47:35.043370 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.043326    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istiod-ca-cert\") pod \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") "
Sep 11 19:47:35.043562 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.043412    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4msb\" (UniqueName: \"kubernetes.io/projected/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-kube-api-access-k4msb\") pod \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") "
Sep 11 19:47:35.043562 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.043446    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-token\") pod \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") "
Sep 11 19:47:35.043562 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.043485    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-data\") pod \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") "
Sep 11 19:47:35.043562 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.043516    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-envoy\") pod \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") "
Sep 11 19:47:35.043562 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.043542    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-workload-socket\") pod \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") "
Sep 11 19:47:35.043859 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.043602    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-credential-socket\") pod \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") "
Sep 11 19:47:35.043859 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.043631    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-podinfo\") pod \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") "
Sep 11 19:47:35.043859 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.043658    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-workload-certs\") pod \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\" (UID: \"d0113daa-a4a1-4bb1-9ad5-835ac98f201d\") "
Sep 11 19:47:35.043859 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.043786    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d0113daa-a4a1-4bb1-9ad5-835ac98f201d" (UID: "d0113daa-a4a1-4bb1-9ad5-835ac98f201d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:47:35.044062 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.043962    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-data" (OuterVolumeSpecName: "istio-data") pod "d0113daa-a4a1-4bb1-9ad5-835ac98f201d" (UID: "d0113daa-a4a1-4bb1-9ad5-835ac98f201d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:47:35.044062 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.044024    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d0113daa-a4a1-4bb1-9ad5-835ac98f201d" (UID: "d0113daa-a4a1-4bb1-9ad5-835ac98f201d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:47:35.044062 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.044035    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d0113daa-a4a1-4bb1-9ad5-835ac98f201d" (UID: "d0113daa-a4a1-4bb1-9ad5-835ac98f201d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:47:35.044188 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.044147    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istiod-ca-cert\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:47:35.044188 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.044168    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-data\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:47:35.044188 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.044168    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d0113daa-a4a1-4bb1-9ad5-835ac98f201d" (UID: "d0113daa-a4a1-4bb1-9ad5-835ac98f201d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:47:35.044188 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.044181    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-workload-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:47:35.044341 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.044193    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-credential-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:47:35.045881 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.045852    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-token" (OuterVolumeSpecName: "istio-token") pod "d0113daa-a4a1-4bb1-9ad5-835ac98f201d" (UID: "d0113daa-a4a1-4bb1-9ad5-835ac98f201d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:47:35.046008 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.045969    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-kube-api-access-k4msb" (OuterVolumeSpecName: "kube-api-access-k4msb") pod "d0113daa-a4a1-4bb1-9ad5-835ac98f201d" (UID: "d0113daa-a4a1-4bb1-9ad5-835ac98f201d"). InnerVolumeSpecName "kube-api-access-k4msb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:47:35.046227 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.046204    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d0113daa-a4a1-4bb1-9ad5-835ac98f201d" (UID: "d0113daa-a4a1-4bb1-9ad5-835ac98f201d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:47:35.046274 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.046219    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d0113daa-a4a1-4bb1-9ad5-835ac98f201d" (UID: "d0113daa-a4a1-4bb1-9ad5-835ac98f201d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:47:35.145679 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.145578    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4msb\" (UniqueName: \"kubernetes.io/projected/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-kube-api-access-k4msb\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:47:35.145679 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.145617    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-token\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:47:35.145679 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.145629    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-envoy\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:47:35.145679 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.145636    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-istio-podinfo\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:47:35.145679 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.145644    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0113daa-a4a1-4bb1-9ad5-835ac98f201d-workload-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:47:35.506784 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.506745    2558 generic.go:356] "Generic (PLEG): container finished" podID="d0113daa-a4a1-4bb1-9ad5-835ac98f201d" containerID="349e8a0cd8ac8ad94589abcd94b387e81495482792f81df76463471ee10db13d" exitCode=0
Sep 11 19:47:35.506944 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.506809    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"
Sep 11 19:47:35.506944 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.506837    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x" event={"ID":"d0113daa-a4a1-4bb1-9ad5-835ac98f201d","Type":"ContainerDied","Data":"349e8a0cd8ac8ad94589abcd94b387e81495482792f81df76463471ee10db13d"}
Sep 11 19:47:35.506944 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.506895    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x" event={"ID":"d0113daa-a4a1-4bb1-9ad5-835ac98f201d","Type":"ContainerDied","Data":"73a49f9c63c199274f40cc328b74b2b1e213dce45c0033be099a49251edc3679"}
Sep 11 19:47:35.506944 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.506917    2558 scope.go:117] "RemoveContainer" containerID="349e8a0cd8ac8ad94589abcd94b387e81495482792f81df76463471ee10db13d"
Sep 11 19:47:35.517286 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.517266    2558 scope.go:117] "RemoveContainer" containerID="3a450071d6b64d0583c7945461ceb53256a1bc60244772d1cd69935e5496bc2b"
Sep 11 19:47:35.526512 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.526484    2558 scope.go:117] "RemoveContainer" containerID="349e8a0cd8ac8ad94589abcd94b387e81495482792f81df76463471ee10db13d"
Sep 11 19:47:35.526886 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:47:35.526865    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"349e8a0cd8ac8ad94589abcd94b387e81495482792f81df76463471ee10db13d\": container with ID starting with 349e8a0cd8ac8ad94589abcd94b387e81495482792f81df76463471ee10db13d not found: ID does not exist" containerID="349e8a0cd8ac8ad94589abcd94b387e81495482792f81df76463471ee10db13d"
Sep 11 19:47:35.526934 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.526896    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"349e8a0cd8ac8ad94589abcd94b387e81495482792f81df76463471ee10db13d"} err="failed to get container status \"349e8a0cd8ac8ad94589abcd94b387e81495482792f81df76463471ee10db13d\": rpc error: code = NotFound desc = could not find container \"349e8a0cd8ac8ad94589abcd94b387e81495482792f81df76463471ee10db13d\": container with ID starting with 349e8a0cd8ac8ad94589abcd94b387e81495482792f81df76463471ee10db13d not found: ID does not exist"
Sep 11 19:47:35.526934 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.526917    2558 scope.go:117] "RemoveContainer" containerID="3a450071d6b64d0583c7945461ceb53256a1bc60244772d1cd69935e5496bc2b"
Sep 11 19:47:35.527207 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:47:35.527182    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a450071d6b64d0583c7945461ceb53256a1bc60244772d1cd69935e5496bc2b\": container with ID starting with 3a450071d6b64d0583c7945461ceb53256a1bc60244772d1cd69935e5496bc2b not found: ID does not exist" containerID="3a450071d6b64d0583c7945461ceb53256a1bc60244772d1cd69935e5496bc2b"
Sep 11 19:47:35.527315 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.527208    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a450071d6b64d0583c7945461ceb53256a1bc60244772d1cd69935e5496bc2b"} err="failed to get container status \"3a450071d6b64d0583c7945461ceb53256a1bc60244772d1cd69935e5496bc2b\": rpc error: code = NotFound desc = could not find container \"3a450071d6b64d0583c7945461ceb53256a1bc60244772d1cd69935e5496bc2b\": container with ID starting with 3a450071d6b64d0583c7945461ceb53256a1bc60244772d1cd69935e5496bc2b not found: ID does not exist"
Sep 11 19:47:35.536050 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.536019    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"]
Sep 11 19:47:35.543608 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.543574    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-0d707fde-openshift-default-75fbb996d8-pj65x"]
Sep 11 19:47:35.815557 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:35.815470    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0113daa-a4a1-4bb1-9ad5-835ac98f201d" path="/var/lib/kubelet/pods/d0113daa-a4a1-4bb1-9ad5-835ac98f201d/volumes"
Sep 11 19:47:36.073207 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.073121    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"]
Sep 11 19:47:36.073703 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.073685    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f7639e1-1c93-432c-bd37-a5c28b2b15a7" containerName="maas-api"
Sep 11 19:47:36.073777 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.073718    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f7639e1-1c93-432c-bd37-a5c28b2b15a7" containerName="maas-api"
Sep 11 19:47:36.073777 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.073731    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0113daa-a4a1-4bb1-9ad5-835ac98f201d" containerName="istio-proxy"
Sep 11 19:47:36.073777 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.073737    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0113daa-a4a1-4bb1-9ad5-835ac98f201d" containerName="istio-proxy"
Sep 11 19:47:36.073777 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.073743    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0113daa-a4a1-4bb1-9ad5-835ac98f201d" containerName="istio-proxy"
Sep 11 19:47:36.073777 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.073750    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0113daa-a4a1-4bb1-9ad5-835ac98f201d" containerName="istio-proxy"
Sep 11 19:47:36.073937 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.073820    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0113daa-a4a1-4bb1-9ad5-835ac98f201d" containerName="istio-proxy"
Sep 11 19:47:36.073937 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.073828    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f7639e1-1c93-432c-bd37-a5c28b2b15a7" containerName="maas-api"
Sep 11 19:47:36.078413 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.078391    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:36.080955 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.080929    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ec80ad\"/\"openshift-service-ca.crt\""
Sep 11 19:47:36.081612 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.081593    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ec80ad\"/\"kube-root-ca.crt\""
Sep 11 19:47:36.081612 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.081604    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ec80ad\"/\"test-model-ec80ad-kserve-self-signed-certs\""
Sep 11 19:47:36.081784 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.081660    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ec80ad\"/\"default-dockercfg-sfjcx\""
Sep 11 19:47:36.091684 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.090476    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"]
Sep 11 19:47:36.256974 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.256928    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-tmp-dir\") pod \"test-model-ec80ad-kserve-79dbb4d787-tlkjf\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") " pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:36.257183 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.257023    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-home\") pod \"test-model-ec80ad-kserve-79dbb4d787-tlkjf\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") " pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:36.257183 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.257048    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-dshm\") pod \"test-model-ec80ad-kserve-79dbb4d787-tlkjf\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") " pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:36.257183 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.257091    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a988a97f-8992-4195-94f0-4c261e839bfa-tls-certs\") pod \"test-model-ec80ad-kserve-79dbb4d787-tlkjf\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") " pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:36.257347 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.257208    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-model-cache\") pod \"test-model-ec80ad-kserve-79dbb4d787-tlkjf\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") " pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:36.257347 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.257261    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lwsc\" (UniqueName: \"kubernetes.io/projected/a988a97f-8992-4195-94f0-4c261e839bfa-kube-api-access-8lwsc\") pod \"test-model-ec80ad-kserve-79dbb4d787-tlkjf\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") " pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:36.358353 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.358251    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-tmp-dir\") pod \"test-model-ec80ad-kserve-79dbb4d787-tlkjf\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") " pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:36.358353 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.358327    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-home\") pod \"test-model-ec80ad-kserve-79dbb4d787-tlkjf\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") " pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:36.358353 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.358345    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-dshm\") pod \"test-model-ec80ad-kserve-79dbb4d787-tlkjf\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") " pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:36.358654 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.358376    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a988a97f-8992-4195-94f0-4c261e839bfa-tls-certs\") pod \"test-model-ec80ad-kserve-79dbb4d787-tlkjf\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") " pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:36.358654 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.358411    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-model-cache\") pod \"test-model-ec80ad-kserve-79dbb4d787-tlkjf\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") " pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:36.358654 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.358435    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lwsc\" (UniqueName: \"kubernetes.io/projected/a988a97f-8992-4195-94f0-4c261e839bfa-kube-api-access-8lwsc\") pod \"test-model-ec80ad-kserve-79dbb4d787-tlkjf\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") " pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:36.358862 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.358807    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-tmp-dir\") pod \"test-model-ec80ad-kserve-79dbb4d787-tlkjf\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") " pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:36.358862 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.358818    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-home\") pod \"test-model-ec80ad-kserve-79dbb4d787-tlkjf\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") " pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:36.358992 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.358896    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-model-cache\") pod \"test-model-ec80ad-kserve-79dbb4d787-tlkjf\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") " pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:36.360695 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.360667    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-dshm\") pod \"test-model-ec80ad-kserve-79dbb4d787-tlkjf\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") " pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:36.361120 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.361098    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a988a97f-8992-4195-94f0-4c261e839bfa-tls-certs\") pod \"test-model-ec80ad-kserve-79dbb4d787-tlkjf\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") " pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:36.367792 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.367767    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lwsc\" (UniqueName: \"kubernetes.io/projected/a988a97f-8992-4195-94f0-4c261e839bfa-kube-api-access-8lwsc\") pod \"test-model-ec80ad-kserve-79dbb4d787-tlkjf\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") " pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:36.392723 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.392675    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:36.683983 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:36.683937    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"]
Sep 11 19:47:36.689906 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:47:36.689872    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda988a97f_8992_4195_94f0_4c261e839bfa.slice/crio-a6c3a685dede09d357b02ea343b85fec2303a79603d42bcdfcdf6637c7bf8269 WatchSource:0}: Error finding container a6c3a685dede09d357b02ea343b85fec2303a79603d42bcdfcdf6637c7bf8269: Status 404 returned error can't find the container with id a6c3a685dede09d357b02ea343b85fec2303a79603d42bcdfcdf6637c7bf8269
Sep 11 19:47:37.519998 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:37.519958    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf" event={"ID":"a988a97f-8992-4195-94f0-4c261e839bfa","Type":"ContainerStarted","Data":"f933b769a9a6182b08318e866e67449a6c2c7dcb9e1dc1f85ea5b8bfe6d96fd6"}
Sep 11 19:47:37.520440 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:37.520010    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf" event={"ID":"a988a97f-8992-4195-94f0-4c261e839bfa","Type":"ContainerStarted","Data":"a6c3a685dede09d357b02ea343b85fec2303a79603d42bcdfcdf6637c7bf8269"}
Sep 11 19:47:37.540673 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:37.540621    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf" podStartSLOduration=1.540603891 podStartE2EDuration="1.540603891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:47:37.539294462 +0000 UTC m=+1274.156463952" watchObservedRunningTime="2026-09-11 19:47:37.540603891 +0000 UTC m=+1274.157773383"
Sep 11 19:47:46.393380 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:46.393338    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:46.393945 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:46.393523    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:46.406703 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:46.406681    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:47:46.580131 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:47:46.580100    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:48:43.096453 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.096413    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"]
Sep 11 19:48:43.096985 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.096971    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0113daa-a4a1-4bb1-9ad5-835ac98f201d" containerName="istio-proxy"
Sep 11 19:48:43.101620 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.101601    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.104044 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.104019    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-a29559-openshift-default-dockercfg-658ds\""
Sep 11 19:48:43.111404 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.111378    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"]
Sep 11 19:48:43.146568 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.146513    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-credential-socket\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.146568 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.146585    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-token\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.146847 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.146617    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istiod-ca-cert\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.146847 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.146637    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-envoy\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.146847 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.146768    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-workload-socket\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.146847 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.146795    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp9qs\" (UniqueName: \"kubernetes.io/projected/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-kube-api-access-bp9qs\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.146847 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.146814    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-podinfo\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.147018 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.146871    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-data\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.147018 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.146894    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-workload-certs\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.247460 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.247418    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-workload-socket\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.247637 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.247470    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp9qs\" (UniqueName: \"kubernetes.io/projected/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-kube-api-access-bp9qs\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.247637 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.247496    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-podinfo\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.247637 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.247523    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-data\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.247637 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.247554    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-workload-certs\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.247637 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.247586    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-credential-socket\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.247939 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.247637    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-token\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.247939 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.247670    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istiod-ca-cert\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.247939 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.247701    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-envoy\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.248095 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.248070    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-workload-certs\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.248176 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.248141    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-credential-socket\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.248176 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.248166    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-data\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.248291 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.248230    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-workload-socket\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.248508 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.248488    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istiod-ca-cert\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.249887 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.249866    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-envoy\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.250134 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.250116    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-podinfo\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.255431 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.255405    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-token\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.255545 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.255529    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp9qs\" (UniqueName: \"kubernetes.io/projected/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-kube-api-access-bp9qs\") pod \"e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") " pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.413999 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.413967    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:43.709161 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.709135    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"]
Sep 11 19:48:43.711178 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:48:43.711147    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21785e5f_a2aa_4779_91a7_2f1cf1d4688b.slice/crio-0670ab8f72509055c7f2affd213e4a5c6840d6203e71d77ca2f0795bce1fae7c WatchSource:0}: Error finding container 0670ab8f72509055c7f2affd213e4a5c6840d6203e71d77ca2f0795bce1fae7c: Status 404 returned error can't find the container with id 0670ab8f72509055c7f2affd213e4a5c6840d6203e71d77ca2f0795bce1fae7c
Sep 11 19:48:43.713504 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.713469    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:48:43.713602 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.713553    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:48:43.713667 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.713599    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:48:43.873881 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.873842    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx" event={"ID":"21785e5f-a2aa-4779-91a7-2f1cf1d4688b","Type":"ContainerStarted","Data":"3c1ad703615d3eef08ea09b80f1fa5297efb15c408a2b70cb1bf9ea59a567f7c"}
Sep 11 19:48:43.874044 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.873907    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx" event={"ID":"21785e5f-a2aa-4779-91a7-2f1cf1d4688b","Type":"ContainerStarted","Data":"0670ab8f72509055c7f2affd213e4a5c6840d6203e71d77ca2f0795bce1fae7c"}
Sep 11 19:48:43.895162 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:43.895094    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx" podStartSLOduration=0.895078154 podStartE2EDuration="895.078154ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:48:43.893974446 +0000 UTC m=+1340.511143974" watchObservedRunningTime="2026-09-11 19:48:43.895078154 +0000 UTC m=+1340.512247645"
Sep 11 19:48:44.414344 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:44.414313    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:44.419676 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:44.419647    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:44.879504 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:44.879426    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:48:44.880415 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:48:44.880396    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:49:05.676258 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.676180    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"]
Sep 11 19:49:05.679381 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.679361    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.681821 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.681799    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-02df91be-openshift-default-dockercfg-rl5vg\""
Sep 11 19:49:05.695339 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.695314    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"]
Sep 11 19:49:05.792409 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.792346    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znscs\" (UniqueName: \"kubernetes.io/projected/701ffef5-2564-403a-be8a-44329cf36cb1-kube-api-access-znscs\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.792612 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.792427    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701ffef5-2564-403a-be8a-44329cf36cb1-istio-podinfo\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.792612 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.792465    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701ffef5-2564-403a-be8a-44329cf36cb1-istiod-ca-cert\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.792612 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.792512    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-workload-socket\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.792612 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.792553    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-workload-certs\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.792854 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.792626    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-credential-socket\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.792854 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.792762    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-istio-data\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.792854 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.792785    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701ffef5-2564-403a-be8a-44329cf36cb1-istio-token\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.792854 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.792805    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-istio-envoy\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.894249 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.894210    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znscs\" (UniqueName: \"kubernetes.io/projected/701ffef5-2564-403a-be8a-44329cf36cb1-kube-api-access-znscs\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.894638 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.894264    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701ffef5-2564-403a-be8a-44329cf36cb1-istio-podinfo\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.894638 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.894314    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701ffef5-2564-403a-be8a-44329cf36cb1-istiod-ca-cert\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.894638 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.894397    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-workload-socket\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.894862 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.894656    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-workload-certs\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.894862 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.894736    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-credential-socket\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.894965 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.894862    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-workload-socket\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.894965 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.894901    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-workload-certs\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.894965 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.894933    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-istio-data\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.895136 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.894966    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701ffef5-2564-403a-be8a-44329cf36cb1-istio-token\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.895136 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.894992    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-istio-envoy\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.895136 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.895040    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-credential-socket\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.895270 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.895142    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701ffef5-2564-403a-be8a-44329cf36cb1-istiod-ca-cert\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.895389 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.895365    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-istio-data\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.897017 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.896994    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-istio-envoy\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.897158 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.897129    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701ffef5-2564-403a-be8a-44329cf36cb1-istio-podinfo\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.915655 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.915628    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znscs\" (UniqueName: \"kubernetes.io/projected/701ffef5-2564-403a-be8a-44329cf36cb1-kube-api-access-znscs\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.915882 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.915864    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701ffef5-2564-403a-be8a-44329cf36cb1-istio-token\") pod \"e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") " pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:05.990515 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:05.990427    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:06.285232 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:06.285196    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"]
Sep 11 19:49:06.286739 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:49:06.286693    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701ffef5_2564_403a_be8a_44329cf36cb1.slice/crio-7978b25b8ceb1012816119291802ea613b38b4c3034df8ae00ff739623ec1791 WatchSource:0}: Error finding container 7978b25b8ceb1012816119291802ea613b38b4c3034df8ae00ff739623ec1791: Status 404 returned error can't find the container with id 7978b25b8ceb1012816119291802ea613b38b4c3034df8ae00ff739623ec1791
Sep 11 19:49:06.288933 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:06.288900    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:49:06.289048 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:06.288980    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:49:06.289048 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:06.289026    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:49:07.003442 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:07.003408    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q" event={"ID":"701ffef5-2564-403a-be8a-44329cf36cb1","Type":"ContainerStarted","Data":"c8b65137908be99929807dda54f71453157170cc51ba38e17aae0f493929b8bd"}
Sep 11 19:49:07.003964 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:07.003471    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q" event={"ID":"701ffef5-2564-403a-be8a-44329cf36cb1","Type":"ContainerStarted","Data":"7978b25b8ceb1012816119291802ea613b38b4c3034df8ae00ff739623ec1791"}
Sep 11 19:49:07.025498 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:07.025449    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q" podStartSLOduration=2.025432876 podStartE2EDuration="2.025432876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:49:07.024101325 +0000 UTC m=+1363.641270817" watchObservedRunningTime="2026-09-11 19:49:07.025432876 +0000 UTC m=+1363.642602373"
Sep 11 19:49:07.991612 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:07.991571    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:07.996968 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:07.996939    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:08.010281 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:08.010251    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:08.011226 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:08.011208    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:10.022934 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:10.022902    2558 generic.go:356] "Generic (PLEG): container finished" podID="21785e5f-a2aa-4779-91a7-2f1cf1d4688b" containerID="3c1ad703615d3eef08ea09b80f1fa5297efb15c408a2b70cb1bf9ea59a567f7c" exitCode=0
Sep 11 19:49:10.023357 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:10.022977    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx" event={"ID":"21785e5f-a2aa-4779-91a7-2f1cf1d4688b","Type":"ContainerDied","Data":"3c1ad703615d3eef08ea09b80f1fa5297efb15c408a2b70cb1bf9ea59a567f7c"}
Sep 11 19:49:10.023411 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:10.023395    2558 scope.go:117] "RemoveContainer" containerID="3c1ad703615d3eef08ea09b80f1fa5297efb15c408a2b70cb1bf9ea59a567f7c"
Sep 11 19:49:10.024616 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:10.024586    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:49:10.024687 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:10.024672    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:49:10.024747 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:10.024733    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:49:11.029717 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:11.029673    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx" event={"ID":"21785e5f-a2aa-4779-91a7-2f1cf1d4688b","Type":"ContainerStarted","Data":"b929a9f74ceeb59b12be59e213aaba00a5385f87023c0a04f1419600bd311097"}
Sep 11 19:49:11.415094 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:11.415053    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:49:11.416288 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:11.416264    2558 patch_prober.go:28] interesting pod/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 11 19:49:11.416349 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:11.416314    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx" podUID="21785e5f-a2aa-4779-91a7-2f1cf1d4688b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 11 19:49:12.414803 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:12.414764    2558 patch_prober.go:28] interesting pod/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 11 19:49:12.415212 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:12.414838    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx" podUID="21785e5f-a2aa-4779-91a7-2f1cf1d4688b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 11 19:49:13.415046 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:13.415008    2558 patch_prober.go:28] interesting pod/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 11 19:49:13.415444 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:13.415064    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx" podUID="21785e5f-a2aa-4779-91a7-2f1cf1d4688b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 11 19:49:14.415006 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:14.414963    2558 patch_prober.go:28] interesting pod/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 11 19:49:14.415192 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:14.415030    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx" podUID="21785e5f-a2aa-4779-91a7-2f1cf1d4688b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 11 19:49:14.879609 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:14.879517    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:49:15.414968 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:15.414933    2558 patch_prober.go:28] interesting pod/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 11 19:49:15.415157 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:15.414994    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx" podUID="21785e5f-a2aa-4779-91a7-2f1cf1d4688b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 11 19:49:16.419236 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:16.419199    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:49:16.420464 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:16.420441    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:49:43.214759 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:43.214725    2558 generic.go:356] "Generic (PLEG): container finished" podID="701ffef5-2564-403a-be8a-44329cf36cb1" containerID="c8b65137908be99929807dda54f71453157170cc51ba38e17aae0f493929b8bd" exitCode=0
Sep 11 19:49:43.215301 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:43.214809    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q" event={"ID":"701ffef5-2564-403a-be8a-44329cf36cb1","Type":"ContainerDied","Data":"c8b65137908be99929807dda54f71453157170cc51ba38e17aae0f493929b8bd"}
Sep 11 19:49:43.215375 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:43.215329    2558 scope.go:117] "RemoveContainer" containerID="c8b65137908be99929807dda54f71453157170cc51ba38e17aae0f493929b8bd"
Sep 11 19:49:43.216783 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:43.216741    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:49:43.216906 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:43.216838    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:49:43.216906 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:43.216883    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:49:44.222009 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:44.221974    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q" event={"ID":"701ffef5-2564-403a-be8a-44329cf36cb1","Type":"ContainerStarted","Data":"7d2b5cb5c03b6675c6c85aa81619701af6738db7d531b6f01264811e07fbf65c"}
Sep 11 19:49:44.991375 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:44.991336    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:44.991633 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:44.991602    2558 patch_prober.go:28] interesting pod/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 11 19:49:44.991680 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:44.991662    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q" podUID="701ffef5-2564-403a-be8a-44329cf36cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 11 19:49:45.990949 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:45.990908    2558 patch_prober.go:28] interesting pod/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 11 19:49:45.991367 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:45.990969    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q" podUID="701ffef5-2564-403a-be8a-44329cf36cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 11 19:49:46.991318 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:46.991276    2558 patch_prober.go:28] interesting pod/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 11 19:49:46.991764 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:46.991338    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q" podUID="701ffef5-2564-403a-be8a-44329cf36cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 11 19:49:47.991449 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:47.991407    2558 patch_prober.go:28] interesting pod/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 11 19:49:47.991881 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:47.991472    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q" podUID="701ffef5-2564-403a-be8a-44329cf36cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 11 19:49:48.991421 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:48.991370    2558 patch_prober.go:28] interesting pod/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Sep 11 19:49:48.991628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:48.991459    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q" podUID="701ffef5-2564-403a-be8a-44329cf36cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Sep 11 19:49:49.995584 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:49.995550    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:49.996051 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:49.995876    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:49:49.996776 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:49:49.996757    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:50:05.971387 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:05.971353    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"]
Sep 11 19:50:05.971884 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:05.971612    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q" podUID="701ffef5-2564-403a-be8a-44329cf36cb1" containerName="istio-proxy" containerID="cri-o://7d2b5cb5c03b6675c6c85aa81619701af6738db7d531b6f01264811e07fbf65c" gracePeriod=30
Sep 11 19:50:11.225423 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.225400    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:50:11.373987 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.373894    2558 generic.go:356] "Generic (PLEG): container finished" podID="701ffef5-2564-403a-be8a-44329cf36cb1" containerID="7d2b5cb5c03b6675c6c85aa81619701af6738db7d531b6f01264811e07fbf65c" exitCode=0
Sep 11 19:50:11.373987 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.373940    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"
Sep 11 19:50:11.373987 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.373940    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q" event={"ID":"701ffef5-2564-403a-be8a-44329cf36cb1","Type":"ContainerDied","Data":"7d2b5cb5c03b6675c6c85aa81619701af6738db7d531b6f01264811e07fbf65c"}
Sep 11 19:50:11.374232 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.374001    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q" event={"ID":"701ffef5-2564-403a-be8a-44329cf36cb1","Type":"ContainerDied","Data":"7978b25b8ceb1012816119291802ea613b38b4c3034df8ae00ff739623ec1791"}
Sep 11 19:50:11.374232 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.374027    2558 scope.go:117] "RemoveContainer" containerID="7d2b5cb5c03b6675c6c85aa81619701af6738db7d531b6f01264811e07fbf65c"
Sep 11 19:50:11.382929 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.382908    2558 scope.go:117] "RemoveContainer" containerID="c8b65137908be99929807dda54f71453157170cc51ba38e17aae0f493929b8bd"
Sep 11 19:50:11.391298 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.391278    2558 scope.go:117] "RemoveContainer" containerID="7d2b5cb5c03b6675c6c85aa81619701af6738db7d531b6f01264811e07fbf65c"
Sep 11 19:50:11.391595 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:50:11.391569    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d2b5cb5c03b6675c6c85aa81619701af6738db7d531b6f01264811e07fbf65c\": container with ID starting with 7d2b5cb5c03b6675c6c85aa81619701af6738db7d531b6f01264811e07fbf65c not found: ID does not exist" containerID="7d2b5cb5c03b6675c6c85aa81619701af6738db7d531b6f01264811e07fbf65c"
Sep 11 19:50:11.391654 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.391601    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d2b5cb5c03b6675c6c85aa81619701af6738db7d531b6f01264811e07fbf65c"} err="failed to get container status \"7d2b5cb5c03b6675c6c85aa81619701af6738db7d531b6f01264811e07fbf65c\": rpc error: code = NotFound desc = could not find container \"7d2b5cb5c03b6675c6c85aa81619701af6738db7d531b6f01264811e07fbf65c\": container with ID starting with 7d2b5cb5c03b6675c6c85aa81619701af6738db7d531b6f01264811e07fbf65c not found: ID does not exist"
Sep 11 19:50:11.391654 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.391619    2558 scope.go:117] "RemoveContainer" containerID="c8b65137908be99929807dda54f71453157170cc51ba38e17aae0f493929b8bd"
Sep 11 19:50:11.391993 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:50:11.391960    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b65137908be99929807dda54f71453157170cc51ba38e17aae0f493929b8bd\": container with ID starting with c8b65137908be99929807dda54f71453157170cc51ba38e17aae0f493929b8bd not found: ID does not exist" containerID="c8b65137908be99929807dda54f71453157170cc51ba38e17aae0f493929b8bd"
Sep 11 19:50:11.392114 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.391994    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b65137908be99929807dda54f71453157170cc51ba38e17aae0f493929b8bd"} err="failed to get container status \"c8b65137908be99929807dda54f71453157170cc51ba38e17aae0f493929b8bd\": rpc error: code = NotFound desc = could not find container \"c8b65137908be99929807dda54f71453157170cc51ba38e17aae0f493929b8bd\": container with ID starting with c8b65137908be99929807dda54f71453157170cc51ba38e17aae0f493929b8bd not found: ID does not exist"
Sep 11 19:50:11.415366 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.415335    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701ffef5-2564-403a-be8a-44329cf36cb1-istio-podinfo\") pod \"701ffef5-2564-403a-be8a-44329cf36cb1\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") "
Sep 11 19:50:11.415366 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.415370    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-credential-socket\") pod \"701ffef5-2564-403a-be8a-44329cf36cb1\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") "
Sep 11 19:50:11.415594 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.415391    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701ffef5-2564-403a-be8a-44329cf36cb1-istiod-ca-cert\") pod \"701ffef5-2564-403a-be8a-44329cf36cb1\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") "
Sep 11 19:50:11.415594 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.415418    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znscs\" (UniqueName: \"kubernetes.io/projected/701ffef5-2564-403a-be8a-44329cf36cb1-kube-api-access-znscs\") pod \"701ffef5-2564-403a-be8a-44329cf36cb1\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") "
Sep 11 19:50:11.415594 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.415449    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-workload-socket\") pod \"701ffef5-2564-403a-be8a-44329cf36cb1\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") "
Sep 11 19:50:11.415594 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.415524    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-istio-envoy\") pod \"701ffef5-2564-403a-be8a-44329cf36cb1\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") "
Sep 11 19:50:11.415594 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.415553    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-istio-data\") pod \"701ffef5-2564-403a-be8a-44329cf36cb1\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") "
Sep 11 19:50:11.415594 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.415593    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-workload-certs\") pod \"701ffef5-2564-403a-be8a-44329cf36cb1\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") "
Sep 11 19:50:11.415954 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.415639    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "701ffef5-2564-403a-be8a-44329cf36cb1" (UID: "701ffef5-2564-403a-be8a-44329cf36cb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:50:11.415954 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.415703    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701ffef5-2564-403a-be8a-44329cf36cb1-istio-token\") pod \"701ffef5-2564-403a-be8a-44329cf36cb1\" (UID: \"701ffef5-2564-403a-be8a-44329cf36cb1\") "
Sep 11 19:50:11.415954 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.415870    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "701ffef5-2564-403a-be8a-44329cf36cb1" (UID: "701ffef5-2564-403a-be8a-44329cf36cb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:50:11.415954 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.415913    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-istio-data" (OuterVolumeSpecName: "istio-data") pod "701ffef5-2564-403a-be8a-44329cf36cb1" (UID: "701ffef5-2564-403a-be8a-44329cf36cb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:50:11.415954 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.415928    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "701ffef5-2564-403a-be8a-44329cf36cb1" (UID: "701ffef5-2564-403a-be8a-44329cf36cb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:50:11.416268 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.415972    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/701ffef5-2564-403a-be8a-44329cf36cb1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "701ffef5-2564-403a-be8a-44329cf36cb1" (UID: "701ffef5-2564-403a-be8a-44329cf36cb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:50:11.416268 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.416100    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-credential-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:50:11.416268 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.416114    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701ffef5-2564-403a-be8a-44329cf36cb1-istiod-ca-cert\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:50:11.416268 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.416124    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-workload-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:50:11.416268 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.416144    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-istio-data\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:50:11.416268 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.416162    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-workload-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:50:11.417764 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.417739    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/701ffef5-2564-403a-be8a-44329cf36cb1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "701ffef5-2564-403a-be8a-44329cf36cb1" (UID: "701ffef5-2564-403a-be8a-44329cf36cb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:50:11.417884 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.417841    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701ffef5-2564-403a-be8a-44329cf36cb1-kube-api-access-znscs" (OuterVolumeSpecName: "kube-api-access-znscs") pod "701ffef5-2564-403a-be8a-44329cf36cb1" (UID: "701ffef5-2564-403a-be8a-44329cf36cb1"). InnerVolumeSpecName "kube-api-access-znscs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:50:11.417958 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.417938    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701ffef5-2564-403a-be8a-44329cf36cb1-istio-token" (OuterVolumeSpecName: "istio-token") pod "701ffef5-2564-403a-be8a-44329cf36cb1" (UID: "701ffef5-2564-403a-be8a-44329cf36cb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:50:11.418068 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.418042    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "701ffef5-2564-403a-be8a-44329cf36cb1" (UID: "701ffef5-2564-403a-be8a-44329cf36cb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:50:11.443972 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.443936    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"]
Sep 11 19:50:11.444619 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.444603    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="701ffef5-2564-403a-be8a-44329cf36cb1" containerName="istio-proxy"
Sep 11 19:50:11.444669 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.444624    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="701ffef5-2564-403a-be8a-44329cf36cb1" containerName="istio-proxy"
Sep 11 19:50:11.444765 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.444753    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="701ffef5-2564-403a-be8a-44329cf36cb1" containerName="istio-proxy"
Sep 11 19:50:11.460659 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.460623    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"]
Sep 11 19:50:11.460844 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.460774    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:11.463143 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.463117    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-a29559\"/\"default-dockercfg-95znp\""
Sep 11 19:50:11.463351 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.463204    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-a29559\"/\"openshift-service-ca.crt\""
Sep 11 19:50:11.463351 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.463236    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-a29559\"/\"explicit-model-a29559-kserve-self-signed-certs\""
Sep 11 19:50:11.463351 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.463117    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-a29559\"/\"kube-root-ca.crt\""
Sep 11 19:50:11.517740 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.517692    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701ffef5-2564-403a-be8a-44329cf36cb1-istio-envoy\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:50:11.517740 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.517739    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701ffef5-2564-403a-be8a-44329cf36cb1-istio-token\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:50:11.518008 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.517753    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701ffef5-2564-403a-be8a-44329cf36cb1-istio-podinfo\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:50:11.518008 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.517766    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znscs\" (UniqueName: \"kubernetes.io/projected/701ffef5-2564-403a-be8a-44329cf36cb1-kube-api-access-znscs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:50:11.618790 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.618745    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-tls-certs\") pod \"explicit-model-a29559-kserve-9847d5cbc-nxg6j\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") " pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:11.618974 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.618816    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-tmp-dir\") pod \"explicit-model-a29559-kserve-9847d5cbc-nxg6j\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") " pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:11.618974 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.618875    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-dshm\") pod \"explicit-model-a29559-kserve-9847d5cbc-nxg6j\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") " pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:11.619067 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.618991    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdxbl\" (UniqueName: \"kubernetes.io/projected/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-kube-api-access-sdxbl\") pod \"explicit-model-a29559-kserve-9847d5cbc-nxg6j\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") " pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:11.619067 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.619022    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-model-cache\") pod \"explicit-model-a29559-kserve-9847d5cbc-nxg6j\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") " pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:11.619067 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.619046    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-home\") pod \"explicit-model-a29559-kserve-9847d5cbc-nxg6j\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") " pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:11.697368 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.697339    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"]
Sep 11 19:50:11.703515 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.703486    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-02df91be-openshift-default-5d5d775544-4vd5q"]
Sep 11 19:50:11.723381 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.719899    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdxbl\" (UniqueName: \"kubernetes.io/projected/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-kube-api-access-sdxbl\") pod \"explicit-model-a29559-kserve-9847d5cbc-nxg6j\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") " pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:11.723381 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.719948    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-model-cache\") pod \"explicit-model-a29559-kserve-9847d5cbc-nxg6j\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") " pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:11.723381 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.719979    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-home\") pod \"explicit-model-a29559-kserve-9847d5cbc-nxg6j\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") " pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:11.723381 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.720039    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-tls-certs\") pod \"explicit-model-a29559-kserve-9847d5cbc-nxg6j\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") " pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:11.723381 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.720094    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-tmp-dir\") pod \"explicit-model-a29559-kserve-9847d5cbc-nxg6j\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") " pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:11.723381 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.720174    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-dshm\") pod \"explicit-model-a29559-kserve-9847d5cbc-nxg6j\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") " pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:11.723381 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.721146    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-home\") pod \"explicit-model-a29559-kserve-9847d5cbc-nxg6j\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") " pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:11.723381 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.721910    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-tmp-dir\") pod \"explicit-model-a29559-kserve-9847d5cbc-nxg6j\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") " pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:11.723381 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.722615    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-model-cache\") pod \"explicit-model-a29559-kserve-9847d5cbc-nxg6j\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") " pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:11.723381 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.723375    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-dshm\") pod \"explicit-model-a29559-kserve-9847d5cbc-nxg6j\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") " pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:11.726361 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.726321    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-tls-certs\") pod \"explicit-model-a29559-kserve-9847d5cbc-nxg6j\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") " pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:11.730441 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.730419    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdxbl\" (UniqueName: \"kubernetes.io/projected/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-kube-api-access-sdxbl\") pod \"explicit-model-a29559-kserve-9847d5cbc-nxg6j\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") " pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:11.771497 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.771460    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:11.815904 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:11.815862    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="701ffef5-2564-403a-be8a-44329cf36cb1" path="/var/lib/kubelet/pods/701ffef5-2564-403a-be8a-44329cf36cb1/volumes"
Sep 11 19:50:12.048013 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:12.047988    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"]
Sep 11 19:50:12.049758 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:50:12.049729    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod673832fc_c81b_4f3c_ba43_98e1dbcfbfbf.slice/crio-c88a2cc85f16554666bc9205e2143dcd872a9e2807aeab63b24af4c7dd25f36a WatchSource:0}: Error finding container c88a2cc85f16554666bc9205e2143dcd872a9e2807aeab63b24af4c7dd25f36a: Status 404 returned error can't find the container with id c88a2cc85f16554666bc9205e2143dcd872a9e2807aeab63b24af4c7dd25f36a
Sep 11 19:50:12.381825 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:12.381728    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j" event={"ID":"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf","Type":"ContainerStarted","Data":"dbf42b5dd988c2863b5a3c21d1dc5a92f59ff5dfde207a176fa806dcb39bd37c"}
Sep 11 19:50:12.381825 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:12.381776    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j" event={"ID":"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf","Type":"ContainerStarted","Data":"c88a2cc85f16554666bc9205e2143dcd872a9e2807aeab63b24af4c7dd25f36a"}
Sep 11 19:50:12.398227 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:12.398179    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j" podStartSLOduration=1.398163444 podStartE2EDuration="1.398163444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:50:12.396793048 +0000 UTC m=+1429.013962541" watchObservedRunningTime="2026-09-11 19:50:12.398163444 +0000 UTC m=+1429.015332938"
Sep 11 19:50:21.772092 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:21.771992    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:21.772092 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:21.772053    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:21.785085 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:21.785057    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:22.446555 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:22.446525    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:38.968298 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:38.968256    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"]
Sep 11 19:50:38.968751 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:38.968717    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="701ffef5-2564-403a-be8a-44329cf36cb1" containerName="istio-proxy"
Sep 11 19:50:38.968751 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:38.968732    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="701ffef5-2564-403a-be8a-44329cf36cb1" containerName="istio-proxy"
Sep 11 19:50:38.968850 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:38.968837    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="701ffef5-2564-403a-be8a-44329cf36cb1" containerName="istio-proxy"
Sep 11 19:50:38.971552 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:38.971535    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:38.974148 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:38.974092    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c7766c\"/\"default-dockercfg-vxzf9\""
Sep 11 19:50:38.974872 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:38.974850    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c7766c\"/\"kube-root-ca.crt\""
Sep 11 19:50:38.974872 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:38.974863    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c7766c\"/\"openshift-service-ca.crt\""
Sep 11 19:50:38.975044 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:38.974871    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c7766c\"/\"e2e-conflict-a-ccaf36e3-kserve-self-signed-certs\""
Sep 11 19:50:38.982124 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:38.982091    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"]
Sep 11 19:50:39.114182 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:39.114148    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-model-cache\") pod \"e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") " pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:39.114381 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:39.114191    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6csv\" (UniqueName: \"kubernetes.io/projected/119033c9-c279-4cd9-abec-a7142451e0e4-kube-api-access-f6csv\") pod \"e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") " pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:39.114381 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:39.114317    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-home\") pod \"e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") " pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:39.114515 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:39.114419    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119033c9-c279-4cd9-abec-a7142451e0e4-tls-certs\") pod \"e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") " pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:39.114515 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:39.114459    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-dshm\") pod \"e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") " pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:39.114629 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:39.114549    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-tmp-dir\") pod \"e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") " pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:39.216206 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:39.216163    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-tmp-dir\") pod \"e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") " pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:39.216409 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:39.216227    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-model-cache\") pod \"e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") " pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:39.216409 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:39.216256    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6csv\" (UniqueName: \"kubernetes.io/projected/119033c9-c279-4cd9-abec-a7142451e0e4-kube-api-access-f6csv\") pod \"e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") " pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:39.216409 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:39.216312    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-home\") pod \"e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") " pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:39.216409 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:39.216382    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119033c9-c279-4cd9-abec-a7142451e0e4-tls-certs\") pod \"e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") " pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:39.216637 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:39.216409    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-dshm\") pod \"e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") " pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:39.216637 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:39.216558    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-tmp-dir\") pod \"e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") " pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:39.216637 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:39.216615    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-model-cache\") pod \"e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") " pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:39.216822 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:39.216690    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-home\") pod \"e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") " pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:39.218692 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:39.218669    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-dshm\") pod \"e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") " pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:39.218944 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:39.218920    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119033c9-c279-4cd9-abec-a7142451e0e4-tls-certs\") pod \"e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") " pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:39.232865 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:39.232797    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6csv\" (UniqueName: \"kubernetes.io/projected/119033c9-c279-4cd9-abec-a7142451e0e4-kube-api-access-f6csv\") pod \"e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") " pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:39.290683 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:39.290643    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:39.564217 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:39.564183    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"]
Sep 11 19:50:39.565237 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:50:39.565207    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod119033c9_c279_4cd9_abec_a7142451e0e4.slice/crio-79ec972fa56f6aed4b7ed8600801a3827dc4417a08ad11591b0ae8e1bdabc673 WatchSource:0}: Error finding container 79ec972fa56f6aed4b7ed8600801a3827dc4417a08ad11591b0ae8e1bdabc673: Status 404 returned error can't find the container with id 79ec972fa56f6aed4b7ed8600801a3827dc4417a08ad11591b0ae8e1bdabc673
Sep 11 19:50:40.532882 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:40.532834    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl" event={"ID":"119033c9-c279-4cd9-abec-a7142451e0e4","Type":"ContainerStarted","Data":"c86cf6476ac1b17552c4a3f13e2833f2c3c9d9b951c5639601ceed40de50cb10"}
Sep 11 19:50:40.532882 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:40.532877    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl" event={"ID":"119033c9-c279-4cd9-abec-a7142451e0e4","Type":"ContainerStarted","Data":"79ec972fa56f6aed4b7ed8600801a3827dc4417a08ad11591b0ae8e1bdabc673"}
Sep 11 19:50:40.547942 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:40.547841    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl" podStartSLOduration=2.547824 podStartE2EDuration="2.547824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:50:40.547371118 +0000 UTC m=+1457.164540609" watchObservedRunningTime="2026-09-11 19:50:40.547824 +0000 UTC m=+1457.164993492"
Sep 11 19:50:43.044813 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:43.044782    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"]
Sep 11 19:50:49.290918 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:49.290873    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:49.291306 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:49.291059    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:49.304011 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:49.303984    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:49.592932 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:49.592843    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:50:55.848088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:55.848053    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"]
Sep 11 19:50:58.064631 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.064560    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j" podUID="673832fc-c81b-4f3c-ba43-98e1dbcfbfbf" containerName="main" containerID="cri-o://dbf42b5dd988c2863b5a3c21d1dc5a92f59ff5dfde207a176fa806dcb39bd37c" gracePeriod=45
Sep 11 19:50:58.316460 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.316398    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:58.420150 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.420112    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdxbl\" (UniqueName: \"kubernetes.io/projected/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-kube-api-access-sdxbl\") pod \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") "
Sep 11 19:50:58.420339 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.420172    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-tls-certs\") pod \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") "
Sep 11 19:50:58.420339 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.420226    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-home\") pod \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") "
Sep 11 19:50:58.420339 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.420311    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-dshm\") pod \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") "
Sep 11 19:50:58.420504 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.420341    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-tmp-dir\") pod \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") "
Sep 11 19:50:58.420504 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.420379    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-model-cache\") pod \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\" (UID: \"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf\") "
Sep 11 19:50:58.420614 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.420584    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-home" (OuterVolumeSpecName: "home") pod "673832fc-c81b-4f3c-ba43-98e1dbcfbfbf" (UID: "673832fc-c81b-4f3c-ba43-98e1dbcfbfbf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:50:58.420672 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.420616    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "673832fc-c81b-4f3c-ba43-98e1dbcfbfbf" (UID: "673832fc-c81b-4f3c-ba43-98e1dbcfbfbf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:50:58.420778 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.420755    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-model-cache" (OuterVolumeSpecName: "model-cache") pod "673832fc-c81b-4f3c-ba43-98e1dbcfbfbf" (UID: "673832fc-c81b-4f3c-ba43-98e1dbcfbfbf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:50:58.422592 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.422564    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-dshm" (OuterVolumeSpecName: "dshm") pod "673832fc-c81b-4f3c-ba43-98e1dbcfbfbf" (UID: "673832fc-c81b-4f3c-ba43-98e1dbcfbfbf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:50:58.422860 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.422841    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "673832fc-c81b-4f3c-ba43-98e1dbcfbfbf" (UID: "673832fc-c81b-4f3c-ba43-98e1dbcfbfbf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:50:58.422912 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.422866    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-kube-api-access-sdxbl" (OuterVolumeSpecName: "kube-api-access-sdxbl") pod "673832fc-c81b-4f3c-ba43-98e1dbcfbfbf" (UID: "673832fc-c81b-4f3c-ba43-98e1dbcfbfbf"). InnerVolumeSpecName "kube-api-access-sdxbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:50:58.521249 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.521216    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-model-cache\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:50:58.521249 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.521246    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdxbl\" (UniqueName: \"kubernetes.io/projected/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-kube-api-access-sdxbl\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:50:58.521454 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.521261    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-tls-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:50:58.521454 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.521274    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-home\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:50:58.521454 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.521284    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-dshm\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:50:58.521454 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.521296    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf-tmp-dir\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:50:58.632038 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.631942    2558 generic.go:356] "Generic (PLEG): container finished" podID="673832fc-c81b-4f3c-ba43-98e1dbcfbfbf" containerID="dbf42b5dd988c2863b5a3c21d1dc5a92f59ff5dfde207a176fa806dcb39bd37c" exitCode=0
Sep 11 19:50:58.632038 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.631992    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"
Sep 11 19:50:58.632038 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.632026    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j" event={"ID":"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf","Type":"ContainerDied","Data":"dbf42b5dd988c2863b5a3c21d1dc5a92f59ff5dfde207a176fa806dcb39bd37c"}
Sep 11 19:50:58.632260 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.632079    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j" event={"ID":"673832fc-c81b-4f3c-ba43-98e1dbcfbfbf","Type":"ContainerDied","Data":"c88a2cc85f16554666bc9205e2143dcd872a9e2807aeab63b24af4c7dd25f36a"}
Sep 11 19:50:58.632260 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.632106    2558 scope.go:117] "RemoveContainer" containerID="dbf42b5dd988c2863b5a3c21d1dc5a92f59ff5dfde207a176fa806dcb39bd37c"
Sep 11 19:50:58.642110 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.642090    2558 scope.go:117] "RemoveContainer" containerID="dbf42b5dd988c2863b5a3c21d1dc5a92f59ff5dfde207a176fa806dcb39bd37c"
Sep 11 19:50:58.642409 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:50:58.642394    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf42b5dd988c2863b5a3c21d1dc5a92f59ff5dfde207a176fa806dcb39bd37c\": container with ID starting with dbf42b5dd988c2863b5a3c21d1dc5a92f59ff5dfde207a176fa806dcb39bd37c not found: ID does not exist" containerID="dbf42b5dd988c2863b5a3c21d1dc5a92f59ff5dfde207a176fa806dcb39bd37c"
Sep 11 19:50:58.642458 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.642416    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf42b5dd988c2863b5a3c21d1dc5a92f59ff5dfde207a176fa806dcb39bd37c"} err="failed to get container status \"dbf42b5dd988c2863b5a3c21d1dc5a92f59ff5dfde207a176fa806dcb39bd37c\": rpc error: code = NotFound desc = could not find container \"dbf42b5dd988c2863b5a3c21d1dc5a92f59ff5dfde207a176fa806dcb39bd37c\": container with ID starting with dbf42b5dd988c2863b5a3c21d1dc5a92f59ff5dfde207a176fa806dcb39bd37c not found: ID does not exist"
Sep 11 19:50:58.657901 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.657864    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"]
Sep 11 19:50:58.660690 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:58.660666    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-a29559/explicit-model-a29559-kserve-9847d5cbc-nxg6j"]
Sep 11 19:50:59.494426 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:59.494391    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"]
Sep 11 19:50:59.495351 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:59.495324    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673832fc-c81b-4f3c-ba43-98e1dbcfbfbf" containerName="main"
Sep 11 19:50:59.495351 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:59.495350    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="673832fc-c81b-4f3c-ba43-98e1dbcfbfbf" containerName="main"
Sep 11 19:50:59.495552 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:59.495461    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="673832fc-c81b-4f3c-ba43-98e1dbcfbfbf" containerName="main"
Sep 11 19:50:59.498762 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:59.498736    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"
Sep 11 19:50:59.501252 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:59.501231    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-d14d7e-serving-cert\""
Sep 11 19:50:59.521152 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:59.521120    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"]
Sep 11 19:50:59.634067 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:59.634016    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a505761d-cc6a-41be-b089-2437e90c755a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t\" (UID: \"a505761d-cc6a-41be-b089-2437e90c755a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"
Sep 11 19:50:59.634376 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:59.634349    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a505761d-cc6a-41be-b089-2437e90c755a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t\" (UID: \"a505761d-cc6a-41be-b089-2437e90c755a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"
Sep 11 19:50:59.634521 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:59.634391    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a505761d-cc6a-41be-b089-2437e90c755a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t\" (UID: \"a505761d-cc6a-41be-b089-2437e90c755a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"
Sep 11 19:50:59.634662 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:59.634632    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc6jg\" (UniqueName: \"kubernetes.io/projected/a505761d-cc6a-41be-b089-2437e90c755a-kube-api-access-hc6jg\") pod \"maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t\" (UID: \"a505761d-cc6a-41be-b089-2437e90c755a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"
Sep 11 19:50:59.736024 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:59.735990    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc6jg\" (UniqueName: \"kubernetes.io/projected/a505761d-cc6a-41be-b089-2437e90c755a-kube-api-access-hc6jg\") pod \"maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t\" (UID: \"a505761d-cc6a-41be-b089-2437e90c755a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"
Sep 11 19:50:59.736218 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:59.736067    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a505761d-cc6a-41be-b089-2437e90c755a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t\" (UID: \"a505761d-cc6a-41be-b089-2437e90c755a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"
Sep 11 19:50:59.736218 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:59.736107    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a505761d-cc6a-41be-b089-2437e90c755a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t\" (UID: \"a505761d-cc6a-41be-b089-2437e90c755a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"
Sep 11 19:50:59.736218 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:59.736137    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a505761d-cc6a-41be-b089-2437e90c755a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t\" (UID: \"a505761d-cc6a-41be-b089-2437e90c755a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"
Sep 11 19:50:59.736382 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:50:59.736279    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-serving-cert: secret "maas-api-e2e-shared-a-w2-d14d7e-serving-cert" not found
Sep 11 19:50:59.736382 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:50:59.736343    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a505761d-cc6a-41be-b089-2437e90c755a-maas-api-tls podName:a505761d-cc6a-41be-b089-2437e90c755a nodeName:}" failed. No retries permitted until 2026-09-11 19:51:00.236323445 +0000 UTC m=+1476.853492914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a505761d-cc6a-41be-b089-2437e90c755a-maas-api-tls") pod "maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t" (UID: "a505761d-cc6a-41be-b089-2437e90c755a") : secret "maas-api-e2e-shared-a-w2-d14d7e-serving-cert" not found
Sep 11 19:50:59.736995 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:59.736965    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a505761d-cc6a-41be-b089-2437e90c755a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t\" (UID: \"a505761d-cc6a-41be-b089-2437e90c755a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"
Sep 11 19:50:59.738678 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:59.738653    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a505761d-cc6a-41be-b089-2437e90c755a-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t\" (UID: \"a505761d-cc6a-41be-b089-2437e90c755a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"
Sep 11 19:50:59.744519 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:59.744450    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc6jg\" (UniqueName: \"kubernetes.io/projected/a505761d-cc6a-41be-b089-2437e90c755a-kube-api-access-hc6jg\") pod \"maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t\" (UID: \"a505761d-cc6a-41be-b089-2437e90c755a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"
Sep 11 19:50:59.815039 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:50:59.815002    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="673832fc-c81b-4f3c-ba43-98e1dbcfbfbf" path="/var/lib/kubelet/pods/673832fc-c81b-4f3c-ba43-98e1dbcfbfbf/volumes"
Sep 11 19:51:00.241528 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:00.241493    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a505761d-cc6a-41be-b089-2437e90c755a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t\" (UID: \"a505761d-cc6a-41be-b089-2437e90c755a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"
Sep 11 19:51:00.244018 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:00.243990    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a505761d-cc6a-41be-b089-2437e90c755a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t\" (UID: \"a505761d-cc6a-41be-b089-2437e90c755a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"
Sep 11 19:51:00.423629 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:00.423593    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"
Sep 11 19:51:00.698906 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:00.698875    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"]
Sep 11 19:51:00.699968 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:51:00.699943    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda505761d_cc6a_41be_b089_2437e90c755a.slice/crio-a3f48018f7e84f6042ff251cf03d2156fd9d4e91d29241b443c29050bf2f6aee WatchSource:0}: Error finding container a3f48018f7e84f6042ff251cf03d2156fd9d4e91d29241b443c29050bf2f6aee: Status 404 returned error can't find the container with id a3f48018f7e84f6042ff251cf03d2156fd9d4e91d29241b443c29050bf2f6aee
Sep 11 19:51:00.701450 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:00.701432    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 19:51:01.657211 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:01.657118    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t" event={"ID":"a505761d-cc6a-41be-b089-2437e90c755a","Type":"ContainerStarted","Data":"ab56831cb8443cc521666f01e6ad84c3428d3a53d0d08613315178ea852dbcb1"}
Sep 11 19:51:01.657211 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:01.657176    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t" event={"ID":"a505761d-cc6a-41be-b089-2437e90c755a","Type":"ContainerStarted","Data":"a3f48018f7e84f6042ff251cf03d2156fd9d4e91d29241b443c29050bf2f6aee"}
Sep 11 19:51:01.657444 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:01.657266    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"
Sep 11 19:51:01.680088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:01.680022    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t" podStartSLOduration=2.21558335 podStartE2EDuration="2.680004447s" totalImagesPullingTime="464.421097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:51:01.678239338 +0000 UTC m=+1478.295408829" watchObservedRunningTime="2026-09-11 19:51:01.680004447 +0000 UTC m=+1478.297173937"
Sep 11 19:51:06.432236 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:06.432197    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"]
Sep 11 19:51:06.432746 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:06.432437    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx" podUID="21785e5f-a2aa-4779-91a7-2f1cf1d4688b" containerName="istio-proxy" containerID="cri-o://b929a9f74ceeb59b12be59e213aaba00a5385f87023c0a04f1419600bd311097" gracePeriod=30
Sep 11 19:51:07.668641 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:07.668610    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"
Sep 11 19:51:08.305541 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.305494    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"]
Sep 11 19:51:08.306004 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.305783    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw" podUID="6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb" containerName="maas-api" containerID="cri-o://3cf070cd00bb0d2c9c84d85c8d49e774ab60d3997871e146e0c4bab0cf455d41" gracePeriod=30
Sep 11 19:51:08.582943 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.582914    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"
Sep 11 19:51:08.700108 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.700071    2558 generic.go:356] "Generic (PLEG): container finished" podID="6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb" containerID="3cf070cd00bb0d2c9c84d85c8d49e774ab60d3997871e146e0c4bab0cf455d41" exitCode=0
Sep 11 19:51:08.700498 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.700154    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"
Sep 11 19:51:08.700498 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.700186    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw" event={"ID":"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb","Type":"ContainerDied","Data":"3cf070cd00bb0d2c9c84d85c8d49e774ab60d3997871e146e0c4bab0cf455d41"}
Sep 11 19:51:08.700498 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.700244    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw" event={"ID":"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb","Type":"ContainerDied","Data":"3928d43a3fe6e5b93e63278b274c3e53d44b72623b685200c0f77f8da3c8a1e5"}
Sep 11 19:51:08.700498 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.700279    2558 scope.go:117] "RemoveContainer" containerID="3cf070cd00bb0d2c9c84d85c8d49e774ab60d3997871e146e0c4bab0cf455d41"
Sep 11 19:51:08.710279 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.710252    2558 scope.go:117] "RemoveContainer" containerID="3cf070cd00bb0d2c9c84d85c8d49e774ab60d3997871e146e0c4bab0cf455d41"
Sep 11 19:51:08.710558 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:51:08.710533    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf070cd00bb0d2c9c84d85c8d49e774ab60d3997871e146e0c4bab0cf455d41\": container with ID starting with 3cf070cd00bb0d2c9c84d85c8d49e774ab60d3997871e146e0c4bab0cf455d41 not found: ID does not exist" containerID="3cf070cd00bb0d2c9c84d85c8d49e774ab60d3997871e146e0c4bab0cf455d41"
Sep 11 19:51:08.710662 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.710571    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf070cd00bb0d2c9c84d85c8d49e774ab60d3997871e146e0c4bab0cf455d41"} err="failed to get container status \"3cf070cd00bb0d2c9c84d85c8d49e774ab60d3997871e146e0c4bab0cf455d41\": rpc error: code = NotFound desc = could not find container \"3cf070cd00bb0d2c9c84d85c8d49e774ab60d3997871e146e0c4bab0cf455d41\": container with ID starting with 3cf070cd00bb0d2c9c84d85c8d49e774ab60d3997871e146e0c4bab0cf455d41 not found: ID does not exist"
Sep 11 19:51:08.726936 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.726911    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-openshift-service-ca\") pod \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\" (UID: \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\") "
Sep 11 19:51:08.727057 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.727006    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-metrics-certs\") pod \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\" (UID: \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\") "
Sep 11 19:51:08.727057 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.727032    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-maas-api-tls\") pod \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\" (UID: \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\") "
Sep 11 19:51:08.727142 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.727103    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zrt6\" (UniqueName: \"kubernetes.io/projected/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-kube-api-access-8zrt6\") pod \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\" (UID: \"6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb\") "
Sep 11 19:51:08.727326 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.727302    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb" (UID: "6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:51:08.727491 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.727470    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-openshift-service-ca\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:08.729178 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.729152    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb" (UID: "6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:51:08.729178 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.729167    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb" (UID: "6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:51:08.729550 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.729249    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-kube-api-access-8zrt6" (OuterVolumeSpecName: "kube-api-access-8zrt6") pod "6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb" (UID: "6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb"). InnerVolumeSpecName "kube-api-access-8zrt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:51:08.828697 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.828614    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-metrics-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:08.828697 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.828647    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-maas-api-tls\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:08.828697 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:08.828657    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zrt6\" (UniqueName: \"kubernetes.io/projected/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb-kube-api-access-8zrt6\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:09.031408 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:09.031373    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"]
Sep 11 19:51:09.038686 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:09.038658    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ec80ad-7d899fc457-f8xpw"]
Sep 11 19:51:09.814879 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:09.814844    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb" path="/var/lib/kubelet/pods/6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb/volumes"
Sep 11 19:51:10.868002 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:10.867962    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf" podUID="a988a97f-8992-4195-94f0-4c261e839bfa" containerName="main" containerID="cri-o://f933b769a9a6182b08318e866e67449a6c2c7dcb9e1dc1f85ea5b8bfe6d96fd6" gracePeriod=45
Sep 11 19:51:11.133616 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.133547    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:51:11.254776 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.254735    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-home\") pod \"a988a97f-8992-4195-94f0-4c261e839bfa\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") "
Sep 11 19:51:11.254965 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.254781    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lwsc\" (UniqueName: \"kubernetes.io/projected/a988a97f-8992-4195-94f0-4c261e839bfa-kube-api-access-8lwsc\") pod \"a988a97f-8992-4195-94f0-4c261e839bfa\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") "
Sep 11 19:51:11.254965 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.254882    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-model-cache\") pod \"a988a97f-8992-4195-94f0-4c261e839bfa\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") "
Sep 11 19:51:11.254965 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.254934    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-tmp-dir\") pod \"a988a97f-8992-4195-94f0-4c261e839bfa\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") "
Sep 11 19:51:11.254965 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.254954    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-dshm\") pod \"a988a97f-8992-4195-94f0-4c261e839bfa\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") "
Sep 11 19:51:11.255181 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.255036    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a988a97f-8992-4195-94f0-4c261e839bfa-tls-certs\") pod \"a988a97f-8992-4195-94f0-4c261e839bfa\" (UID: \"a988a97f-8992-4195-94f0-4c261e839bfa\") "
Sep 11 19:51:11.255181 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.255064    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-home" (OuterVolumeSpecName: "home") pod "a988a97f-8992-4195-94f0-4c261e839bfa" (UID: "a988a97f-8992-4195-94f0-4c261e839bfa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:51:11.255282 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.255184    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a988a97f-8992-4195-94f0-4c261e839bfa" (UID: "a988a97f-8992-4195-94f0-4c261e839bfa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:51:11.255282 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.255192    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-model-cache" (OuterVolumeSpecName: "model-cache") pod "a988a97f-8992-4195-94f0-4c261e839bfa" (UID: "a988a97f-8992-4195-94f0-4c261e839bfa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:51:11.255413 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.255396    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-home\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:11.255469 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.255420    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-model-cache\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:11.255469 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.255434    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-tmp-dir\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:11.257117 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.257089    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a988a97f-8992-4195-94f0-4c261e839bfa-kube-api-access-8lwsc" (OuterVolumeSpecName: "kube-api-access-8lwsc") pod "a988a97f-8992-4195-94f0-4c261e839bfa" (UID: "a988a97f-8992-4195-94f0-4c261e839bfa"). InnerVolumeSpecName "kube-api-access-8lwsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:51:11.257237 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.257150    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-dshm" (OuterVolumeSpecName: "dshm") pod "a988a97f-8992-4195-94f0-4c261e839bfa" (UID: "a988a97f-8992-4195-94f0-4c261e839bfa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:51:11.257237 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.257191    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a988a97f-8992-4195-94f0-4c261e839bfa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a988a97f-8992-4195-94f0-4c261e839bfa" (UID: "a988a97f-8992-4195-94f0-4c261e839bfa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:51:11.356529 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.356483    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a988a97f-8992-4195-94f0-4c261e839bfa-tls-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:11.356529 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.356515    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lwsc\" (UniqueName: \"kubernetes.io/projected/a988a97f-8992-4195-94f0-4c261e839bfa-kube-api-access-8lwsc\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:11.356529 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.356525    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a988a97f-8992-4195-94f0-4c261e839bfa-dshm\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:11.685402 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.685375    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:51:11.720159 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.720075    2558 generic.go:356] "Generic (PLEG): container finished" podID="21785e5f-a2aa-4779-91a7-2f1cf1d4688b" containerID="b929a9f74ceeb59b12be59e213aaba00a5385f87023c0a04f1419600bd311097" exitCode=0
Sep 11 19:51:11.720159 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.720139    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"
Sep 11 19:51:11.720159 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.720159    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx" event={"ID":"21785e5f-a2aa-4779-91a7-2f1cf1d4688b","Type":"ContainerDied","Data":"b929a9f74ceeb59b12be59e213aaba00a5385f87023c0a04f1419600bd311097"}
Sep 11 19:51:11.720474 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.720219    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx" event={"ID":"21785e5f-a2aa-4779-91a7-2f1cf1d4688b","Type":"ContainerDied","Data":"0670ab8f72509055c7f2affd213e4a5c6840d6203e71d77ca2f0795bce1fae7c"}
Sep 11 19:51:11.720474 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.720247    2558 scope.go:117] "RemoveContainer" containerID="b929a9f74ceeb59b12be59e213aaba00a5385f87023c0a04f1419600bd311097"
Sep 11 19:51:11.721609 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.721588    2558 generic.go:356] "Generic (PLEG): container finished" podID="a988a97f-8992-4195-94f0-4c261e839bfa" containerID="f933b769a9a6182b08318e866e67449a6c2c7dcb9e1dc1f85ea5b8bfe6d96fd6" exitCode=0
Sep 11 19:51:11.721744 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.721652    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"
Sep 11 19:51:11.721744 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.721679    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf" event={"ID":"a988a97f-8992-4195-94f0-4c261e839bfa","Type":"ContainerDied","Data":"f933b769a9a6182b08318e866e67449a6c2c7dcb9e1dc1f85ea5b8bfe6d96fd6"}
Sep 11 19:51:11.721848 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.721771    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf" event={"ID":"a988a97f-8992-4195-94f0-4c261e839bfa","Type":"ContainerDied","Data":"a6c3a685dede09d357b02ea343b85fec2303a79603d42bcdfcdf6637c7bf8269"}
Sep 11 19:51:11.730868 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.730754    2558 scope.go:117] "RemoveContainer" containerID="3c1ad703615d3eef08ea09b80f1fa5297efb15c408a2b70cb1bf9ea59a567f7c"
Sep 11 19:51:11.741646 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.741624    2558 scope.go:117] "RemoveContainer" containerID="b929a9f74ceeb59b12be59e213aaba00a5385f87023c0a04f1419600bd311097"
Sep 11 19:51:11.741969 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:51:11.741951    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b929a9f74ceeb59b12be59e213aaba00a5385f87023c0a04f1419600bd311097\": container with ID starting with b929a9f74ceeb59b12be59e213aaba00a5385f87023c0a04f1419600bd311097 not found: ID does not exist" containerID="b929a9f74ceeb59b12be59e213aaba00a5385f87023c0a04f1419600bd311097"
Sep 11 19:51:11.742050 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.741982    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b929a9f74ceeb59b12be59e213aaba00a5385f87023c0a04f1419600bd311097"} err="failed to get container status \"b929a9f74ceeb59b12be59e213aaba00a5385f87023c0a04f1419600bd311097\": rpc error: code = NotFound desc = could not find container \"b929a9f74ceeb59b12be59e213aaba00a5385f87023c0a04f1419600bd311097\": container with ID starting with b929a9f74ceeb59b12be59e213aaba00a5385f87023c0a04f1419600bd311097 not found: ID does not exist"
Sep 11 19:51:11.742050 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.742005    2558 scope.go:117] "RemoveContainer" containerID="3c1ad703615d3eef08ea09b80f1fa5297efb15c408a2b70cb1bf9ea59a567f7c"
Sep 11 19:51:11.742289 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:51:11.742259    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c1ad703615d3eef08ea09b80f1fa5297efb15c408a2b70cb1bf9ea59a567f7c\": container with ID starting with 3c1ad703615d3eef08ea09b80f1fa5297efb15c408a2b70cb1bf9ea59a567f7c not found: ID does not exist" containerID="3c1ad703615d3eef08ea09b80f1fa5297efb15c408a2b70cb1bf9ea59a567f7c"
Sep 11 19:51:11.742332 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.742298    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c1ad703615d3eef08ea09b80f1fa5297efb15c408a2b70cb1bf9ea59a567f7c"} err="failed to get container status \"3c1ad703615d3eef08ea09b80f1fa5297efb15c408a2b70cb1bf9ea59a567f7c\": rpc error: code = NotFound desc = could not find container \"3c1ad703615d3eef08ea09b80f1fa5297efb15c408a2b70cb1bf9ea59a567f7c\": container with ID starting with 3c1ad703615d3eef08ea09b80f1fa5297efb15c408a2b70cb1bf9ea59a567f7c not found: ID does not exist"
Sep 11 19:51:11.742332 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.742319    2558 scope.go:117] "RemoveContainer" containerID="f933b769a9a6182b08318e866e67449a6c2c7dcb9e1dc1f85ea5b8bfe6d96fd6"
Sep 11 19:51:11.744125 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.744104    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"]
Sep 11 19:51:11.748654 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.748631    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ec80ad/test-model-ec80ad-kserve-79dbb4d787-tlkjf"]
Sep 11 19:51:11.751501 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.751479    2558 scope.go:117] "RemoveContainer" containerID="f933b769a9a6182b08318e866e67449a6c2c7dcb9e1dc1f85ea5b8bfe6d96fd6"
Sep 11 19:51:11.751816 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:51:11.751787    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f933b769a9a6182b08318e866e67449a6c2c7dcb9e1dc1f85ea5b8bfe6d96fd6\": container with ID starting with f933b769a9a6182b08318e866e67449a6c2c7dcb9e1dc1f85ea5b8bfe6d96fd6 not found: ID does not exist" containerID="f933b769a9a6182b08318e866e67449a6c2c7dcb9e1dc1f85ea5b8bfe6d96fd6"
Sep 11 19:51:11.751886 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.751822    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f933b769a9a6182b08318e866e67449a6c2c7dcb9e1dc1f85ea5b8bfe6d96fd6"} err="failed to get container status \"f933b769a9a6182b08318e866e67449a6c2c7dcb9e1dc1f85ea5b8bfe6d96fd6\": rpc error: code = NotFound desc = could not find container \"f933b769a9a6182b08318e866e67449a6c2c7dcb9e1dc1f85ea5b8bfe6d96fd6\": container with ID starting with f933b769a9a6182b08318e866e67449a6c2c7dcb9e1dc1f85ea5b8bfe6d96fd6 not found: ID does not exist"
Sep 11 19:51:11.813967 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.813933    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a988a97f-8992-4195-94f0-4c261e839bfa" path="/var/lib/kubelet/pods/a988a97f-8992-4195-94f0-4c261e839bfa/volumes"
Sep 11 19:51:11.862113 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.862077    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-data\") pod \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") "
Sep 11 19:51:11.862304 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.862141    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istiod-ca-cert\") pod \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") "
Sep 11 19:51:11.862304 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.862189    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-credential-socket\") pod \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") "
Sep 11 19:51:11.862304 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.862248    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-token\") pod \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") "
Sep 11 19:51:11.862304 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.862287    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp9qs\" (UniqueName: \"kubernetes.io/projected/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-kube-api-access-bp9qs\") pod \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") "
Sep 11 19:51:11.862536 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.862319    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-workload-certs\") pod \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") "
Sep 11 19:51:11.862536 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.862344    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-workload-socket\") pod \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") "
Sep 11 19:51:11.862536 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.862368    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-podinfo\") pod \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") "
Sep 11 19:51:11.862536 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.862437    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-data" (OuterVolumeSpecName: "istio-data") pod "21785e5f-a2aa-4779-91a7-2f1cf1d4688b" (UID: "21785e5f-a2aa-4779-91a7-2f1cf1d4688b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:51:11.862536 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.862468    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-envoy\") pod \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\" (UID: \"21785e5f-a2aa-4779-91a7-2f1cf1d4688b\") "
Sep 11 19:51:11.862536 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.862524    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "21785e5f-a2aa-4779-91a7-2f1cf1d4688b" (UID: "21785e5f-a2aa-4779-91a7-2f1cf1d4688b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:51:11.862851 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.862572    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "21785e5f-a2aa-4779-91a7-2f1cf1d4688b" (UID: "21785e5f-a2aa-4779-91a7-2f1cf1d4688b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:51:11.862851 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.862750    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "21785e5f-a2aa-4779-91a7-2f1cf1d4688b" (UID: "21785e5f-a2aa-4779-91a7-2f1cf1d4688b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:51:11.862956 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.862915    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "21785e5f-a2aa-4779-91a7-2f1cf1d4688b" (UID: "21785e5f-a2aa-4779-91a7-2f1cf1d4688b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:51:11.863071 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.863050    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-workload-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:11.863141 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.863076    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-workload-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:11.863141 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.863090    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-data\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:11.863141 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.863103    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istiod-ca-cert\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:11.863141 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.863115    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-credential-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:11.864808 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.864781    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "21785e5f-a2aa-4779-91a7-2f1cf1d4688b" (UID: "21785e5f-a2aa-4779-91a7-2f1cf1d4688b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:51:11.864934 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.864827    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-token" (OuterVolumeSpecName: "istio-token") pod "21785e5f-a2aa-4779-91a7-2f1cf1d4688b" (UID: "21785e5f-a2aa-4779-91a7-2f1cf1d4688b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:51:11.864934 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.864892    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-kube-api-access-bp9qs" (OuterVolumeSpecName: "kube-api-access-bp9qs") pod "21785e5f-a2aa-4779-91a7-2f1cf1d4688b" (UID: "21785e5f-a2aa-4779-91a7-2f1cf1d4688b"). InnerVolumeSpecName "kube-api-access-bp9qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:51:11.865017 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.864994    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "21785e5f-a2aa-4779-91a7-2f1cf1d4688b" (UID: "21785e5f-a2aa-4779-91a7-2f1cf1d4688b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:51:11.963875 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.963836    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-token\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:11.963875 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.963866    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp9qs\" (UniqueName: \"kubernetes.io/projected/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-kube-api-access-bp9qs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:11.963875 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.963876    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-podinfo\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:11.963875 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:11.963884    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21785e5f-a2aa-4779-91a7-2f1cf1d4688b-istio-envoy\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:12.041694 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:12.041662    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"]
Sep 11 19:51:12.049684 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:12.049656    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-a29559-openshift-default-68765dfb8d-lxvcx"]
Sep 11 19:51:13.814512 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:13.814477    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21785e5f-a2aa-4779-91a7-2f1cf1d4688b" path="/var/lib/kubelet/pods/21785e5f-a2aa-4779-91a7-2f1cf1d4688b/volumes"
Sep 11 19:51:14.924142 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.924108    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"]
Sep 11 19:51:14.924622 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.924605    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a988a97f-8992-4195-94f0-4c261e839bfa" containerName="main"
Sep 11 19:51:14.924739 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.924626    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a988a97f-8992-4195-94f0-4c261e839bfa" containerName="main"
Sep 11 19:51:14.924739 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.924648    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21785e5f-a2aa-4779-91a7-2f1cf1d4688b" containerName="istio-proxy"
Sep 11 19:51:14.924739 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.924656    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="21785e5f-a2aa-4779-91a7-2f1cf1d4688b" containerName="istio-proxy"
Sep 11 19:51:14.924739 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.924664    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb" containerName="maas-api"
Sep 11 19:51:14.924739 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.924672    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb" containerName="maas-api"
Sep 11 19:51:14.924739 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.924681    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21785e5f-a2aa-4779-91a7-2f1cf1d4688b" containerName="istio-proxy"
Sep 11 19:51:14.924739 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.924686    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="21785e5f-a2aa-4779-91a7-2f1cf1d4688b" containerName="istio-proxy"
Sep 11 19:51:14.924983 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.924788    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a988a97f-8992-4195-94f0-4c261e839bfa" containerName="main"
Sep 11 19:51:14.924983 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.924802    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6368fdd9-5b9c-4d0d-9f42-ddfba3f052bb" containerName="maas-api"
Sep 11 19:51:14.924983 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.924807    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="21785e5f-a2aa-4779-91a7-2f1cf1d4688b" containerName="istio-proxy"
Sep 11 19:51:14.924983 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.924812    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="21785e5f-a2aa-4779-91a7-2f1cf1d4688b" containerName="istio-proxy"
Sep 11 19:51:14.929597 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.929574    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:14.932102 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.932080    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-58f57d-openshift-default-dockercfg-m7q87\""
Sep 11 19:51:14.938995 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.938968    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"]
Sep 11 19:51:14.993987 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.993947    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm8px\" (UniqueName: \"kubernetes.io/projected/17652cd9-6ad9-4318-bee9-a09da4162e94-kube-api-access-sm8px\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:14.994146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.993994    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-token\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:14.994146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.994081    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-credential-socket\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:14.994146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.994115    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-workload-socket\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:14.994146 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.994136    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-podinfo\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:14.994340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.994216    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17652cd9-6ad9-4318-bee9-a09da4162e94-istiod-ca-cert\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:14.994340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.994236    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-data\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:14.994340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.994263    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-workload-certs\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:14.994340 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:14.994282    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-envoy\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:15.095339 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.095299    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-workload-socket\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:15.095339 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.095342    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-podinfo\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:15.095613 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.095370    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17652cd9-6ad9-4318-bee9-a09da4162e94-istiod-ca-cert\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:15.095613 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.095390    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-data\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:15.095613 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.095421    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-workload-certs\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:15.095613 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.095448    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-envoy\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:15.095613 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.095550    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm8px\" (UniqueName: \"kubernetes.io/projected/17652cd9-6ad9-4318-bee9-a09da4162e94-kube-api-access-sm8px\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:15.095613 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.095586    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-token\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:15.095960 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.095631    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-credential-socket\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:15.095960 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.095813    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-workload-socket\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:15.095960 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.095865    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-data\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:15.096156 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.096128    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-workload-certs\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:15.096222 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.096168    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-credential-socket\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:15.096263 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.096221    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17652cd9-6ad9-4318-bee9-a09da4162e94-istiod-ca-cert\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:15.097924 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.097894    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-envoy\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:15.098001 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.097953    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-podinfo\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:15.103598 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.103568    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-token\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:15.103765 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.103744    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm8px\" (UniqueName: \"kubernetes.io/projected/17652cd9-6ad9-4318-bee9-a09da4162e94-kube-api-access-sm8px\") pod \"e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") " pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:15.241124 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.241028    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:15.519560 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.519533    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"]
Sep 11 19:51:15.522066 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:51:15.522039    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17652cd9_6ad9_4318_bee9_a09da4162e94.slice/crio-86e3585490e5d9519d6ea158ca105e867e0ba8912e74f3966f0dc4feedfe9149 WatchSource:0}: Error finding container 86e3585490e5d9519d6ea158ca105e867e0ba8912e74f3966f0dc4feedfe9149: Status 404 returned error can't find the container with id 86e3585490e5d9519d6ea158ca105e867e0ba8912e74f3966f0dc4feedfe9149
Sep 11 19:51:15.524576 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.524542    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:51:15.524651 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.524610    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:51:15.524651 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.524637    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:51:15.750115 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.750071    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz" event={"ID":"17652cd9-6ad9-4318-bee9-a09da4162e94","Type":"ContainerStarted","Data":"c78d728f36119988833e0a2f8fdef8378130b0493471d45d69d3be292278d5b6"}
Sep 11 19:51:15.750321 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.750137    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz" event={"ID":"17652cd9-6ad9-4318-bee9-a09da4162e94","Type":"ContainerStarted","Data":"86e3585490e5d9519d6ea158ca105e867e0ba8912e74f3966f0dc4feedfe9149"}
Sep 11 19:51:15.778155 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:15.778042    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz" podStartSLOduration=1.778023778 podStartE2EDuration="1.778023778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:51:15.775059989 +0000 UTC m=+1492.392229480" watchObservedRunningTime="2026-09-11 19:51:15.778023778 +0000 UTC m=+1492.395193269"
Sep 11 19:51:16.141368 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:16.141269    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"]
Sep 11 19:51:16.141849 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:16.141475    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8" podUID="e3adf554-075e-45e2-8e5c-0019ce755a1a" containerName="istio-proxy" containerID="cri-o://8323aa4f9f3fa53993063e6d78a0ece1091d91d815f36572e1fab07a34555b4d" gracePeriod=30
Sep 11 19:51:16.242022 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:16.241981    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:16.270277 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:16.270244    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:16.756737 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:16.756681    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:16.757598 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:16.757574    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:17.857125 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:17.857089    2558 patch_prober.go:28] interesting pod/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 19:51:17.857485 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:17.857158    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8" podUID="e3adf554-075e-45e2-8e5c-0019ce755a1a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 19:51:21.396973 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.396944    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:51:21.403469 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.403388    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"]
Sep 11 19:51:21.404141 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.404117    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3adf554-075e-45e2-8e5c-0019ce755a1a" containerName="istio-proxy"
Sep 11 19:51:21.404141 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.404142    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3adf554-075e-45e2-8e5c-0019ce755a1a" containerName="istio-proxy"
Sep 11 19:51:21.404326 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.404273    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3adf554-075e-45e2-8e5c-0019ce755a1a" containerName="istio-proxy"
Sep 11 19:51:21.404326 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.404289    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3adf554-075e-45e2-8e5c-0019ce755a1a" containerName="istio-proxy"
Sep 11 19:51:21.407756 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.407734    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"
Sep 11 19:51:21.409968 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.409916    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-58f57d-serving-cert\""
Sep 11 19:51:21.415670 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.415649    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"]
Sep 11 19:51:21.461723 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.461683    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-envoy\") pod \"e3adf554-075e-45e2-8e5c-0019ce755a1a\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") "
Sep 11 19:51:21.461879 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.461739    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-credential-socket\") pod \"e3adf554-075e-45e2-8e5c-0019ce755a1a\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") "
Sep 11 19:51:21.461879 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.461768    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-workload-socket\") pod \"e3adf554-075e-45e2-8e5c-0019ce755a1a\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") "
Sep 11 19:51:21.461879 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.461801    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-workload-certs\") pod \"e3adf554-075e-45e2-8e5c-0019ce755a1a\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") "
Sep 11 19:51:21.461879 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.461821    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqc82\" (UniqueName: \"kubernetes.io/projected/e3adf554-075e-45e2-8e5c-0019ce755a1a-kube-api-access-gqc82\") pod \"e3adf554-075e-45e2-8e5c-0019ce755a1a\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") "
Sep 11 19:51:21.461879 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.461843    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-podinfo\") pod \"e3adf554-075e-45e2-8e5c-0019ce755a1a\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") "
Sep 11 19:51:21.462150 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.461884    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-data\") pod \"e3adf554-075e-45e2-8e5c-0019ce755a1a\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") "
Sep 11 19:51:21.462150 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.461926    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3adf554-075e-45e2-8e5c-0019ce755a1a-istiod-ca-cert\") pod \"e3adf554-075e-45e2-8e5c-0019ce755a1a\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") "
Sep 11 19:51:21.462150 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.462028    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3adf554-075e-45e2-8e5c-0019ce755a1a" (UID: "e3adf554-075e-45e2-8e5c-0019ce755a1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:51:21.462150 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.462052    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-token\") pod \"e3adf554-075e-45e2-8e5c-0019ce755a1a\" (UID: \"e3adf554-075e-45e2-8e5c-0019ce755a1a\") "
Sep 11 19:51:21.462362 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.462153    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3adf554-075e-45e2-8e5c-0019ce755a1a" (UID: "e3adf554-075e-45e2-8e5c-0019ce755a1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:51:21.462362 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.462272    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4\" (UID: \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"
Sep 11 19:51:21.462362 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.462315    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3adf554-075e-45e2-8e5c-0019ce755a1a" (UID: "e3adf554-075e-45e2-8e5c-0019ce755a1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:51:21.462503 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.462401    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4\" (UID: \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"
Sep 11 19:51:21.462559 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.462497    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3adf554-075e-45e2-8e5c-0019ce755a1a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3adf554-075e-45e2-8e5c-0019ce755a1a" (UID: "e3adf554-075e-45e2-8e5c-0019ce755a1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:51:21.462626 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.462585    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nn7p\" (UniqueName: \"kubernetes.io/projected/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-kube-api-access-2nn7p\") pod \"maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4\" (UID: \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"
Sep 11 19:51:21.462626 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.462589    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3adf554-075e-45e2-8e5c-0019ce755a1a" (UID: "e3adf554-075e-45e2-8e5c-0019ce755a1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:51:21.462739 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.462660    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4\" (UID: \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"
Sep 11 19:51:21.462821 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.462805    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-credential-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:21.462865 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.462828    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-workload-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:21.462865 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.462844    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-workload-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:21.462865 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.462860    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-data\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:21.462956 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.462876    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3adf554-075e-45e2-8e5c-0019ce755a1a-istiod-ca-cert\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:21.464055 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.464022    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3adf554-075e-45e2-8e5c-0019ce755a1a" (UID: "e3adf554-075e-45e2-8e5c-0019ce755a1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:51:21.464168 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.464108    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3adf554-075e-45e2-8e5c-0019ce755a1a-kube-api-access-gqc82" (OuterVolumeSpecName: "kube-api-access-gqc82") pod "e3adf554-075e-45e2-8e5c-0019ce755a1a" (UID: "e3adf554-075e-45e2-8e5c-0019ce755a1a"). InnerVolumeSpecName "kube-api-access-gqc82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:51:21.464503 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.464485    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3adf554-075e-45e2-8e5c-0019ce755a1a" (UID: "e3adf554-075e-45e2-8e5c-0019ce755a1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:51:21.464503 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.464489    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3adf554-075e-45e2-8e5c-0019ce755a1a" (UID: "e3adf554-075e-45e2-8e5c-0019ce755a1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:51:21.564274 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.564207    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nn7p\" (UniqueName: \"kubernetes.io/projected/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-kube-api-access-2nn7p\") pod \"maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4\" (UID: \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"
Sep 11 19:51:21.564474 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.564295    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4\" (UID: \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"
Sep 11 19:51:21.564474 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.564341    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4\" (UID: \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"
Sep 11 19:51:21.564474 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.564395    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4\" (UID: \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"
Sep 11 19:51:21.564474 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.564448    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-token\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:21.564474 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:51:21.564457    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-serving-cert: secret "maas-api-e2e-shared-b-w2-58f57d-serving-cert" not found
Sep 11 19:51:21.564782 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:51:21.564552    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-maas-api-tls podName:1dad3dae-b05e-4d88-bdf0-c31406c7ede3 nodeName:}" failed. No retries permitted until 2026-09-11 19:51:22.064529458 +0000 UTC m=+1498.681698929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-maas-api-tls") pod "maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4" (UID: "1dad3dae-b05e-4d88-bdf0-c31406c7ede3") : secret "maas-api-e2e-shared-b-w2-58f57d-serving-cert" not found
Sep 11 19:51:21.564782 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.564462    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-envoy\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:21.564782 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.564606    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqc82\" (UniqueName: \"kubernetes.io/projected/e3adf554-075e-45e2-8e5c-0019ce755a1a-kube-api-access-gqc82\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:21.564782 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.564620    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3adf554-075e-45e2-8e5c-0019ce755a1a-istio-podinfo\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:51:21.565132 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.565114    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4\" (UID: \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"
Sep 11 19:51:21.566874 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.566858    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4\" (UID: \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"
Sep 11 19:51:21.574471 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.574448    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nn7p\" (UniqueName: \"kubernetes.io/projected/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-kube-api-access-2nn7p\") pod \"maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4\" (UID: \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"
Sep 11 19:51:21.784905 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.784867    2558 generic.go:356] "Generic (PLEG): container finished" podID="e3adf554-075e-45e2-8e5c-0019ce755a1a" containerID="8323aa4f9f3fa53993063e6d78a0ece1091d91d815f36572e1fab07a34555b4d" exitCode=0
Sep 11 19:51:21.785096 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.784917    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"
Sep 11 19:51:21.785096 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.784956    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8" event={"ID":"e3adf554-075e-45e2-8e5c-0019ce755a1a","Type":"ContainerDied","Data":"8323aa4f9f3fa53993063e6d78a0ece1091d91d815f36572e1fab07a34555b4d"}
Sep 11 19:51:21.785096 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.785011    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8" event={"ID":"e3adf554-075e-45e2-8e5c-0019ce755a1a","Type":"ContainerDied","Data":"c809204cfe4cb334dd61f361d8f477e33d8840e12110c675d75781bb75fbfcdf"}
Sep 11 19:51:21.785096 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.785043    2558 scope.go:117] "RemoveContainer" containerID="8323aa4f9f3fa53993063e6d78a0ece1091d91d815f36572e1fab07a34555b4d"
Sep 11 19:51:21.794433 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.794416    2558 scope.go:117] "RemoveContainer" containerID="bcff301226a83253444b84d95dd932bddca541d1abd2c8cc3e7f1f65f9bfa193"
Sep 11 19:51:21.802774 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.802749    2558 scope.go:117] "RemoveContainer" containerID="8323aa4f9f3fa53993063e6d78a0ece1091d91d815f36572e1fab07a34555b4d"
Sep 11 19:51:21.803125 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:51:21.803075    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8323aa4f9f3fa53993063e6d78a0ece1091d91d815f36572e1fab07a34555b4d\": container with ID starting with 8323aa4f9f3fa53993063e6d78a0ece1091d91d815f36572e1fab07a34555b4d not found: ID does not exist" containerID="8323aa4f9f3fa53993063e6d78a0ece1091d91d815f36572e1fab07a34555b4d"
Sep 11 19:51:21.803125 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.803114    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8323aa4f9f3fa53993063e6d78a0ece1091d91d815f36572e1fab07a34555b4d"} err="failed to get container status \"8323aa4f9f3fa53993063e6d78a0ece1091d91d815f36572e1fab07a34555b4d\": rpc error: code = NotFound desc = could not find container \"8323aa4f9f3fa53993063e6d78a0ece1091d91d815f36572e1fab07a34555b4d\": container with ID starting with 8323aa4f9f3fa53993063e6d78a0ece1091d91d815f36572e1fab07a34555b4d not found: ID does not exist"
Sep 11 19:51:21.803310 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.803138    2558 scope.go:117] "RemoveContainer" containerID="bcff301226a83253444b84d95dd932bddca541d1abd2c8cc3e7f1f65f9bfa193"
Sep 11 19:51:21.803856 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:51:21.803574    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcff301226a83253444b84d95dd932bddca541d1abd2c8cc3e7f1f65f9bfa193\": container with ID starting with bcff301226a83253444b84d95dd932bddca541d1abd2c8cc3e7f1f65f9bfa193 not found: ID does not exist" containerID="bcff301226a83253444b84d95dd932bddca541d1abd2c8cc3e7f1f65f9bfa193"
Sep 11 19:51:21.803856 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.803607    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcff301226a83253444b84d95dd932bddca541d1abd2c8cc3e7f1f65f9bfa193"} err="failed to get container status \"bcff301226a83253444b84d95dd932bddca541d1abd2c8cc3e7f1f65f9bfa193\": rpc error: code = NotFound desc = could not find container \"bcff301226a83253444b84d95dd932bddca541d1abd2c8cc3e7f1f65f9bfa193\": container with ID starting with bcff301226a83253444b84d95dd932bddca541d1abd2c8cc3e7f1f65f9bfa193 not found: ID does not exist"
Sep 11 19:51:21.807006 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.806978    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"]
Sep 11 19:51:21.814533 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:21.814513    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ec80ad-openshift-default-766db9fd84-tqdk8"]
Sep 11 19:51:22.069658 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:22.069571    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4\" (UID: \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"
Sep 11 19:51:22.071912 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:22.071894    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4\" (UID: \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"
Sep 11 19:51:22.323272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:22.323186    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"
Sep 11 19:51:22.601556 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:22.601529    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"]
Sep 11 19:51:22.603261 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:51:22.603233    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dad3dae_b05e_4d88_bdf0_c31406c7ede3.slice/crio-80fdf15f8d5fd9303cc406c98ad83efd0769d3ee78f9ece8528f2137023f36e5 WatchSource:0}: Error finding container 80fdf15f8d5fd9303cc406c98ad83efd0769d3ee78f9ece8528f2137023f36e5: Status 404 returned error can't find the container with id 80fdf15f8d5fd9303cc406c98ad83efd0769d3ee78f9ece8528f2137023f36e5
Sep 11 19:51:22.792671 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:22.792636    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4" event={"ID":"1dad3dae-b05e-4d88-bdf0-c31406c7ede3","Type":"ContainerStarted","Data":"80fdf15f8d5fd9303cc406c98ad83efd0769d3ee78f9ece8528f2137023f36e5"}
Sep 11 19:51:23.799945 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:23.799909    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4" event={"ID":"1dad3dae-b05e-4d88-bdf0-c31406c7ede3","Type":"ContainerStarted","Data":"c8301399abfff3bf0ff61ea7ec6388b45ae110f01dece49bc5428de3c6750e3f"}
Sep 11 19:51:23.800343 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:23.800027    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"
Sep 11 19:51:23.814168 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:23.814138    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3adf554-075e-45e2-8e5c-0019ce755a1a" path="/var/lib/kubelet/pods/e3adf554-075e-45e2-8e5c-0019ce755a1a/volumes"
Sep 11 19:51:23.817871 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:23.817827    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4" podStartSLOduration=2.48100271 podStartE2EDuration="2.817812189s" totalImagesPullingTime="336.809479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:51:23.816127038 +0000 UTC m=+1500.433296519" watchObservedRunningTime="2026-09-11 19:51:23.817812189 +0000 UTC m=+1500.434981680"
Sep 11 19:51:29.814680 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:29.814652    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"
Sep 11 19:51:35.198812 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.198775    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"]
Sep 11 19:51:35.199315 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.199297    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3adf554-075e-45e2-8e5c-0019ce755a1a" containerName="istio-proxy"
Sep 11 19:51:35.199383 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.199318    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3adf554-075e-45e2-8e5c-0019ce755a1a" containerName="istio-proxy"
Sep 11 19:51:35.202786 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.202763    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.205029 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.205008    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-58c78c-openshift-default-dockercfg-gbj7g\""
Sep 11 19:51:35.212464 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.212434    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"]
Sep 11 19:51:35.314933 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.314889    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-token\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.314933 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.314935    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istiod-ca-cert\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.315165 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.314967    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-credential-socket\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.315165 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.315018    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-workload-certs\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.315165 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.315105    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-envoy\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.315279 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.315166    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ljlg\" (UniqueName: \"kubernetes.io/projected/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-kube-api-access-7ljlg\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.315279 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.315189    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-workload-socket\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.315358 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.315273    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-podinfo\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.315405 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.315364    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-data\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.416421 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.416379    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-data\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.416615 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.416448    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-token\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.416615 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.416475    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istiod-ca-cert\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.416615 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.416502    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-credential-socket\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.416615 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.416537    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-workload-certs\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.416615 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.416591    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-envoy\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.416898 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.416621    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ljlg\" (UniqueName: \"kubernetes.io/projected/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-kube-api-access-7ljlg\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.416898 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.416652    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-workload-socket\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.416898 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.416750    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-podinfo\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.417038 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.417007    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-credential-socket\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.417102 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.417058    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-workload-certs\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.417160 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.417125    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-data\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.417265 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.417222    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-workload-socket\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.417344 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.417279    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istiod-ca-cert\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.419043 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.419020    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-envoy\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.419167 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.419149    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-podinfo\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.427164 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.427130    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ljlg\" (UniqueName: \"kubernetes.io/projected/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-kube-api-access-7ljlg\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.427315 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.427196    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-token\") pod \"e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") " pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.515289 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.515180    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:35.819244 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.819205    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"]
Sep 11 19:51:35.820461 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:51:35.820425    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a0e675_d5b8_4b37_b40a_cd57e627b4b7.slice/crio-3e79b09caceb06340e7009a01dc8c9a395f49464d381414c3c8be9f83b5e1bf9 WatchSource:0}: Error finding container 3e79b09caceb06340e7009a01dc8c9a395f49464d381414c3c8be9f83b5e1bf9: Status 404 returned error can't find the container with id 3e79b09caceb06340e7009a01dc8c9a395f49464d381414c3c8be9f83b5e1bf9
Sep 11 19:51:35.823082 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.823047    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:51:35.823184 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.823116    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:51:35.823184 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.823144    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:51:35.872645 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:35.872599    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g" event={"ID":"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7","Type":"ContainerStarted","Data":"3e79b09caceb06340e7009a01dc8c9a395f49464d381414c3c8be9f83b5e1bf9"}
Sep 11 19:51:36.879870 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:36.879831    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g" event={"ID":"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7","Type":"ContainerStarted","Data":"0c724a2aa1d16e2858302dcb758fdeecc0a6f35800b0e1e785b8cb57428e199f"}
Sep 11 19:51:37.515777 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:37.515739    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:37.521135 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:37.521110    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:37.543207 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:37.543136    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g" podStartSLOduration=2.543112763 podStartE2EDuration="2.543112763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:51:36.905022202 +0000 UTC m=+1513.522191696" watchObservedRunningTime="2026-09-11 19:51:37.543112763 +0000 UTC m=+1514.160282258"
Sep 11 19:51:37.885491 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:37.885405    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:37.886463 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:37.886442    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:51:40.905100 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:40.905063    2558 generic.go:356] "Generic (PLEG): container finished" podID="17652cd9-6ad9-4318-bee9-a09da4162e94" containerID="c78d728f36119988833e0a2f8fdef8378130b0493471d45d69d3be292278d5b6" exitCode=0
Sep 11 19:51:40.905485 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:40.905136    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz" event={"ID":"17652cd9-6ad9-4318-bee9-a09da4162e94","Type":"ContainerDied","Data":"c78d728f36119988833e0a2f8fdef8378130b0493471d45d69d3be292278d5b6"}
Sep 11 19:51:40.905561 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:40.905546    2558 scope.go:117] "RemoveContainer" containerID="c78d728f36119988833e0a2f8fdef8378130b0493471d45d69d3be292278d5b6"
Sep 11 19:51:40.906816 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:40.906786    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:51:40.906888 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:40.906866    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:51:40.906929 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:40.906900    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:51:41.915434 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:41.915396    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz" event={"ID":"17652cd9-6ad9-4318-bee9-a09da4162e94","Type":"ContainerStarted","Data":"f936ffe594add77eaeb8cbfa10176e35d3ea8ffae86ca5066173104593548219"}
Sep 11 19:51:42.241982 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:42.241899    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:42.243024 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:42.242998    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 11 19:51:42.243100 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:42.243053    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz" podUID="17652cd9-6ad9-4318-bee9-a09da4162e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 11 19:51:43.241841 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:43.241798    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 11 19:51:43.242233 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:43.241867    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz" podUID="17652cd9-6ad9-4318-bee9-a09da4162e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 11 19:51:44.241491 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:44.241441    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 11 19:51:44.241756 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:44.241535    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz" podUID="17652cd9-6ad9-4318-bee9-a09da4162e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 11 19:51:45.242315 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:45.242265    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 11 19:51:45.242847 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:45.242347    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz" podUID="17652cd9-6ad9-4318-bee9-a09da4162e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 11 19:51:46.241472 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:46.241369    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 11 19:51:46.241472 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:46.241447    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz" podUID="17652cd9-6ad9-4318-bee9-a09da4162e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 11 19:51:46.757453 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:46.757401    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:47.241820 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:47.241778    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 11 19:51:47.242023 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:47.241850    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz" podUID="17652cd9-6ad9-4318-bee9-a09da4162e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 11 19:51:48.241737 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:48.241677    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 11 19:51:48.242236 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:48.241761    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz" podUID="17652cd9-6ad9-4318-bee9-a09da4162e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 11 19:51:49.245797 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:49.245760    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:49.246749 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:49.246730    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 19:51:49.700651 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:49.700616    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"]
Sep 11 19:51:49.704274 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:49.704250    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"
Sep 11 19:51:49.706252 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:49.706228    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c0517f60-serving-cert\""
Sep 11 19:51:49.718523 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:49.718493    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"]
Sep 11 19:51:49.776734 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:49.776668    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-openshift-service-ca\") pod \"maas-api-e2e-mt-c0517f60-68759fcb64-n4htw\" (UID: \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"
Sep 11 19:51:49.776734 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:49.776735    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-maas-api-tls\") pod \"maas-api-e2e-mt-c0517f60-68759fcb64-n4htw\" (UID: \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"
Sep 11 19:51:49.776953 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:49.776786    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bvqq\" (UniqueName: \"kubernetes.io/projected/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-kube-api-access-8bvqq\") pod \"maas-api-e2e-mt-c0517f60-68759fcb64-n4htw\" (UID: \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"
Sep 11 19:51:49.776953 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:49.776871    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-metrics-certs\") pod \"maas-api-e2e-mt-c0517f60-68759fcb64-n4htw\" (UID: \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"
Sep 11 19:51:49.877647 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:49.877609    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-openshift-service-ca\") pod \"maas-api-e2e-mt-c0517f60-68759fcb64-n4htw\" (UID: \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"
Sep 11 19:51:49.877647 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:49.877649    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-maas-api-tls\") pod \"maas-api-e2e-mt-c0517f60-68759fcb64-n4htw\" (UID: \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"
Sep 11 19:51:49.877940 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:49.877686    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bvqq\" (UniqueName: \"kubernetes.io/projected/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-kube-api-access-8bvqq\") pod \"maas-api-e2e-mt-c0517f60-68759fcb64-n4htw\" (UID: \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"
Sep 11 19:51:49.877940 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:49.877765    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-metrics-certs\") pod \"maas-api-e2e-mt-c0517f60-68759fcb64-n4htw\" (UID: \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"
Sep 11 19:51:49.877940 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:51:49.877792    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-serving-cert: secret "maas-api-e2e-mt-c0517f60-serving-cert" not found
Sep 11 19:51:49.877940 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:51:49.877862    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-maas-api-tls podName:a60aea1c-add5-4a57-84f8-b5c0e9ab4006 nodeName:}" failed. No retries permitted until 2026-09-11 19:51:50.3778446 +0000 UTC m=+1526.995014071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-maas-api-tls") pod "maas-api-e2e-mt-c0517f60-68759fcb64-n4htw" (UID: "a60aea1c-add5-4a57-84f8-b5c0e9ab4006") : secret "maas-api-e2e-mt-c0517f60-serving-cert" not found
Sep 11 19:51:49.878355 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:49.878332    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-openshift-service-ca\") pod \"maas-api-e2e-mt-c0517f60-68759fcb64-n4htw\" (UID: \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"
Sep 11 19:51:49.880132 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:49.880107    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-metrics-certs\") pod \"maas-api-e2e-mt-c0517f60-68759fcb64-n4htw\" (UID: \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"
Sep 11 19:51:49.887773 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:49.887751    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bvqq\" (UniqueName: \"kubernetes.io/projected/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-kube-api-access-8bvqq\") pod \"maas-api-e2e-mt-c0517f60-68759fcb64-n4htw\" (UID: \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"
Sep 11 19:51:50.382628 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:50.382590    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-maas-api-tls\") pod \"maas-api-e2e-mt-c0517f60-68759fcb64-n4htw\" (UID: \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"
Sep 11 19:51:50.385085 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:50.385064    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-maas-api-tls\") pod \"maas-api-e2e-mt-c0517f60-68759fcb64-n4htw\" (UID: \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"
Sep 11 19:51:50.616217 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:50.616184    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"
Sep 11 19:51:50.887852 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:50.887823    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"]
Sep 11 19:51:50.888608 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:51:50.888579    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60aea1c_add5_4a57_84f8_b5c0e9ab4006.slice/crio-0da3e87b2d5805f50db0b16bcc4c5a3913f657310f0017cfc2139f4efb2ba8d9 WatchSource:0}: Error finding container 0da3e87b2d5805f50db0b16bcc4c5a3913f657310f0017cfc2139f4efb2ba8d9: Status 404 returned error can't find the container with id 0da3e87b2d5805f50db0b16bcc4c5a3913f657310f0017cfc2139f4efb2ba8d9
Sep 11 19:51:50.974645 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:50.974557    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw" event={"ID":"a60aea1c-add5-4a57-84f8-b5c0e9ab4006","Type":"ContainerStarted","Data":"0da3e87b2d5805f50db0b16bcc4c5a3913f657310f0017cfc2139f4efb2ba8d9"}
Sep 11 19:51:51.982133 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:51.982090    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw" event={"ID":"a60aea1c-add5-4a57-84f8-b5c0e9ab4006","Type":"ContainerStarted","Data":"07da1c87ed2f549a439e0a005be8d79892f012c44fc864593cbb7bcae4583d00"}
Sep 11 19:51:51.982545 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:51.982163    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"
Sep 11 19:51:51.999155 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:51.999104    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw" podStartSLOduration=2.7251208780000002 podStartE2EDuration="2.999086868s" totalImagesPullingTime="273.96599ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:51:51.996465571 +0000 UTC m=+1528.613635063" watchObservedRunningTime="2026-09-11 19:51:51.999086868 +0000 UTC m=+1528.616256358"
Sep 11 19:51:57.999397 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:51:57.999361    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"
Sep 11 19:52:08.927885 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:08.927842    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 19:52:08.928366 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:08.927914    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g" podUID="d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 19:52:09.453207 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.453172    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"]
Sep 11 19:52:09.458933 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.458906    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.462300 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.462273    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-ae268e-openshift-default-dockercfg-4fd64\""
Sep 11 19:52:09.468889 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.468856    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"]
Sep 11 19:52:09.570492 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.570439    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eaede7b-f137-4263-942b-468e5fd15cc9-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.570719 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.570507    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-envoy\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.570719 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.570555    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-token\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.570719 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.570632    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-credential-socket\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.570719 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.570666    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-podinfo\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.570719 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.570693    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-workload-certs\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.571004 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.570819    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-workload-socket\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.571004 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.570851    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-data\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.571004 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.570929    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdp29\" (UniqueName: \"kubernetes.io/projected/6eaede7b-f137-4263-942b-468e5fd15cc9-kube-api-access-xdp29\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.672177 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.672130    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-workload-socket\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.672177 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.672181    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-data\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.672441 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.672252    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdp29\" (UniqueName: \"kubernetes.io/projected/6eaede7b-f137-4263-942b-468e5fd15cc9-kube-api-access-xdp29\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.672441 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.672302    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eaede7b-f137-4263-942b-468e5fd15cc9-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.672441 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.672342    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-envoy\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.672441 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.672369    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-token\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.672441 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.672434    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-credential-socket\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.672722 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.672472    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-podinfo\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.672722 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.672503    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-workload-certs\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.672722 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.672606    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-workload-socket\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.672722 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.672627    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-data\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.672951 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.672907    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-workload-certs\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.672951 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.672941    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-credential-socket\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.673059 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.673035    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eaede7b-f137-4263-942b-468e5fd15cc9-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.675072 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.675051    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-podinfo\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.675211 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.675190    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-envoy\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.681380 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.681361    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-token\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.681478 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.681428    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdp29\" (UniqueName: \"kubernetes.io/projected/6eaede7b-f137-4263-942b-468e5fd15cc9-kube-api-access-xdp29\") pod \"e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") " pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:09.772954 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:09.772854    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:10.067487 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:52:10.067445    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eaede7b_f137_4263_942b_468e5fd15cc9.slice/crio-459deee8eff461b38067a3779fd5cbba89e83057e1d32db0a094a877b37ba3cc WatchSource:0}: Error finding container 459deee8eff461b38067a3779fd5cbba89e83057e1d32db0a094a877b37ba3cc: Status 404 returned error can't find the container with id 459deee8eff461b38067a3779fd5cbba89e83057e1d32db0a094a877b37ba3cc
Sep 11 19:52:10.067487 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:10.067470    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"]
Sep 11 19:52:10.070266 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:10.070220    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:52:10.070377 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:10.070312    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:52:10.070377 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:10.070358    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:52:10.099001 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:10.098960    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9" event={"ID":"6eaede7b-f137-4263-942b-468e5fd15cc9","Type":"ContainerStarted","Data":"459deee8eff461b38067a3779fd5cbba89e83057e1d32db0a094a877b37ba3cc"}
Sep 11 19:52:11.106276 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:11.106183    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9" event={"ID":"6eaede7b-f137-4263-942b-468e5fd15cc9","Type":"ContainerStarted","Data":"c3d2d43249f8553977b8dacb8e9d3b2788fd27062a5763f30084a48bae51c5c4"}
Sep 11 19:52:11.108153 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:11.108055    2558 generic.go:356] "Generic (PLEG): container finished" podID="d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" containerID="0c724a2aa1d16e2858302dcb758fdeecc0a6f35800b0e1e785b8cb57428e199f" exitCode=0
Sep 11 19:52:11.108153 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:11.108105    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g" event={"ID":"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7","Type":"ContainerDied","Data":"0c724a2aa1d16e2858302dcb758fdeecc0a6f35800b0e1e785b8cb57428e199f"}
Sep 11 19:52:11.108857 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:11.108560    2558 scope.go:117] "RemoveContainer" containerID="0c724a2aa1d16e2858302dcb758fdeecc0a6f35800b0e1e785b8cb57428e199f"
Sep 11 19:52:11.109995 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:11.109960    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:52:11.110086 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:11.110034    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:52:11.110086 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:11.110070    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:52:11.125631 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:11.125592    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9" podStartSLOduration=2.125577573 podStartE2EDuration="2.125577573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:52:11.124460253 +0000 UTC m=+1547.741629760" watchObservedRunningTime="2026-09-11 19:52:11.125577573 +0000 UTC m=+1547.742747096"
Sep 11 19:52:11.773479 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:11.773443    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:11.779063 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:11.779030    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:12.115590 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:12.115494    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g" event={"ID":"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7","Type":"ContainerStarted","Data":"1f4df180bba4a9f368e2075e9fa994e7c8589a9a0694e9b62a3c04cc9ff35cef"}
Sep 11 19:52:12.116049 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:12.115672    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:12.116625 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:12.116609    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:12.515475 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:12.515436    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:52:12.515794 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:12.515766    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 11 19:52:12.515858 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:12.515835    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g" podUID="d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 11 19:52:13.515679 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:13.515635    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 11 19:52:13.516066 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:13.515699    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g" podUID="d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 11 19:52:14.516485 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:14.516446    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 11 19:52:14.516904 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:14.516508    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g" podUID="d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 11 19:52:15.516389 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:15.516353    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 11 19:52:15.516590 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:15.516415    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g" podUID="d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 11 19:52:16.516239 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:16.516200    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 11 19:52:16.516419 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:16.516262    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g" podUID="d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 11 19:52:17.519810 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:17.519775    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:52:17.520194 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:17.520062    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:52:17.520880 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:17.520858    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 19:52:41.293695 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:41.293656    2558 generic.go:356] "Generic (PLEG): container finished" podID="6eaede7b-f137-4263-942b-468e5fd15cc9" containerID="c3d2d43249f8553977b8dacb8e9d3b2788fd27062a5763f30084a48bae51c5c4" exitCode=0
Sep 11 19:52:41.294137 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:41.293735    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9" event={"ID":"6eaede7b-f137-4263-942b-468e5fd15cc9","Type":"ContainerDied","Data":"c3d2d43249f8553977b8dacb8e9d3b2788fd27062a5763f30084a48bae51c5c4"}
Sep 11 19:52:41.294208 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:41.294192    2558 scope.go:117] "RemoveContainer" containerID="c3d2d43249f8553977b8dacb8e9d3b2788fd27062a5763f30084a48bae51c5c4"
Sep 11 19:52:41.295623 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:41.295593    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:52:41.295683 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:41.295664    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:52:41.295743 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:41.295696    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 19:52:42.116687 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:42.116657    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:42.301254 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:42.301221    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9" event={"ID":"6eaede7b-f137-4263-942b-468e5fd15cc9","Type":"ContainerStarted","Data":"8747c4b39aa66482dab0ee6e7e2b4ba85100b9adbdc18f946197c635cda05960"}
Sep 11 19:52:42.773769 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:42.773701    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:42.774734 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:42.774687    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 11 19:52:42.774851 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:42.774764    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9" podUID="6eaede7b-f137-4263-942b-468e5fd15cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 11 19:52:43.774118 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:43.774076    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 11 19:52:43.774525 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:43.774143    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9" podUID="6eaede7b-f137-4263-942b-468e5fd15cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 11 19:52:44.773895 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:44.773857    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 11 19:52:44.774103 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:44.773915    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9" podUID="6eaede7b-f137-4263-942b-468e5fd15cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 11 19:52:45.774088 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:45.774048    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 11 19:52:45.774479 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:45.774104    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9" podUID="6eaede7b-f137-4263-942b-468e5fd15cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 11 19:52:46.774232 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:46.774195    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Sep 11 19:52:46.774744 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:46.774279    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9" podUID="6eaede7b-f137-4263-942b-468e5fd15cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Sep 11 19:52:47.816606 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:47.816573    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:47.817101 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:47.817031    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:52:47.826141 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:52:47.826111    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 19:54:17.851907 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:17.851865    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"]
Sep 11 19:54:32.871080 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:32.871036    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl" podUID="119033c9-c279-4cd9-abec-a7142451e0e4" containerName="main" containerID="cri-o://c86cf6476ac1b17552c4a3f13e2833f2c3c9d9b951c5639601ceed40de50cb10" gracePeriod=45
Sep 11 19:54:33.119954 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.119927    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:54:33.197667 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.197629    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-home\") pod \"119033c9-c279-4cd9-abec-a7142451e0e4\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") "
Sep 11 19:54:33.197859 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.197745    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-dshm\") pod \"119033c9-c279-4cd9-abec-a7142451e0e4\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") "
Sep 11 19:54:33.197859 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.197790    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119033c9-c279-4cd9-abec-a7142451e0e4-tls-certs\") pod \"119033c9-c279-4cd9-abec-a7142451e0e4\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") "
Sep 11 19:54:33.197859 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.197809    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-tmp-dir\") pod \"119033c9-c279-4cd9-abec-a7142451e0e4\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") "
Sep 11 19:54:33.198006 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.197859    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6csv\" (UniqueName: \"kubernetes.io/projected/119033c9-c279-4cd9-abec-a7142451e0e4-kube-api-access-f6csv\") pod \"119033c9-c279-4cd9-abec-a7142451e0e4\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") "
Sep 11 19:54:33.198006 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.197881    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-model-cache\") pod \"119033c9-c279-4cd9-abec-a7142451e0e4\" (UID: \"119033c9-c279-4cd9-abec-a7142451e0e4\") "
Sep 11 19:54:33.198006 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.197950    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-home" (OuterVolumeSpecName: "home") pod "119033c9-c279-4cd9-abec-a7142451e0e4" (UID: "119033c9-c279-4cd9-abec-a7142451e0e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:54:33.198182 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.198151    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "119033c9-c279-4cd9-abec-a7142451e0e4" (UID: "119033c9-c279-4cd9-abec-a7142451e0e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:54:33.198269 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.198248    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-home\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:54:33.198269 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.198238    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "119033c9-c279-4cd9-abec-a7142451e0e4" (UID: "119033c9-c279-4cd9-abec-a7142451e0e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:54:33.200078 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.200054    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119033c9-c279-4cd9-abec-a7142451e0e4-kube-api-access-f6csv" (OuterVolumeSpecName: "kube-api-access-f6csv") pod "119033c9-c279-4cd9-abec-a7142451e0e4" (UID: "119033c9-c279-4cd9-abec-a7142451e0e4"). InnerVolumeSpecName "kube-api-access-f6csv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:54:33.200302 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.200281    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-dshm" (OuterVolumeSpecName: "dshm") pod "119033c9-c279-4cd9-abec-a7142451e0e4" (UID: "119033c9-c279-4cd9-abec-a7142451e0e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:54:33.200378 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.200287    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/119033c9-c279-4cd9-abec-a7142451e0e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "119033c9-c279-4cd9-abec-a7142451e0e4" (UID: "119033c9-c279-4cd9-abec-a7142451e0e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:54:33.299238 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.299198    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-dshm\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:54:33.299238 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.299232    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/119033c9-c279-4cd9-abec-a7142451e0e4-tls-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:54:33.299238 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.299243    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-tmp-dir\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:54:33.299490 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.299253    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6csv\" (UniqueName: \"kubernetes.io/projected/119033c9-c279-4cd9-abec-a7142451e0e4-kube-api-access-f6csv\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:54:33.299490 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.299263    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/119033c9-c279-4cd9-abec-a7142451e0e4-model-cache\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:54:33.923794 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.923758    2558 generic.go:356] "Generic (PLEG): container finished" podID="119033c9-c279-4cd9-abec-a7142451e0e4" containerID="c86cf6476ac1b17552c4a3f13e2833f2c3c9d9b951c5639601ceed40de50cb10" exitCode=0
Sep 11 19:54:33.924289 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.923806    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"
Sep 11 19:54:33.924289 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.923848    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl" event={"ID":"119033c9-c279-4cd9-abec-a7142451e0e4","Type":"ContainerDied","Data":"c86cf6476ac1b17552c4a3f13e2833f2c3c9d9b951c5639601ceed40de50cb10"}
Sep 11 19:54:33.924289 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.923912    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl" event={"ID":"119033c9-c279-4cd9-abec-a7142451e0e4","Type":"ContainerDied","Data":"79ec972fa56f6aed4b7ed8600801a3827dc4417a08ad11591b0ae8e1bdabc673"}
Sep 11 19:54:33.924289 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.923939    2558 scope.go:117] "RemoveContainer" containerID="c86cf6476ac1b17552c4a3f13e2833f2c3c9d9b951c5639601ceed40de50cb10"
Sep 11 19:54:33.932798 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.932775    2558 scope.go:117] "RemoveContainer" containerID="c86cf6476ac1b17552c4a3f13e2833f2c3c9d9b951c5639601ceed40de50cb10"
Sep 11 19:54:33.933062 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:54:33.933041    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c86cf6476ac1b17552c4a3f13e2833f2c3c9d9b951c5639601ceed40de50cb10\": container with ID starting with c86cf6476ac1b17552c4a3f13e2833f2c3c9d9b951c5639601ceed40de50cb10 not found: ID does not exist" containerID="c86cf6476ac1b17552c4a3f13e2833f2c3c9d9b951c5639601ceed40de50cb10"
Sep 11 19:54:33.933122 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.933072    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c86cf6476ac1b17552c4a3f13e2833f2c3c9d9b951c5639601ceed40de50cb10"} err="failed to get container status \"c86cf6476ac1b17552c4a3f13e2833f2c3c9d9b951c5639601ceed40de50cb10\": rpc error: code = NotFound desc = could not find container \"c86cf6476ac1b17552c4a3f13e2833f2c3c9d9b951c5639601ceed40de50cb10\": container with ID starting with c86cf6476ac1b17552c4a3f13e2833f2c3c9d9b951c5639601ceed40de50cb10 not found: ID does not exist"
Sep 11 19:54:33.940675 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.940642    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"]
Sep 11 19:54:33.944225 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:33.944202    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c7766c/e2e-conflict-a-ccaf36e3-kserve-786b8f8b77-jkzdl"]
Sep 11 19:54:35.815191 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:35.815152    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="119033c9-c279-4cd9-abec-a7142451e0e4" path="/var/lib/kubelet/pods/119033c9-c279-4cd9-abec-a7142451e0e4/volumes"
Sep 11 19:54:43.295425 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.295387    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"]
Sep 11 19:54:43.295862 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.295849    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="119033c9-c279-4cd9-abec-a7142451e0e4" containerName="main"
Sep 11 19:54:43.295908 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.295864    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="119033c9-c279-4cd9-abec-a7142451e0e4" containerName="main"
Sep 11 19:54:43.295955 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.295946    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="119033c9-c279-4cd9-abec-a7142451e0e4" containerName="main"
Sep 11 19:54:43.298635 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.298617    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:43.302845 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.302821    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-10cc24\"/\"kube-root-ca.crt\""
Sep 11 19:54:43.303001 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.302877    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-10cc24\"/\"openshift-service-ca.crt\""
Sep 11 19:54:43.303001 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.302940    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-10cc24\"/\"distinct-2-w0-10cc24-kserve-self-signed-certs\""
Sep 11 19:54:43.303001 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.302949    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-10cc24\"/\"default-dockercfg-z9db5\""
Sep 11 19:54:43.314117 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.314090    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"]
Sep 11 19:54:43.404618 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.404335    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-dshm\") pod \"distinct-2-w0-10cc24-kserve-c465b5977-bpngk\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") " pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:43.404618 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.404438    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-model-cache\") pod \"distinct-2-w0-10cc24-kserve-c465b5977-bpngk\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") " pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:43.404618 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.404470    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-home\") pod \"distinct-2-w0-10cc24-kserve-c465b5977-bpngk\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") " pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:43.404618 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.404494    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-tmp-dir\") pod \"distinct-2-w0-10cc24-kserve-c465b5977-bpngk\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") " pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:43.404618 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.404557    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtxlc\" (UniqueName: \"kubernetes.io/projected/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-kube-api-access-qtxlc\") pod \"distinct-2-w0-10cc24-kserve-c465b5977-bpngk\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") " pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:43.404618 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.404621    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-tls-certs\") pod \"distinct-2-w0-10cc24-kserve-c465b5977-bpngk\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") " pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:43.506397 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.506353    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-model-cache\") pod \"distinct-2-w0-10cc24-kserve-c465b5977-bpngk\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") " pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:43.506581 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.506405    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-home\") pod \"distinct-2-w0-10cc24-kserve-c465b5977-bpngk\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") " pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:43.506581 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.506434    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-tmp-dir\") pod \"distinct-2-w0-10cc24-kserve-c465b5977-bpngk\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") " pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:43.506581 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.506506    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtxlc\" (UniqueName: \"kubernetes.io/projected/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-kube-api-access-qtxlc\") pod \"distinct-2-w0-10cc24-kserve-c465b5977-bpngk\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") " pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:43.506581 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.506576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-tls-certs\") pod \"distinct-2-w0-10cc24-kserve-c465b5977-bpngk\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") " pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:43.506816 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.506659    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-dshm\") pod \"distinct-2-w0-10cc24-kserve-c465b5977-bpngk\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") " pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:43.506876 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.506831    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-model-cache\") pod \"distinct-2-w0-10cc24-kserve-c465b5977-bpngk\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") " pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:43.506932 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.506862    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-tmp-dir\") pod \"distinct-2-w0-10cc24-kserve-c465b5977-bpngk\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") " pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:43.507067 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.507042    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-home\") pod \"distinct-2-w0-10cc24-kserve-c465b5977-bpngk\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") " pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:43.509050 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.509016    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-dshm\") pod \"distinct-2-w0-10cc24-kserve-c465b5977-bpngk\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") " pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:43.509447 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.509431    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-tls-certs\") pod \"distinct-2-w0-10cc24-kserve-c465b5977-bpngk\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") " pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:43.517529 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.517494    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtxlc\" (UniqueName: \"kubernetes.io/projected/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-kube-api-access-qtxlc\") pod \"distinct-2-w0-10cc24-kserve-c465b5977-bpngk\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") " pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:43.608521 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.608425    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:43.901388 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.901358    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"]
Sep 11 19:54:43.903060 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:54:43.903033    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ab47224_e82d_4d05_ad6c_bfa34e8dc0da.slice/crio-2781bdde11cfcf14500af33c627a9c8e63bda8e5a43b16fe0cc461f3b2c1fc77 WatchSource:0}: Error finding container 2781bdde11cfcf14500af33c627a9c8e63bda8e5a43b16fe0cc461f3b2c1fc77: Status 404 returned error can't find the container with id 2781bdde11cfcf14500af33c627a9c8e63bda8e5a43b16fe0cc461f3b2c1fc77
Sep 11 19:54:43.983131 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.983090    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk" event={"ID":"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da","Type":"ContainerStarted","Data":"2d36c7f8a4e2aa9f3fac4c21239ad748f8a3023d004eb857b0f3a530bdd1fc3f"}
Sep 11 19:54:43.983285 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:43.983152    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk" event={"ID":"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da","Type":"ContainerStarted","Data":"2781bdde11cfcf14500af33c627a9c8e63bda8e5a43b16fe0cc461f3b2c1fc77"}
Sep 11 19:54:44.016485 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:44.016429    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk" podStartSLOduration=1.016410747 podStartE2EDuration="1.016410747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:54:44.014817374 +0000 UTC m=+1700.631986876" watchObservedRunningTime="2026-09-11 19:54:44.016410747 +0000 UTC m=+1700.633580238"
Sep 11 19:54:53.080111 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:53.080021    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"]
Sep 11 19:54:53.080702 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:53.080278    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw" podUID="a60aea1c-add5-4a57-84f8-b5c0e9ab4006" containerName="maas-api" containerID="cri-o://07da1c87ed2f549a439e0a005be8d79892f012c44fc864593cbb7bcae4583d00" gracePeriod=30
Sep 11 19:54:53.343218 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:53.343147    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"
Sep 11 19:54:53.416860 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:53.416827    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-metrics-certs\") pod \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\" (UID: \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\") "
Sep 11 19:54:53.417066 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:53.416937    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bvqq\" (UniqueName: \"kubernetes.io/projected/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-kube-api-access-8bvqq\") pod \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\" (UID: \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\") "
Sep 11 19:54:53.417066 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:53.416964    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-openshift-service-ca\") pod \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\" (UID: \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\") "
Sep 11 19:54:53.417066 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:53.417010    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-maas-api-tls\") pod \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\" (UID: \"a60aea1c-add5-4a57-84f8-b5c0e9ab4006\") "
Sep 11 19:54:53.417352 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:53.417323    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a60aea1c-add5-4a57-84f8-b5c0e9ab4006" (UID: "a60aea1c-add5-4a57-84f8-b5c0e9ab4006"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:54:53.417529 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:53.417508    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-openshift-service-ca\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:54:53.419203 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:53.419167    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-kube-api-access-8bvqq" (OuterVolumeSpecName: "kube-api-access-8bvqq") pod "a60aea1c-add5-4a57-84f8-b5c0e9ab4006" (UID: "a60aea1c-add5-4a57-84f8-b5c0e9ab4006"). InnerVolumeSpecName "kube-api-access-8bvqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:54:53.419203 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:53.419173    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a60aea1c-add5-4a57-84f8-b5c0e9ab4006" (UID: "a60aea1c-add5-4a57-84f8-b5c0e9ab4006"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:54:53.419348 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:53.419243    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a60aea1c-add5-4a57-84f8-b5c0e9ab4006" (UID: "a60aea1c-add5-4a57-84f8-b5c0e9ab4006"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:54:53.519855 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:53.518484    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bvqq\" (UniqueName: \"kubernetes.io/projected/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-kube-api-access-8bvqq\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:54:53.519855 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:53.518522    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-maas-api-tls\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:54:53.519855 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:53.518536    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a60aea1c-add5-4a57-84f8-b5c0e9ab4006-metrics-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:54:53.609410 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:53.609247    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:53.609410 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:53.609388    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:53.622784 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:53.622755    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:54.050725 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:54.050670    2558 generic.go:356] "Generic (PLEG): container finished" podID="a60aea1c-add5-4a57-84f8-b5c0e9ab4006" containerID="07da1c87ed2f549a439e0a005be8d79892f012c44fc864593cbb7bcae4583d00" exitCode=0
Sep 11 19:54:54.050925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:54.050732    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"
Sep 11 19:54:54.050925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:54.050762    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw" event={"ID":"a60aea1c-add5-4a57-84f8-b5c0e9ab4006","Type":"ContainerDied","Data":"07da1c87ed2f549a439e0a005be8d79892f012c44fc864593cbb7bcae4583d00"}
Sep 11 19:54:54.050925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:54.050826    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw" event={"ID":"a60aea1c-add5-4a57-84f8-b5c0e9ab4006","Type":"ContainerDied","Data":"0da3e87b2d5805f50db0b16bcc4c5a3913f657310f0017cfc2139f4efb2ba8d9"}
Sep 11 19:54:54.050925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:54.050858    2558 scope.go:117] "RemoveContainer" containerID="07da1c87ed2f549a439e0a005be8d79892f012c44fc864593cbb7bcae4583d00"
Sep 11 19:54:54.061956 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:54.061927    2558 scope.go:117] "RemoveContainer" containerID="07da1c87ed2f549a439e0a005be8d79892f012c44fc864593cbb7bcae4583d00"
Sep 11 19:54:54.062318 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:54:54.062275    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07da1c87ed2f549a439e0a005be8d79892f012c44fc864593cbb7bcae4583d00\": container with ID starting with 07da1c87ed2f549a439e0a005be8d79892f012c44fc864593cbb7bcae4583d00 not found: ID does not exist" containerID="07da1c87ed2f549a439e0a005be8d79892f012c44fc864593cbb7bcae4583d00"
Sep 11 19:54:54.062451 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:54.062323    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07da1c87ed2f549a439e0a005be8d79892f012c44fc864593cbb7bcae4583d00"} err="failed to get container status \"07da1c87ed2f549a439e0a005be8d79892f012c44fc864593cbb7bcae4583d00\": rpc error: code = NotFound desc = could not find container \"07da1c87ed2f549a439e0a005be8d79892f012c44fc864593cbb7bcae4583d00\": container with ID starting with 07da1c87ed2f549a439e0a005be8d79892f012c44fc864593cbb7bcae4583d00 not found: ID does not exist"
Sep 11 19:54:54.063446 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:54.063418    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:54:54.069535 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:54.069510    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"]
Sep 11 19:54:54.072673 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:54.072648    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c0517f60-68759fcb64-n4htw"]
Sep 11 19:54:55.814341 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:54:55.814302    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a60aea1c-add5-4a57-84f8-b5c0e9ab4006" path="/var/lib/kubelet/pods/a60aea1c-add5-4a57-84f8-b5c0e9ab4006/volumes"
Sep 11 19:55:07.367836 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.367795    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"]
Sep 11 19:55:07.368524 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.368502    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a60aea1c-add5-4a57-84f8-b5c0e9ab4006" containerName="maas-api"
Sep 11 19:55:07.368570 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.368530    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a60aea1c-add5-4a57-84f8-b5c0e9ab4006" containerName="maas-api"
Sep 11 19:55:07.368686 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.368673    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a60aea1c-add5-4a57-84f8-b5c0e9ab4006" containerName="maas-api"
Sep 11 19:55:07.377637 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.377608    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:07.378957 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.378928    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"]
Sep 11 19:55:07.380338 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.380312    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c7766c\"/\"kube-root-ca.crt\""
Sep 11 19:55:07.380770 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.380752    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c7766c\"/\"default-dockercfg-vxzf9\""
Sep 11 19:55:07.380954 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.380932    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c7766c\"/\"distinct-w4-c7766c-kserve-self-signed-certs\""
Sep 11 19:55:07.381078 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.380970    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c7766c\"/\"openshift-service-ca.crt\""
Sep 11 19:55:07.571925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.571880    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcf8n\" (UniqueName: \"kubernetes.io/projected/e738a24a-b4af-43ec-978f-e3f5fbc11a54-kube-api-access-kcf8n\") pod \"distinct-w4-c7766c-kserve-6d98876f59-5kfmn\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") " pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:07.571925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.571924    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-model-cache\") pod \"distinct-w4-c7766c-kserve-6d98876f59-5kfmn\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") " pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:07.572186 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.572107    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-home\") pod \"distinct-w4-c7766c-kserve-6d98876f59-5kfmn\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") " pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:07.572186 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.572136    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e738a24a-b4af-43ec-978f-e3f5fbc11a54-tls-certs\") pod \"distinct-w4-c7766c-kserve-6d98876f59-5kfmn\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") " pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:07.572315 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.572239    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-dshm\") pod \"distinct-w4-c7766c-kserve-6d98876f59-5kfmn\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") " pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:07.572315 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.572277    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-tmp-dir\") pod \"distinct-w4-c7766c-kserve-6d98876f59-5kfmn\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") " pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:07.673419 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.673381    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-home\") pod \"distinct-w4-c7766c-kserve-6d98876f59-5kfmn\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") " pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:07.673419 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.673419    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e738a24a-b4af-43ec-978f-e3f5fbc11a54-tls-certs\") pod \"distinct-w4-c7766c-kserve-6d98876f59-5kfmn\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") " pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:07.673616 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.673485    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-dshm\") pod \"distinct-w4-c7766c-kserve-6d98876f59-5kfmn\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") " pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:07.673616 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.673520    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-tmp-dir\") pod \"distinct-w4-c7766c-kserve-6d98876f59-5kfmn\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") " pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:07.673616 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.673567    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcf8n\" (UniqueName: \"kubernetes.io/projected/e738a24a-b4af-43ec-978f-e3f5fbc11a54-kube-api-access-kcf8n\") pod \"distinct-w4-c7766c-kserve-6d98876f59-5kfmn\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") " pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:07.673754 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.673614    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-model-cache\") pod \"distinct-w4-c7766c-kserve-6d98876f59-5kfmn\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") " pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:07.673819 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.673794    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-home\") pod \"distinct-w4-c7766c-kserve-6d98876f59-5kfmn\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") " pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:07.673942 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.673917    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-tmp-dir\") pod \"distinct-w4-c7766c-kserve-6d98876f59-5kfmn\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") " pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:07.673998 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.673972    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-model-cache\") pod \"distinct-w4-c7766c-kserve-6d98876f59-5kfmn\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") " pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:07.675883 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.675862    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-dshm\") pod \"distinct-w4-c7766c-kserve-6d98876f59-5kfmn\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") " pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:07.676060 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.676043    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e738a24a-b4af-43ec-978f-e3f5fbc11a54-tls-certs\") pod \"distinct-w4-c7766c-kserve-6d98876f59-5kfmn\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") " pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:07.681941 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.681914    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcf8n\" (UniqueName: \"kubernetes.io/projected/e738a24a-b4af-43ec-978f-e3f5fbc11a54-kube-api-access-kcf8n\") pod \"distinct-w4-c7766c-kserve-6d98876f59-5kfmn\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") " pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:07.688833 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.688807    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:07.962783 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:07.962753    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"]
Sep 11 19:55:07.963612 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:55:07.963590    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode738a24a_b4af_43ec_978f_e3f5fbc11a54.slice/crio-2d1a4e8935738ec6f1c6d68ea522eb3daca255ca7e0feecb01771232228f9d08 WatchSource:0}: Error finding container 2d1a4e8935738ec6f1c6d68ea522eb3daca255ca7e0feecb01771232228f9d08: Status 404 returned error can't find the container with id 2d1a4e8935738ec6f1c6d68ea522eb3daca255ca7e0feecb01771232228f9d08
Sep 11 19:55:08.129944 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:08.129908    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn" event={"ID":"e738a24a-b4af-43ec-978f-e3f5fbc11a54","Type":"ContainerStarted","Data":"4edcd00518f4d895ff3cb88ffe6f73eedcdd3523327f182fd5b37d741242f638"}
Sep 11 19:55:08.130113 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:08.129956    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn" event={"ID":"e738a24a-b4af-43ec-978f-e3f5fbc11a54","Type":"ContainerStarted","Data":"2d1a4e8935738ec6f1c6d68ea522eb3daca255ca7e0feecb01771232228f9d08"}
Sep 11 19:55:08.148073 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:08.148021    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn" podStartSLOduration=1.148004813 podStartE2EDuration="1.148004813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:55:08.145317859 +0000 UTC m=+1724.762487350" watchObservedRunningTime="2026-09-11 19:55:08.148004813 +0000 UTC m=+1724.765174305"
Sep 11 19:55:17.689455 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:17.689416    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:17.689975 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:17.689733    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:17.702100 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:17.702076    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:55:18.195553 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:55:18.195520    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:57:28.622138 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:28.622098    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"]
Sep 11 19:57:28.625817 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:28.625790    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"
Sep 11 19:57:28.628120 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:28.628080    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d88566-serving-cert\""
Sep 11 19:57:28.635621 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:28.635597    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"]
Sep 11 19:57:28.702092 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:28.702054    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a895a8e7-aee0-4585-b288-d275d9c8f2ec-maas-api-tls\") pod \"maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt\" (UID: \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"
Sep 11 19:57:28.702270 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:28.702121    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a895a8e7-aee0-4585-b288-d275d9c8f2ec-openshift-service-ca\") pod \"maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt\" (UID: \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"
Sep 11 19:57:28.702270 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:28.702161    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a895a8e7-aee0-4585-b288-d275d9c8f2ec-metrics-certs\") pod \"maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt\" (UID: \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"
Sep 11 19:57:28.702270 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:28.702251    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb7rd\" (UniqueName: \"kubernetes.io/projected/a895a8e7-aee0-4585-b288-d275d9c8f2ec-kube-api-access-pb7rd\") pod \"maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt\" (UID: \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"
Sep 11 19:57:28.803683 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:28.803649    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a895a8e7-aee0-4585-b288-d275d9c8f2ec-maas-api-tls\") pod \"maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt\" (UID: \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"
Sep 11 19:57:28.803866 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:28.803727    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a895a8e7-aee0-4585-b288-d275d9c8f2ec-openshift-service-ca\") pod \"maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt\" (UID: \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"
Sep 11 19:57:28.803866 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:28.803760    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a895a8e7-aee0-4585-b288-d275d9c8f2ec-metrics-certs\") pod \"maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt\" (UID: \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"
Sep 11 19:57:28.803866 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:28.803823    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb7rd\" (UniqueName: \"kubernetes.io/projected/a895a8e7-aee0-4585-b288-d275d9c8f2ec-kube-api-access-pb7rd\") pod \"maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt\" (UID: \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"
Sep 11 19:57:28.803866 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:57:28.803825    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-serving-cert: secret "maas-api-e2e-api-a-d88566-serving-cert" not found
Sep 11 19:57:28.804013 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:57:28.803926    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a895a8e7-aee0-4585-b288-d275d9c8f2ec-maas-api-tls podName:a895a8e7-aee0-4585-b288-d275d9c8f2ec nodeName:}" failed. No retries permitted until 2026-09-11 19:57:29.303904282 +0000 UTC m=+1865.921073767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a895a8e7-aee0-4585-b288-d275d9c8f2ec-maas-api-tls") pod "maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt" (UID: "a895a8e7-aee0-4585-b288-d275d9c8f2ec") : secret "maas-api-e2e-api-a-d88566-serving-cert" not found
Sep 11 19:57:28.804396 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:28.804376    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a895a8e7-aee0-4585-b288-d275d9c8f2ec-openshift-service-ca\") pod \"maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt\" (UID: \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"
Sep 11 19:57:28.806454 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:28.806429    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a895a8e7-aee0-4585-b288-d275d9c8f2ec-metrics-certs\") pod \"maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt\" (UID: \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"
Sep 11 19:57:28.814990 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:28.814965    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb7rd\" (UniqueName: \"kubernetes.io/projected/a895a8e7-aee0-4585-b288-d275d9c8f2ec-kube-api-access-pb7rd\") pod \"maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt\" (UID: \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"
Sep 11 19:57:29.308889 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:29.308844    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a895a8e7-aee0-4585-b288-d275d9c8f2ec-maas-api-tls\") pod \"maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt\" (UID: \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"
Sep 11 19:57:29.311975 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:29.311920    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a895a8e7-aee0-4585-b288-d275d9c8f2ec-maas-api-tls\") pod \"maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt\" (UID: \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"
Sep 11 19:57:29.539188 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:29.539147    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"
Sep 11 19:57:29.823250 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:57:29.823213    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda895a8e7_aee0_4585_b288_d275d9c8f2ec.slice/crio-1b76c225898668c4083b28fe48ede84643241c25cf7728f26e565bd0b53597fd WatchSource:0}: Error finding container 1b76c225898668c4083b28fe48ede84643241c25cf7728f26e565bd0b53597fd: Status 404 returned error can't find the container with id 1b76c225898668c4083b28fe48ede84643241c25cf7728f26e565bd0b53597fd
Sep 11 19:57:29.824552 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:29.824438    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"]
Sep 11 19:57:29.825099 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:29.825081    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 19:57:29.917981 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:29.917936    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt" event={"ID":"a895a8e7-aee0-4585-b288-d275d9c8f2ec","Type":"ContainerStarted","Data":"1b76c225898668c4083b28fe48ede84643241c25cf7728f26e565bd0b53597fd"}
Sep 11 19:57:30.925270 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:30.925234    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt" event={"ID":"a895a8e7-aee0-4585-b288-d275d9c8f2ec","Type":"ContainerStarted","Data":"bce4fa9a92cba9b44413ee3fb6a03761df644b6fd04525c866500d66843161af"}
Sep 11 19:57:30.925674 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:30.925431    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"
Sep 11 19:57:30.942279 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:30.942224    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt" podStartSLOduration=2.494746431 podStartE2EDuration="2.942207854s" totalImagesPullingTime="447.461423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:57:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:57:30.940759964 +0000 UTC m=+1867.557929455" watchObservedRunningTime="2026-09-11 19:57:30.942207854 +0000 UTC m=+1867.559377345"
Sep 11 19:57:36.936760 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:36.936725    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"
Sep 11 19:57:40.281194 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.281150    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"]
Sep 11 19:57:40.285107 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.285088    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:40.290921 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.290898    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-10cc24\"/\"embedding-w0-10cc24-kserve-self-signed-certs\""
Sep 11 19:57:40.306380 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.306343    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"]
Sep 11 19:57:40.322935 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.322892    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-tls-certs\") pod \"embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") " pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:40.323102 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.323027    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-tmp-dir\") pod \"embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") " pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:40.323102 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.323068    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-dshm\") pod \"embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") " pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:40.323183 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.323138    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-model-cache\") pod \"embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") " pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:40.323227 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.323187    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv4jv\" (UniqueName: \"kubernetes.io/projected/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-kube-api-access-mv4jv\") pod \"embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") " pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:40.323227 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.323219    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-home\") pod \"embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") " pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:40.424262 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.424224    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-tmp-dir\") pod \"embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") " pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:40.424262 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.424265    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-dshm\") pod \"embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") " pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:40.424512 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.424314    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-model-cache\") pod \"embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") " pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:40.424512 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.424344    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv4jv\" (UniqueName: \"kubernetes.io/projected/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-kube-api-access-mv4jv\") pod \"embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") " pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:40.424512 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.424367    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-home\") pod \"embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") " pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:40.424512 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.424461    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-tls-certs\") pod \"embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") " pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:40.424802 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.424749    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-tmp-dir\") pod \"embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") " pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:40.424944 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.424917    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-home\") pod \"embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") " pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:40.425044 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.425025    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-model-cache\") pod \"embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") " pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:40.426786 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.426760    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-dshm\") pod \"embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") " pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:40.427047 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.427027    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-tls-certs\") pod \"embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") " pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:40.433311 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.433283    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv4jv\" (UniqueName: \"kubernetes.io/projected/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-kube-api-access-mv4jv\") pod \"embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") " pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:40.595028 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.594937    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:40.887893 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.876130    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"]
Sep 11 19:57:40.887893 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:57:40.877895    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3ab7c50_67ce_4b5d_a86c_5132e63f7188.slice/crio-94d461e631a0f9a7ae1ccf25a8b4d6fa841091928352e32e8606b255108fa743 WatchSource:0}: Error finding container 94d461e631a0f9a7ae1ccf25a8b4d6fa841091928352e32e8606b255108fa743: Status 404 returned error can't find the container with id 94d461e631a0f9a7ae1ccf25a8b4d6fa841091928352e32e8606b255108fa743
Sep 11 19:57:40.980081 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.980044    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9" event={"ID":"a3ab7c50-67ce-4b5d-a86c-5132e63f7188","Type":"ContainerStarted","Data":"7c206e4852195a9977648c0eb05fdb9657a8fe9bed91c51b5bec57f2abda36fa"}
Sep 11 19:57:40.980258 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:40.980090    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9" event={"ID":"a3ab7c50-67ce-4b5d-a86c-5132e63f7188","Type":"ContainerStarted","Data":"94d461e631a0f9a7ae1ccf25a8b4d6fa841091928352e32e8606b255108fa743"}
Sep 11 19:57:41.004328 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:41.004261    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9" podStartSLOduration=1.0042429 podStartE2EDuration="1.0042429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:57:41.00123526 +0000 UTC m=+1877.618404753" watchObservedRunningTime="2026-09-11 19:57:41.0042429 +0000 UTC m=+1877.621412391"
Sep 11 19:57:50.595696 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:50.595602    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:50.595696 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:50.595664    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:50.608272 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:50.608245    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:57:51.046273 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:57:51.046246    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:58:03.802683 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:03.802649    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"]
Sep 11 19:58:03.806623 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:03.806599    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"
Sep 11 19:58:03.808941 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:03.808920    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6b3e57-serving-cert\""
Sep 11 19:58:03.817658 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:03.817633    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"]
Sep 11 19:58:03.885162 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:03.885115    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2310873-9576-4655-ae15-faa08ed4fd7c-maas-api-tls\") pod \"maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h\" (UID: \"e2310873-9576-4655-ae15-faa08ed4fd7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"
Sep 11 19:58:03.885380 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:03.885180    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vl26\" (UniqueName: \"kubernetes.io/projected/e2310873-9576-4655-ae15-faa08ed4fd7c-kube-api-access-4vl26\") pod \"maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h\" (UID: \"e2310873-9576-4655-ae15-faa08ed4fd7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"
Sep 11 19:58:03.885380 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:03.885228    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2310873-9576-4655-ae15-faa08ed4fd7c-metrics-certs\") pod \"maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h\" (UID: \"e2310873-9576-4655-ae15-faa08ed4fd7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"
Sep 11 19:58:03.885496 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:03.885435    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2310873-9576-4655-ae15-faa08ed4fd7c-openshift-service-ca\") pod \"maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h\" (UID: \"e2310873-9576-4655-ae15-faa08ed4fd7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"
Sep 11 19:58:03.986348 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:03.986314    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2310873-9576-4655-ae15-faa08ed4fd7c-maas-api-tls\") pod \"maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h\" (UID: \"e2310873-9576-4655-ae15-faa08ed4fd7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"
Sep 11 19:58:03.986570 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:03.986469    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vl26\" (UniqueName: \"kubernetes.io/projected/e2310873-9576-4655-ae15-faa08ed4fd7c-kube-api-access-4vl26\") pod \"maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h\" (UID: \"e2310873-9576-4655-ae15-faa08ed4fd7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"
Sep 11 19:58:03.986570 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:03.986521    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2310873-9576-4655-ae15-faa08ed4fd7c-metrics-certs\") pod \"maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h\" (UID: \"e2310873-9576-4655-ae15-faa08ed4fd7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"
Sep 11 19:58:03.986570 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:03.986560    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2310873-9576-4655-ae15-faa08ed4fd7c-openshift-service-ca\") pod \"maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h\" (UID: \"e2310873-9576-4655-ae15-faa08ed4fd7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"
Sep 11 19:58:03.988237 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:03.987525    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2310873-9576-4655-ae15-faa08ed4fd7c-openshift-service-ca\") pod \"maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h\" (UID: \"e2310873-9576-4655-ae15-faa08ed4fd7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"
Sep 11 19:58:03.996166 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:03.996106    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2310873-9576-4655-ae15-faa08ed4fd7c-maas-api-tls\") pod \"maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h\" (UID: \"e2310873-9576-4655-ae15-faa08ed4fd7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"
Sep 11 19:58:03.996344 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:03.996301    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2310873-9576-4655-ae15-faa08ed4fd7c-metrics-certs\") pod \"maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h\" (UID: \"e2310873-9576-4655-ae15-faa08ed4fd7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"
Sep 11 19:58:04.002067 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:04.002005    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vl26\" (UniqueName: \"kubernetes.io/projected/e2310873-9576-4655-ae15-faa08ed4fd7c-kube-api-access-4vl26\") pod \"maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h\" (UID: \"e2310873-9576-4655-ae15-faa08ed4fd7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"
Sep 11 19:58:04.120042 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:04.119946    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"
Sep 11 19:58:04.398813 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:04.398784    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"]
Sep 11 19:58:04.399257 ip-10-0-138-43 kubenswrapper[2558]: W0911 19:58:04.399230    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2310873_9576_4655_ae15_faa08ed4fd7c.slice/crio-77f241162091e8b125d2d077d0049cdd3e65f0e2d6bcbc0aa783804514253df5 WatchSource:0}: Error finding container 77f241162091e8b125d2d077d0049cdd3e65f0e2d6bcbc0aa783804514253df5: Status 404 returned error can't find the container with id 77f241162091e8b125d2d077d0049cdd3e65f0e2d6bcbc0aa783804514253df5
Sep 11 19:58:05.123482 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:05.123379    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h" event={"ID":"e2310873-9576-4655-ae15-faa08ed4fd7c","Type":"ContainerStarted","Data":"c9925e42a83e4b9d5d2ff29d22e653def5f418aec69cc8d5abb8366b5806e5ca"}
Sep 11 19:58:05.123482 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:05.123436    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h" event={"ID":"e2310873-9576-4655-ae15-faa08ed4fd7c","Type":"ContainerStarted","Data":"77f241162091e8b125d2d077d0049cdd3e65f0e2d6bcbc0aa783804514253df5"}
Sep 11 19:58:05.123482 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:05.123464    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"
Sep 11 19:58:05.144847 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:05.144788    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h" podStartSLOduration=1.838846964 podStartE2EDuration="2.144770302s" totalImagesPullingTime="305.923338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:58:05.141661111 +0000 UTC m=+1901.758830603" watchObservedRunningTime="2026-09-11 19:58:05.144770302 +0000 UTC m=+1901.761939792"
Sep 11 19:58:11.134621 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:11.134588    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"
Sep 11 19:58:46.415515 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:46.415477    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"]
Sep 11 19:58:46.416141 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:46.415697    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt" podUID="a895a8e7-aee0-4585-b288-d275d9c8f2ec" containerName="maas-api" containerID="cri-o://bce4fa9a92cba9b44413ee3fb6a03761df644b6fd04525c866500d66843161af" gracePeriod=30
Sep 11 19:58:46.683582 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:46.683520    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"
Sep 11 19:58:46.746256 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:46.746213    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a895a8e7-aee0-4585-b288-d275d9c8f2ec-metrics-certs\") pod \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\" (UID: \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\") "
Sep 11 19:58:46.746511 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:46.746403    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb7rd\" (UniqueName: \"kubernetes.io/projected/a895a8e7-aee0-4585-b288-d275d9c8f2ec-kube-api-access-pb7rd\") pod \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\" (UID: \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\") "
Sep 11 19:58:46.746511 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:46.746436    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a895a8e7-aee0-4585-b288-d275d9c8f2ec-maas-api-tls\") pod \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\" (UID: \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\") "
Sep 11 19:58:46.746511 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:46.746462    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a895a8e7-aee0-4585-b288-d275d9c8f2ec-openshift-service-ca\") pod \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\" (UID: \"a895a8e7-aee0-4585-b288-d275d9c8f2ec\") "
Sep 11 19:58:46.747015 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:46.746973    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a895a8e7-aee0-4585-b288-d275d9c8f2ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a895a8e7-aee0-4585-b288-d275d9c8f2ec" (UID: "a895a8e7-aee0-4585-b288-d275d9c8f2ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:58:46.748741 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:46.748681    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a895a8e7-aee0-4585-b288-d275d9c8f2ec-kube-api-access-pb7rd" (OuterVolumeSpecName: "kube-api-access-pb7rd") pod "a895a8e7-aee0-4585-b288-d275d9c8f2ec" (UID: "a895a8e7-aee0-4585-b288-d275d9c8f2ec"). InnerVolumeSpecName "kube-api-access-pb7rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:58:46.749033 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:46.749011    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a895a8e7-aee0-4585-b288-d275d9c8f2ec-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a895a8e7-aee0-4585-b288-d275d9c8f2ec" (UID: "a895a8e7-aee0-4585-b288-d275d9c8f2ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:58:46.749105 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:46.749025    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a895a8e7-aee0-4585-b288-d275d9c8f2ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a895a8e7-aee0-4585-b288-d275d9c8f2ec" (UID: "a895a8e7-aee0-4585-b288-d275d9c8f2ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:58:46.849564 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:46.849468    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb7rd\" (UniqueName: \"kubernetes.io/projected/a895a8e7-aee0-4585-b288-d275d9c8f2ec-kube-api-access-pb7rd\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:58:46.849564 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:46.849507    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a895a8e7-aee0-4585-b288-d275d9c8f2ec-maas-api-tls\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:58:46.849564 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:46.849520    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a895a8e7-aee0-4585-b288-d275d9c8f2ec-openshift-service-ca\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:58:46.849564 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:46.849532    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a895a8e7-aee0-4585-b288-d275d9c8f2ec-metrics-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:58:47.371382 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:47.371348    2558 generic.go:356] "Generic (PLEG): container finished" podID="a895a8e7-aee0-4585-b288-d275d9c8f2ec" containerID="bce4fa9a92cba9b44413ee3fb6a03761df644b6fd04525c866500d66843161af" exitCode=0
Sep 11 19:58:47.371594 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:47.371393    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"
Sep 11 19:58:47.371594 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:47.371397    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt" event={"ID":"a895a8e7-aee0-4585-b288-d275d9c8f2ec","Type":"ContainerDied","Data":"bce4fa9a92cba9b44413ee3fb6a03761df644b6fd04525c866500d66843161af"}
Sep 11 19:58:47.371594 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:47.371538    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt" event={"ID":"a895a8e7-aee0-4585-b288-d275d9c8f2ec","Type":"ContainerDied","Data":"1b76c225898668c4083b28fe48ede84643241c25cf7728f26e565bd0b53597fd"}
Sep 11 19:58:47.371594 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:47.371576    2558 scope.go:117] "RemoveContainer" containerID="bce4fa9a92cba9b44413ee3fb6a03761df644b6fd04525c866500d66843161af"
Sep 11 19:58:47.381269 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:47.381246    2558 scope.go:117] "RemoveContainer" containerID="bce4fa9a92cba9b44413ee3fb6a03761df644b6fd04525c866500d66843161af"
Sep 11 19:58:47.381545 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:58:47.381526    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce4fa9a92cba9b44413ee3fb6a03761df644b6fd04525c866500d66843161af\": container with ID starting with bce4fa9a92cba9b44413ee3fb6a03761df644b6fd04525c866500d66843161af not found: ID does not exist" containerID="bce4fa9a92cba9b44413ee3fb6a03761df644b6fd04525c866500d66843161af"
Sep 11 19:58:47.381616 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:47.381551    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce4fa9a92cba9b44413ee3fb6a03761df644b6fd04525c866500d66843161af"} err="failed to get container status \"bce4fa9a92cba9b44413ee3fb6a03761df644b6fd04525c866500d66843161af\": rpc error: code = NotFound desc = could not find container \"bce4fa9a92cba9b44413ee3fb6a03761df644b6fd04525c866500d66843161af\": container with ID starting with bce4fa9a92cba9b44413ee3fb6a03761df644b6fd04525c866500d66843161af not found: ID does not exist"
Sep 11 19:58:47.398437 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:47.398405    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"]
Sep 11 19:58:47.402535 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:47.402503    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d88566-57f789f6b8-bp7lt"]
Sep 11 19:58:47.815345 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:58:47.815310    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a895a8e7-aee0-4585-b288-d275d9c8f2ec" path="/var/lib/kubelet/pods/a895a8e7-aee0-4585-b288-d275d9c8f2ec/volumes"
Sep 11 19:59:04.647762 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:04.647725    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"]
Sep 11 19:59:04.666829 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:04.666799    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"]
Sep 11 19:59:04.957638 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:04.957599    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"]
Sep 11 19:59:12.390885 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:12.390849    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"]
Sep 11 19:59:12.391408 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:12.391105    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h" podUID="e2310873-9576-4655-ae15-faa08ed4fd7c" containerName="maas-api" containerID="cri-o://c9925e42a83e4b9d5d2ff29d22e653def5f418aec69cc8d5abb8366b5806e5ca" gracePeriod=30
Sep 11 19:59:12.533478 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:12.533439    2558 generic.go:356] "Generic (PLEG): container finished" podID="e2310873-9576-4655-ae15-faa08ed4fd7c" containerID="c9925e42a83e4b9d5d2ff29d22e653def5f418aec69cc8d5abb8366b5806e5ca" exitCode=0
Sep 11 19:59:12.533664 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:12.533507    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h" event={"ID":"e2310873-9576-4655-ae15-faa08ed4fd7c","Type":"ContainerDied","Data":"c9925e42a83e4b9d5d2ff29d22e653def5f418aec69cc8d5abb8366b5806e5ca"}
Sep 11 19:59:12.650556 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:12.650482    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"
Sep 11 19:59:12.737582 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:12.737533    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2310873-9576-4655-ae15-faa08ed4fd7c-metrics-certs\") pod \"e2310873-9576-4655-ae15-faa08ed4fd7c\" (UID: \"e2310873-9576-4655-ae15-faa08ed4fd7c\") "
Sep 11 19:59:12.737810 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:12.737630    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vl26\" (UniqueName: \"kubernetes.io/projected/e2310873-9576-4655-ae15-faa08ed4fd7c-kube-api-access-4vl26\") pod \"e2310873-9576-4655-ae15-faa08ed4fd7c\" (UID: \"e2310873-9576-4655-ae15-faa08ed4fd7c\") "
Sep 11 19:59:12.737810 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:12.737676    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2310873-9576-4655-ae15-faa08ed4fd7c-openshift-service-ca\") pod \"e2310873-9576-4655-ae15-faa08ed4fd7c\" (UID: \"e2310873-9576-4655-ae15-faa08ed4fd7c\") "
Sep 11 19:59:12.737810 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:12.737789    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2310873-9576-4655-ae15-faa08ed4fd7c-maas-api-tls\") pod \"e2310873-9576-4655-ae15-faa08ed4fd7c\" (UID: \"e2310873-9576-4655-ae15-faa08ed4fd7c\") "
Sep 11 19:59:12.738177 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:12.738136    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2310873-9576-4655-ae15-faa08ed4fd7c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e2310873-9576-4655-ae15-faa08ed4fd7c" (UID: "e2310873-9576-4655-ae15-faa08ed4fd7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:59:12.740034 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:12.739988    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2310873-9576-4655-ae15-faa08ed4fd7c-kube-api-access-4vl26" (OuterVolumeSpecName: "kube-api-access-4vl26") pod "e2310873-9576-4655-ae15-faa08ed4fd7c" (UID: "e2310873-9576-4655-ae15-faa08ed4fd7c"). InnerVolumeSpecName "kube-api-access-4vl26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:59:12.740261 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:12.740235    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2310873-9576-4655-ae15-faa08ed4fd7c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e2310873-9576-4655-ae15-faa08ed4fd7c" (UID: "e2310873-9576-4655-ae15-faa08ed4fd7c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:59:12.740337 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:12.740270    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2310873-9576-4655-ae15-faa08ed4fd7c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e2310873-9576-4655-ae15-faa08ed4fd7c" (UID: "e2310873-9576-4655-ae15-faa08ed4fd7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:59:12.839717 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:12.839665    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2310873-9576-4655-ae15-faa08ed4fd7c-openshift-service-ca\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:12.839717 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:12.839701    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2310873-9576-4655-ae15-faa08ed4fd7c-maas-api-tls\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:12.839717 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:12.839724    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2310873-9576-4655-ae15-faa08ed4fd7c-metrics-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:12.839970 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:12.839733    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vl26\" (UniqueName: \"kubernetes.io/projected/e2310873-9576-4655-ae15-faa08ed4fd7c-kube-api-access-4vl26\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:13.540487 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:13.540456    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"
Sep 11 19:59:13.540933 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:13.540453    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h" event={"ID":"e2310873-9576-4655-ae15-faa08ed4fd7c","Type":"ContainerDied","Data":"77f241162091e8b125d2d077d0049cdd3e65f0e2d6bcbc0aa783804514253df5"}
Sep 11 19:59:13.540933 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:13.540599    2558 scope.go:117] "RemoveContainer" containerID="c9925e42a83e4b9d5d2ff29d22e653def5f418aec69cc8d5abb8366b5806e5ca"
Sep 11 19:59:13.563218 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:13.563183    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"]
Sep 11 19:59:13.567333 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:13.567305    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6b3e57-7cdcfdf496-x8h4h"]
Sep 11 19:59:13.673870 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:13.673832    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"]
Sep 11 19:59:13.674768 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:13.674155    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp" podUID="a1086d50-c963-4705-92da-51a1c144de7b" containerName="maas-api" containerID="cri-o://470d86a243a924640afdf4686d06a4c78d049333618f4cc87a4d27aee906648e" gracePeriod=30
Sep 11 19:59:13.815949 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:13.815865    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2310873-9576-4655-ae15-faa08ed4fd7c" path="/var/lib/kubelet/pods/e2310873-9576-4655-ae15-faa08ed4fd7c/volumes"
Sep 11 19:59:14.141881 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.141853    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"
Sep 11 19:59:14.253791 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.253758    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f96l8\" (UniqueName: \"kubernetes.io/projected/a1086d50-c963-4705-92da-51a1c144de7b-kube-api-access-f96l8\") pod \"a1086d50-c963-4705-92da-51a1c144de7b\" (UID: \"a1086d50-c963-4705-92da-51a1c144de7b\") "
Sep 11 19:59:14.253972 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.253805    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1086d50-c963-4705-92da-51a1c144de7b-openshift-service-ca\") pod \"a1086d50-c963-4705-92da-51a1c144de7b\" (UID: \"a1086d50-c963-4705-92da-51a1c144de7b\") "
Sep 11 19:59:14.253972 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.253883    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1086d50-c963-4705-92da-51a1c144de7b-maas-api-tls\") pod \"a1086d50-c963-4705-92da-51a1c144de7b\" (UID: \"a1086d50-c963-4705-92da-51a1c144de7b\") "
Sep 11 19:59:14.253972 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.253912    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1086d50-c963-4705-92da-51a1c144de7b-metrics-certs\") pod \"a1086d50-c963-4705-92da-51a1c144de7b\" (UID: \"a1086d50-c963-4705-92da-51a1c144de7b\") "
Sep 11 19:59:14.254267 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.254237    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1086d50-c963-4705-92da-51a1c144de7b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a1086d50-c963-4705-92da-51a1c144de7b" (UID: "a1086d50-c963-4705-92da-51a1c144de7b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:59:14.256020 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.255976    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1086d50-c963-4705-92da-51a1c144de7b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a1086d50-c963-4705-92da-51a1c144de7b" (UID: "a1086d50-c963-4705-92da-51a1c144de7b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:59:14.256134 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.256063    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1086d50-c963-4705-92da-51a1c144de7b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a1086d50-c963-4705-92da-51a1c144de7b" (UID: "a1086d50-c963-4705-92da-51a1c144de7b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:59:14.256134 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.256087    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1086d50-c963-4705-92da-51a1c144de7b-kube-api-access-f96l8" (OuterVolumeSpecName: "kube-api-access-f96l8") pod "a1086d50-c963-4705-92da-51a1c144de7b" (UID: "a1086d50-c963-4705-92da-51a1c144de7b"). InnerVolumeSpecName "kube-api-access-f96l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:59:14.355419 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.355386    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1086d50-c963-4705-92da-51a1c144de7b-maas-api-tls\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:14.355419 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.355417    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1086d50-c963-4705-92da-51a1c144de7b-metrics-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:14.355419 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.355426    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f96l8\" (UniqueName: \"kubernetes.io/projected/a1086d50-c963-4705-92da-51a1c144de7b-kube-api-access-f96l8\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:14.355666 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.355435    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1086d50-c963-4705-92da-51a1c144de7b-openshift-service-ca\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:14.549044 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.549011    2558 generic.go:356] "Generic (PLEG): container finished" podID="a1086d50-c963-4705-92da-51a1c144de7b" containerID="470d86a243a924640afdf4686d06a4c78d049333618f4cc87a4d27aee906648e" exitCode=0
Sep 11 19:59:14.549548 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.549095    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"
Sep 11 19:59:14.549548 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.549104    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp" event={"ID":"a1086d50-c963-4705-92da-51a1c144de7b","Type":"ContainerDied","Data":"470d86a243a924640afdf4686d06a4c78d049333618f4cc87a4d27aee906648e"}
Sep 11 19:59:14.549548 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.549154    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp" event={"ID":"a1086d50-c963-4705-92da-51a1c144de7b","Type":"ContainerDied","Data":"dcc0cc16a403f99cfc715c1f1bf493a746853c976120eb3e252e8642feedd81e"}
Sep 11 19:59:14.549548 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.549176    2558 scope.go:117] "RemoveContainer" containerID="470d86a243a924640afdf4686d06a4c78d049333618f4cc87a4d27aee906648e"
Sep 11 19:59:14.559460 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.559439    2558 scope.go:117] "RemoveContainer" containerID="470d86a243a924640afdf4686d06a4c78d049333618f4cc87a4d27aee906648e"
Sep 11 19:59:14.559810 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:59:14.559788    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"470d86a243a924640afdf4686d06a4c78d049333618f4cc87a4d27aee906648e\": container with ID starting with 470d86a243a924640afdf4686d06a4c78d049333618f4cc87a4d27aee906648e not found: ID does not exist" containerID="470d86a243a924640afdf4686d06a4c78d049333618f4cc87a4d27aee906648e"
Sep 11 19:59:14.559898 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.559819    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470d86a243a924640afdf4686d06a4c78d049333618f4cc87a4d27aee906648e"} err="failed to get container status \"470d86a243a924640afdf4686d06a4c78d049333618f4cc87a4d27aee906648e\": rpc error: code = NotFound desc = could not find container \"470d86a243a924640afdf4686d06a4c78d049333618f4cc87a4d27aee906648e\": container with ID starting with 470d86a243a924640afdf4686d06a4c78d049333618f4cc87a4d27aee906648e not found: ID does not exist"
Sep 11 19:59:14.588283 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.588245    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"]
Sep 11 19:59:14.593792 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:14.593763    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-c7766c-7944b4598f-mlkdp"]
Sep 11 19:59:15.814912 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:15.814872    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1086d50-c963-4705-92da-51a1c144de7b" path="/var/lib/kubelet/pods/a1086d50-c963-4705-92da-51a1c144de7b/volumes"
Sep 11 19:59:17.348162 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.346938    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj"]
Sep 11 19:59:17.348675 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.348229    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj" podUID="8e865ae0-9dab-49b1-ad48-732afc99bfff" containerName="payload-processing" containerID="cri-o://5ab2dd185aae09ca63d4b25e87896564f6d1abf44e6a6c70ba2cd3461e84d0ef" gracePeriod=30
Sep 11 19:59:17.357503 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.355416    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn"]
Sep 11 19:59:17.357503 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.356462    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn" podUID="6b95b4ef-0d68-4e4a-a326-fd35bf643b09" containerName="payload-pre-processing" containerID="cri-o://669b867329fc39388c96e7586727651681cb4d45bf1e65ed7667b14dd61fd364" gracePeriod=30
Sep 11 19:59:17.570333 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.570295    2558 generic.go:356] "Generic (PLEG): container finished" podID="6b95b4ef-0d68-4e4a-a326-fd35bf643b09" containerID="669b867329fc39388c96e7586727651681cb4d45bf1e65ed7667b14dd61fd364" exitCode=0
Sep 11 19:59:17.570591 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.570454    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn" event={"ID":"6b95b4ef-0d68-4e4a-a326-fd35bf643b09","Type":"ContainerDied","Data":"669b867329fc39388c96e7586727651681cb4d45bf1e65ed7667b14dd61fd364"}
Sep 11 19:59:17.573045 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.572905    2558 generic.go:356] "Generic (PLEG): container finished" podID="8e865ae0-9dab-49b1-ad48-732afc99bfff" containerID="5ab2dd185aae09ca63d4b25e87896564f6d1abf44e6a6c70ba2cd3461e84d0ef" exitCode=0
Sep 11 19:59:17.573045 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.572972    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj" event={"ID":"8e865ae0-9dab-49b1-ad48-732afc99bfff","Type":"ContainerDied","Data":"5ab2dd185aae09ca63d4b25e87896564f6d1abf44e6a6c70ba2cd3461e84d0ef"}
Sep 11 19:59:17.688418 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.688374    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj"
Sep 11 19:59:17.693782 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.693754    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn"
Sep 11 19:59:17.794357 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.794322    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f59v4\" (UniqueName: \"kubernetes.io/projected/6b95b4ef-0d68-4e4a-a326-fd35bf643b09-kube-api-access-f59v4\") pod \"6b95b4ef-0d68-4e4a-a326-fd35bf643b09\" (UID: \"6b95b4ef-0d68-4e4a-a326-fd35bf643b09\") "
Sep 11 19:59:17.794551 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.794415    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e865ae0-9dab-49b1-ad48-732afc99bfff-plugins-config-volume\") pod \"8e865ae0-9dab-49b1-ad48-732afc99bfff\" (UID: \"8e865ae0-9dab-49b1-ad48-732afc99bfff\") "
Sep 11 19:59:17.794551 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.794463    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tdkx\" (UniqueName: \"kubernetes.io/projected/8e865ae0-9dab-49b1-ad48-732afc99bfff-kube-api-access-8tdkx\") pod \"8e865ae0-9dab-49b1-ad48-732afc99bfff\" (UID: \"8e865ae0-9dab-49b1-ad48-732afc99bfff\") "
Sep 11 19:59:17.794551 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.794490    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b95b4ef-0d68-4e4a-a326-fd35bf643b09-plugins-config-volume\") pod \"6b95b4ef-0d68-4e4a-a326-fd35bf643b09\" (UID: \"6b95b4ef-0d68-4e4a-a326-fd35bf643b09\") "
Sep 11 19:59:17.794876 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.794846    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e865ae0-9dab-49b1-ad48-732afc99bfff-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e865ae0-9dab-49b1-ad48-732afc99bfff" (UID: "8e865ae0-9dab-49b1-ad48-732afc99bfff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:59:17.794975 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.794929    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b95b4ef-0d68-4e4a-a326-fd35bf643b09-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b95b4ef-0d68-4e4a-a326-fd35bf643b09" (UID: "6b95b4ef-0d68-4e4a-a326-fd35bf643b09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:59:17.795034 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.795009    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e865ae0-9dab-49b1-ad48-732afc99bfff-plugins-config-volume\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:17.796616 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.796592    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e865ae0-9dab-49b1-ad48-732afc99bfff-kube-api-access-8tdkx" (OuterVolumeSpecName: "kube-api-access-8tdkx") pod "8e865ae0-9dab-49b1-ad48-732afc99bfff" (UID: "8e865ae0-9dab-49b1-ad48-732afc99bfff"). InnerVolumeSpecName "kube-api-access-8tdkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:59:17.796691 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.796623    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b95b4ef-0d68-4e4a-a326-fd35bf643b09-kube-api-access-f59v4" (OuterVolumeSpecName: "kube-api-access-f59v4") pod "6b95b4ef-0d68-4e4a-a326-fd35bf643b09" (UID: "6b95b4ef-0d68-4e4a-a326-fd35bf643b09"). InnerVolumeSpecName "kube-api-access-f59v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:59:17.896590 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.896491    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f59v4\" (UniqueName: \"kubernetes.io/projected/6b95b4ef-0d68-4e4a-a326-fd35bf643b09-kube-api-access-f59v4\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:17.896590 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.896523    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tdkx\" (UniqueName: \"kubernetes.io/projected/8e865ae0-9dab-49b1-ad48-732afc99bfff-kube-api-access-8tdkx\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:17.896590 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:17.896533    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b95b4ef-0d68-4e4a-a326-fd35bf643b09-plugins-config-volume\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:18.580046 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:18.580013    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj"
Sep 11 19:59:18.580505 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:18.580046    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj" event={"ID":"8e865ae0-9dab-49b1-ad48-732afc99bfff","Type":"ContainerDied","Data":"da37fa707c5dc42ceb355159b74be7674b0dab4a53d1ad3e7b38ff7092aee0ca"}
Sep 11 19:59:18.580505 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:18.580110    2558 scope.go:117] "RemoveContainer" containerID="5ab2dd185aae09ca63d4b25e87896564f6d1abf44e6a6c70ba2cd3461e84d0ef"
Sep 11 19:59:18.581556 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:18.581480    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn" event={"ID":"6b95b4ef-0d68-4e4a-a326-fd35bf643b09","Type":"ContainerDied","Data":"4e9d17051e16cf4f9fc7944d27be41cc8148a499d361145d2a4d177f45e79956"}
Sep 11 19:59:18.581556 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:18.581521    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn"
Sep 11 19:59:18.592592 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:18.592566    2558 scope.go:117] "RemoveContainer" containerID="669b867329fc39388c96e7586727651681cb4d45bf1e65ed7667b14dd61fd364"
Sep 11 19:59:18.602752 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:18.602725    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn"]
Sep 11 19:59:18.607188 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:18.607163    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-10cc24-695b775c54-7zksn"]
Sep 11 19:59:18.618477 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:18.618451    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj"]
Sep 11 19:59:18.623006 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:18.622982    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-10cc24-75b4bf6dc7-67bzj"]
Sep 11 19:59:19.667806 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:19.667760    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk" podUID="6ab47224-e82d-4d05-ad6c-bfa34e8dc0da" containerName="main" containerID="cri-o://2d36c7f8a4e2aa9f3fac4c21239ad748f8a3023d004eb857b0f3a530bdd1fc3f" gracePeriod=45
Sep 11 19:59:19.686749 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:19.686686    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9" podUID="a3ab7c50-67ce-4b5d-a86c-5132e63f7188" containerName="main" containerID="cri-o://7c206e4852195a9977648c0eb05fdb9657a8fe9bed91c51b5bec57f2abda36fa" gracePeriod=45
Sep 11 19:59:19.816770 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:19.816737    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b95b4ef-0d68-4e4a-a326-fd35bf643b09" path="/var/lib/kubelet/pods/6b95b4ef-0d68-4e4a-a326-fd35bf643b09/volumes"
Sep 11 19:59:19.817160 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:19.817143    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e865ae0-9dab-49b1-ad48-732afc99bfff" path="/var/lib/kubelet/pods/8e865ae0-9dab-49b1-ad48-732afc99bfff/volumes"
Sep 11 19:59:19.954687 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:19.953868    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:59:19.985051 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:19.985013    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn" podUID="e738a24a-b4af-43ec-978f-e3f5fbc11a54" containerName="main" containerID="cri-o://4edcd00518f4d895ff3cb88ffe6f73eedcdd3523327f182fd5b37d741242f638" gracePeriod=45
Sep 11 19:59:19.987283 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:19.987257    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:59:20.018046 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.018018    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-model-cache\") pod \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") "
Sep 11 19:59:20.018290 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.018055    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-tmp-dir\") pod \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") "
Sep 11 19:59:20.018290 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.018094    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtxlc\" (UniqueName: \"kubernetes.io/projected/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-kube-api-access-qtxlc\") pod \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") "
Sep 11 19:59:20.018290 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.018157    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-tls-certs\") pod \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") "
Sep 11 19:59:20.018290 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.018194    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-dshm\") pod \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") "
Sep 11 19:59:20.018290 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.018288    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-home\") pod \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\" (UID: \"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da\") "
Sep 11 19:59:20.018571 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.018375    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ab47224-e82d-4d05-ad6c-bfa34e8dc0da" (UID: "6ab47224-e82d-4d05-ad6c-bfa34e8dc0da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:59:20.018639 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.018622    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-home" (OuterVolumeSpecName: "home") pod "6ab47224-e82d-4d05-ad6c-bfa34e8dc0da" (UID: "6ab47224-e82d-4d05-ad6c-bfa34e8dc0da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:59:20.018876 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.018850    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-tmp-dir\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:20.018981 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.018884    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-home\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:20.019277 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.019138    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ab47224-e82d-4d05-ad6c-bfa34e8dc0da" (UID: "6ab47224-e82d-4d05-ad6c-bfa34e8dc0da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:59:20.021455 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.021418    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-kube-api-access-qtxlc" (OuterVolumeSpecName: "kube-api-access-qtxlc") pod "6ab47224-e82d-4d05-ad6c-bfa34e8dc0da" (UID: "6ab47224-e82d-4d05-ad6c-bfa34e8dc0da"). InnerVolumeSpecName "kube-api-access-qtxlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:59:20.021596 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.021533    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-dshm" (OuterVolumeSpecName: "dshm") pod "6ab47224-e82d-4d05-ad6c-bfa34e8dc0da" (UID: "6ab47224-e82d-4d05-ad6c-bfa34e8dc0da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:59:20.022005 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.021977    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ab47224-e82d-4d05-ad6c-bfa34e8dc0da" (UID: "6ab47224-e82d-4d05-ad6c-bfa34e8dc0da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:59:20.119589 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.119554    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-dshm\") pod \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") "
Sep 11 19:59:20.119800 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.119685    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-tls-certs\") pod \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") "
Sep 11 19:59:20.119800 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.119702    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv4jv\" (UniqueName: \"kubernetes.io/projected/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-kube-api-access-mv4jv\") pod \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") "
Sep 11 19:59:20.119800 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.119755    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-tmp-dir\") pod \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") "
Sep 11 19:59:20.119990 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.119938    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-home\") pod \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") "
Sep 11 19:59:20.120047 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.120010    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3ab7c50-67ce-4b5d-a86c-5132e63f7188" (UID: "a3ab7c50-67ce-4b5d-a86c-5132e63f7188"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:59:20.120047 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.120028    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-model-cache\") pod \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\" (UID: \"a3ab7c50-67ce-4b5d-a86c-5132e63f7188\") "
Sep 11 19:59:20.120441 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.120413    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-home" (OuterVolumeSpecName: "home") pod "a3ab7c50-67ce-4b5d-a86c-5132e63f7188" (UID: "a3ab7c50-67ce-4b5d-a86c-5132e63f7188"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:59:20.120633 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.120440    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-tls-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:20.120785 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.120679    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-dshm\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:20.120785 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.120697    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-tmp-dir\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:20.120785 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.120536    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3ab7c50-67ce-4b5d-a86c-5132e63f7188" (UID: "a3ab7c50-67ce-4b5d-a86c-5132e63f7188"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:59:20.120785 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.120768    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-model-cache\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:20.120785 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.120786    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtxlc\" (UniqueName: \"kubernetes.io/projected/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da-kube-api-access-qtxlc\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:20.122171 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.122149    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-dshm" (OuterVolumeSpecName: "dshm") pod "a3ab7c50-67ce-4b5d-a86c-5132e63f7188" (UID: "a3ab7c50-67ce-4b5d-a86c-5132e63f7188"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:59:20.122279 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.122168    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3ab7c50-67ce-4b5d-a86c-5132e63f7188" (UID: "a3ab7c50-67ce-4b5d-a86c-5132e63f7188"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:59:20.122563 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.122540    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-kube-api-access-mv4jv" (OuterVolumeSpecName: "kube-api-access-mv4jv") pod "a3ab7c50-67ce-4b5d-a86c-5132e63f7188" (UID: "a3ab7c50-67ce-4b5d-a86c-5132e63f7188"). InnerVolumeSpecName "kube-api-access-mv4jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:59:20.222155 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.222062    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-home\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:20.222155 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.222093    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-model-cache\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:20.222155 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.222106    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-dshm\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:20.222155 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.222118    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-tls-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:20.222155 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.222130    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv4jv\" (UniqueName: \"kubernetes.io/projected/a3ab7c50-67ce-4b5d-a86c-5132e63f7188-kube-api-access-mv4jv\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:20.236380 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.236346    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:59:20.323297 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.323254    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-home\") pod \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") "
Sep 11 19:59:20.323804 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.323440    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e738a24a-b4af-43ec-978f-e3f5fbc11a54-tls-certs\") pod \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") "
Sep 11 19:59:20.323804 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.323483    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-model-cache\") pod \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") "
Sep 11 19:59:20.323804 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.323510    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-tmp-dir\") pod \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") "
Sep 11 19:59:20.323804 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.323576    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-dshm\") pod \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") "
Sep 11 19:59:20.324089 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.323812    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-model-cache" (OuterVolumeSpecName: "model-cache") pod "e738a24a-b4af-43ec-978f-e3f5fbc11a54" (UID: "e738a24a-b4af-43ec-978f-e3f5fbc11a54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:59:20.324089 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.323858    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e738a24a-b4af-43ec-978f-e3f5fbc11a54" (UID: "e738a24a-b4af-43ec-978f-e3f5fbc11a54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:59:20.324089 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.323913    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-home" (OuterVolumeSpecName: "home") pod "e738a24a-b4af-43ec-978f-e3f5fbc11a54" (UID: "e738a24a-b4af-43ec-978f-e3f5fbc11a54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:59:20.324089 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.323988    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcf8n\" (UniqueName: \"kubernetes.io/projected/e738a24a-b4af-43ec-978f-e3f5fbc11a54-kube-api-access-kcf8n\") pod \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\" (UID: \"e738a24a-b4af-43ec-978f-e3f5fbc11a54\") "
Sep 11 19:59:20.324389 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.324367    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-model-cache\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:20.324452 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.324395    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-tmp-dir\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:20.324452 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.324408    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-home\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:20.326002 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.325966    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-dshm" (OuterVolumeSpecName: "dshm") pod "e738a24a-b4af-43ec-978f-e3f5fbc11a54" (UID: "e738a24a-b4af-43ec-978f-e3f5fbc11a54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:59:20.326333 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.326307    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e738a24a-b4af-43ec-978f-e3f5fbc11a54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e738a24a-b4af-43ec-978f-e3f5fbc11a54" (UID: "e738a24a-b4af-43ec-978f-e3f5fbc11a54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:59:20.326333 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.326319    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e738a24a-b4af-43ec-978f-e3f5fbc11a54-kube-api-access-kcf8n" (OuterVolumeSpecName: "kube-api-access-kcf8n") pod "e738a24a-b4af-43ec-978f-e3f5fbc11a54" (UID: "e738a24a-b4af-43ec-978f-e3f5fbc11a54"). InnerVolumeSpecName "kube-api-access-kcf8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:59:20.425259 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.425221    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e738a24a-b4af-43ec-978f-e3f5fbc11a54-tls-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:20.425259 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.425252    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e738a24a-b4af-43ec-978f-e3f5fbc11a54-dshm\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:20.425259 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.425261    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcf8n\" (UniqueName: \"kubernetes.io/projected/e738a24a-b4af-43ec-978f-e3f5fbc11a54-kube-api-access-kcf8n\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:20.595968 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.595870    2558 generic.go:356] "Generic (PLEG): container finished" podID="a3ab7c50-67ce-4b5d-a86c-5132e63f7188" containerID="7c206e4852195a9977648c0eb05fdb9657a8fe9bed91c51b5bec57f2abda36fa" exitCode=0
Sep 11 19:59:20.595968 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.595932    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"
Sep 11 19:59:20.596221 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.595931    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9" event={"ID":"a3ab7c50-67ce-4b5d-a86c-5132e63f7188","Type":"ContainerDied","Data":"7c206e4852195a9977648c0eb05fdb9657a8fe9bed91c51b5bec57f2abda36fa"}
Sep 11 19:59:20.596221 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.596113    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9" event={"ID":"a3ab7c50-67ce-4b5d-a86c-5132e63f7188","Type":"ContainerDied","Data":"94d461e631a0f9a7ae1ccf25a8b4d6fa841091928352e32e8606b255108fa743"}
Sep 11 19:59:20.596221 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.596151    2558 scope.go:117] "RemoveContainer" containerID="7c206e4852195a9977648c0eb05fdb9657a8fe9bed91c51b5bec57f2abda36fa"
Sep 11 19:59:20.597521 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.597498    2558 generic.go:356] "Generic (PLEG): container finished" podID="6ab47224-e82d-4d05-ad6c-bfa34e8dc0da" containerID="2d36c7f8a4e2aa9f3fac4c21239ad748f8a3023d004eb857b0f3a530bdd1fc3f" exitCode=0
Sep 11 19:59:20.597647 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.597543    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"
Sep 11 19:59:20.597647 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.597580    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk" event={"ID":"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da","Type":"ContainerDied","Data":"2d36c7f8a4e2aa9f3fac4c21239ad748f8a3023d004eb857b0f3a530bdd1fc3f"}
Sep 11 19:59:20.597647 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.597621    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk" event={"ID":"6ab47224-e82d-4d05-ad6c-bfa34e8dc0da","Type":"ContainerDied","Data":"2781bdde11cfcf14500af33c627a9c8e63bda8e5a43b16fe0cc461f3b2c1fc77"}
Sep 11 19:59:20.599384 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.599344    2558 generic.go:356] "Generic (PLEG): container finished" podID="e738a24a-b4af-43ec-978f-e3f5fbc11a54" containerID="4edcd00518f4d895ff3cb88ffe6f73eedcdd3523327f182fd5b37d741242f638" exitCode=0
Sep 11 19:59:20.599384 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.599379    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"
Sep 11 19:59:20.599594 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.599396    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn" event={"ID":"e738a24a-b4af-43ec-978f-e3f5fbc11a54","Type":"ContainerDied","Data":"4edcd00518f4d895ff3cb88ffe6f73eedcdd3523327f182fd5b37d741242f638"}
Sep 11 19:59:20.599594 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.599423    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn" event={"ID":"e738a24a-b4af-43ec-978f-e3f5fbc11a54","Type":"ContainerDied","Data":"2d1a4e8935738ec6f1c6d68ea522eb3daca255ca7e0feecb01771232228f9d08"}
Sep 11 19:59:20.609040 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.609018    2558 scope.go:117] "RemoveContainer" containerID="7c206e4852195a9977648c0eb05fdb9657a8fe9bed91c51b5bec57f2abda36fa"
Sep 11 19:59:20.609492 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:59:20.609459    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c206e4852195a9977648c0eb05fdb9657a8fe9bed91c51b5bec57f2abda36fa\": container with ID starting with 7c206e4852195a9977648c0eb05fdb9657a8fe9bed91c51b5bec57f2abda36fa not found: ID does not exist" containerID="7c206e4852195a9977648c0eb05fdb9657a8fe9bed91c51b5bec57f2abda36fa"
Sep 11 19:59:20.609607 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.609508    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c206e4852195a9977648c0eb05fdb9657a8fe9bed91c51b5bec57f2abda36fa"} err="failed to get container status \"7c206e4852195a9977648c0eb05fdb9657a8fe9bed91c51b5bec57f2abda36fa\": rpc error: code = NotFound desc = could not find container \"7c206e4852195a9977648c0eb05fdb9657a8fe9bed91c51b5bec57f2abda36fa\": container with ID starting with 7c206e4852195a9977648c0eb05fdb9657a8fe9bed91c51b5bec57f2abda36fa not found: ID does not exist"
Sep 11 19:59:20.609607 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.609531    2558 scope.go:117] "RemoveContainer" containerID="2d36c7f8a4e2aa9f3fac4c21239ad748f8a3023d004eb857b0f3a530bdd1fc3f"
Sep 11 19:59:20.620419 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.620347    2558 scope.go:117] "RemoveContainer" containerID="2d36c7f8a4e2aa9f3fac4c21239ad748f8a3023d004eb857b0f3a530bdd1fc3f"
Sep 11 19:59:20.620759 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:59:20.620733    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d36c7f8a4e2aa9f3fac4c21239ad748f8a3023d004eb857b0f3a530bdd1fc3f\": container with ID starting with 2d36c7f8a4e2aa9f3fac4c21239ad748f8a3023d004eb857b0f3a530bdd1fc3f not found: ID does not exist" containerID="2d36c7f8a4e2aa9f3fac4c21239ad748f8a3023d004eb857b0f3a530bdd1fc3f"
Sep 11 19:59:20.620848 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.620767    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d36c7f8a4e2aa9f3fac4c21239ad748f8a3023d004eb857b0f3a530bdd1fc3f"} err="failed to get container status \"2d36c7f8a4e2aa9f3fac4c21239ad748f8a3023d004eb857b0f3a530bdd1fc3f\": rpc error: code = NotFound desc = could not find container \"2d36c7f8a4e2aa9f3fac4c21239ad748f8a3023d004eb857b0f3a530bdd1fc3f\": container with ID starting with 2d36c7f8a4e2aa9f3fac4c21239ad748f8a3023d004eb857b0f3a530bdd1fc3f not found: ID does not exist"
Sep 11 19:59:20.620848 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.620785    2558 scope.go:117] "RemoveContainer" containerID="4edcd00518f4d895ff3cb88ffe6f73eedcdd3523327f182fd5b37d741242f638"
Sep 11 19:59:20.623289 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.623268    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"]
Sep 11 19:59:20.629939 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.629912    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-10cc24/embedding-w0-10cc24-kserve-78dc8d48cf-4dlh9"]
Sep 11 19:59:20.630918 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.630903    2558 scope.go:117] "RemoveContainer" containerID="4edcd00518f4d895ff3cb88ffe6f73eedcdd3523327f182fd5b37d741242f638"
Sep 11 19:59:20.631242 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:59:20.631213    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4edcd00518f4d895ff3cb88ffe6f73eedcdd3523327f182fd5b37d741242f638\": container with ID starting with 4edcd00518f4d895ff3cb88ffe6f73eedcdd3523327f182fd5b37d741242f638 not found: ID does not exist" containerID="4edcd00518f4d895ff3cb88ffe6f73eedcdd3523327f182fd5b37d741242f638"
Sep 11 19:59:20.631332 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.631242    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4edcd00518f4d895ff3cb88ffe6f73eedcdd3523327f182fd5b37d741242f638"} err="failed to get container status \"4edcd00518f4d895ff3cb88ffe6f73eedcdd3523327f182fd5b37d741242f638\": rpc error: code = NotFound desc = could not find container \"4edcd00518f4d895ff3cb88ffe6f73eedcdd3523327f182fd5b37d741242f638\": container with ID starting with 4edcd00518f4d895ff3cb88ffe6f73eedcdd3523327f182fd5b37d741242f638 not found: ID does not exist"
Sep 11 19:59:20.638031 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.638007    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"]
Sep 11 19:59:20.640585 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.640565    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c7766c/distinct-w4-c7766c-kserve-6d98876f59-5kfmn"]
Sep 11 19:59:20.650032 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.650009    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"]
Sep 11 19:59:20.656506 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:20.656485    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-10cc24/distinct-2-w0-10cc24-kserve-c465b5977-bpngk"]
Sep 11 19:59:21.814002 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:21.813971    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ab47224-e82d-4d05-ad6c-bfa34e8dc0da" path="/var/lib/kubelet/pods/6ab47224-e82d-4d05-ad6c-bfa34e8dc0da/volumes"
Sep 11 19:59:21.814412 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:21.814398    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3ab7c50-67ce-4b5d-a86c-5132e63f7188" path="/var/lib/kubelet/pods/a3ab7c50-67ce-4b5d-a86c-5132e63f7188/volumes"
Sep 11 19:59:21.814783 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:21.814769    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e738a24a-b4af-43ec-978f-e3f5fbc11a54" path="/var/lib/kubelet/pods/e738a24a-b4af-43ec-978f-e3f5fbc11a54/volumes"
Sep 11 19:59:24.568494 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:24.568455    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"]
Sep 11 19:59:24.568970 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:24.568776    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq" podUID="195a2bd1-6a70-4e0a-a728-27fcb3ead58b" containerName="istio-proxy" containerID="cri-o://b8a66eaa68e39e048d81941ef66dcafcdf4351ecbd20835f7c2df706959036b5" gracePeriod=30
Sep 11 19:59:29.830034 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.830009    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:59:29.920923 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.920886    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjgkr\" (UniqueName: \"kubernetes.io/projected/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-kube-api-access-qjgkr\") pod \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") "
Sep 11 19:59:29.921128 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.920946    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-token\") pod \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") "
Sep 11 19:59:29.921128 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.921040    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istiod-ca-cert\") pod \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") "
Sep 11 19:59:29.921128 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.921080    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-credential-socket\") pod \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") "
Sep 11 19:59:29.921128 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.921101    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-workload-certs\") pod \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") "
Sep 11 19:59:29.921347 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.921131    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-envoy\") pod \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") "
Sep 11 19:59:29.921347 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.921166    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-data\") pod \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") "
Sep 11 19:59:29.921347 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.921206    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-podinfo\") pod \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") "
Sep 11 19:59:29.921347 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.921283    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-workload-socket\") pod \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\" (UID: \"195a2bd1-6a70-4e0a-a728-27fcb3ead58b\") "
Sep 11 19:59:29.921548 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.921403    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "195a2bd1-6a70-4e0a-a728-27fcb3ead58b" (UID: "195a2bd1-6a70-4e0a-a728-27fcb3ead58b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:59:29.921548 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.921522    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-data" (OuterVolumeSpecName: "istio-data") pod "195a2bd1-6a70-4e0a-a728-27fcb3ead58b" (UID: "195a2bd1-6a70-4e0a-a728-27fcb3ead58b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:59:29.921651 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.921549    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "195a2bd1-6a70-4e0a-a728-27fcb3ead58b" (UID: "195a2bd1-6a70-4e0a-a728-27fcb3ead58b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:59:29.921651 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.921627    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "195a2bd1-6a70-4e0a-a728-27fcb3ead58b" (UID: "195a2bd1-6a70-4e0a-a728-27fcb3ead58b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:59:29.921759 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.921656    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "195a2bd1-6a70-4e0a-a728-27fcb3ead58b" (UID: "195a2bd1-6a70-4e0a-a728-27fcb3ead58b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:59:29.921979 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.921956    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-workload-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:29.922084 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.921982    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istiod-ca-cert\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:29.922084 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.921997    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-credential-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:29.922084 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.922011    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-workload-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:29.922084 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.922025    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-data\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:29.923403 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.923377    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-kube-api-access-qjgkr" (OuterVolumeSpecName: "kube-api-access-qjgkr") pod "195a2bd1-6a70-4e0a-a728-27fcb3ead58b" (UID: "195a2bd1-6a70-4e0a-a728-27fcb3ead58b"). InnerVolumeSpecName "kube-api-access-qjgkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:59:29.923494 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.923439    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-token" (OuterVolumeSpecName: "istio-token") pod "195a2bd1-6a70-4e0a-a728-27fcb3ead58b" (UID: "195a2bd1-6a70-4e0a-a728-27fcb3ead58b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:59:29.923573 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.923557    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "195a2bd1-6a70-4e0a-a728-27fcb3ead58b" (UID: "195a2bd1-6a70-4e0a-a728-27fcb3ead58b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 19:59:29.923621 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:29.923572    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "195a2bd1-6a70-4e0a-a728-27fcb3ead58b" (UID: "195a2bd1-6a70-4e0a-a728-27fcb3ead58b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 19:59:30.023080 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:30.023038    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-envoy\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:30.023080 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:30.023072    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-podinfo\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:30.023080 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:30.023082    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjgkr\" (UniqueName: \"kubernetes.io/projected/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-kube-api-access-qjgkr\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:30.023080 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:30.023092    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/195a2bd1-6a70-4e0a-a728-27fcb3ead58b-istio-token\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:30.659227 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:30.659194    2558 generic.go:356] "Generic (PLEG): container finished" podID="195a2bd1-6a70-4e0a-a728-27fcb3ead58b" containerID="b8a66eaa68e39e048d81941ef66dcafcdf4351ecbd20835f7c2df706959036b5" exitCode=0
Sep 11 19:59:30.659409 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:30.659245    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"
Sep 11 19:59:30.659409 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:30.659257    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq" event={"ID":"195a2bd1-6a70-4e0a-a728-27fcb3ead58b","Type":"ContainerDied","Data":"b8a66eaa68e39e048d81941ef66dcafcdf4351ecbd20835f7c2df706959036b5"}
Sep 11 19:59:30.659409 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:30.659302    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq" event={"ID":"195a2bd1-6a70-4e0a-a728-27fcb3ead58b","Type":"ContainerDied","Data":"cb12df8619c9cde92f31086c9d78b8ae4333363a3e0a16f16ca918258a49af72"}
Sep 11 19:59:30.659409 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:30.659325    2558 scope.go:117] "RemoveContainer" containerID="b8a66eaa68e39e048d81941ef66dcafcdf4351ecbd20835f7c2df706959036b5"
Sep 11 19:59:30.675258 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:30.675233    2558 scope.go:117] "RemoveContainer" containerID="02e8b243217eccca6b797952f9013e5239aa9821207978fe2687e28b22756580"
Sep 11 19:59:30.684604 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:30.684570    2558 scope.go:117] "RemoveContainer" containerID="b8a66eaa68e39e048d81941ef66dcafcdf4351ecbd20835f7c2df706959036b5"
Sep 11 19:59:30.684974 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:59:30.684953    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8a66eaa68e39e048d81941ef66dcafcdf4351ecbd20835f7c2df706959036b5\": container with ID starting with b8a66eaa68e39e048d81941ef66dcafcdf4351ecbd20835f7c2df706959036b5 not found: ID does not exist" containerID="b8a66eaa68e39e048d81941ef66dcafcdf4351ecbd20835f7c2df706959036b5"
Sep 11 19:59:30.685047 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:30.684983    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a66eaa68e39e048d81941ef66dcafcdf4351ecbd20835f7c2df706959036b5"} err="failed to get container status \"b8a66eaa68e39e048d81941ef66dcafcdf4351ecbd20835f7c2df706959036b5\": rpc error: code = NotFound desc = could not find container \"b8a66eaa68e39e048d81941ef66dcafcdf4351ecbd20835f7c2df706959036b5\": container with ID starting with b8a66eaa68e39e048d81941ef66dcafcdf4351ecbd20835f7c2df706959036b5 not found: ID does not exist"
Sep 11 19:59:30.685047 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:30.685003    2558 scope.go:117] "RemoveContainer" containerID="02e8b243217eccca6b797952f9013e5239aa9821207978fe2687e28b22756580"
Sep 11 19:59:30.685287 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:59:30.685272    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02e8b243217eccca6b797952f9013e5239aa9821207978fe2687e28b22756580\": container with ID starting with 02e8b243217eccca6b797952f9013e5239aa9821207978fe2687e28b22756580 not found: ID does not exist" containerID="02e8b243217eccca6b797952f9013e5239aa9821207978fe2687e28b22756580"
Sep 11 19:59:30.685327 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:30.685292    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e8b243217eccca6b797952f9013e5239aa9821207978fe2687e28b22756580"} err="failed to get container status \"02e8b243217eccca6b797952f9013e5239aa9821207978fe2687e28b22756580\": rpc error: code = NotFound desc = could not find container \"02e8b243217eccca6b797952f9013e5239aa9821207978fe2687e28b22756580\": container with ID starting with 02e8b243217eccca6b797952f9013e5239aa9821207978fe2687e28b22756580 not found: ID does not exist"
Sep 11 19:59:30.685925 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:30.685906    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"]
Sep 11 19:59:30.690798 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:30.690777    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-10cc24-openshift-default-759d75854b-q6xjq"]
Sep 11 19:59:30.986684 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:30.986653    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"]
Sep 11 19:59:30.987165 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:30.986902    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t" podUID="a505761d-cc6a-41be-b089-2437e90c755a" containerName="maas-api" containerID="cri-o://ab56831cb8443cc521666f01e6ad84c3428d3a53d0d08613315178ea852dbcb1" gracePeriod=30
Sep 11 19:59:31.242926 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.242862    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"
Sep 11 19:59:31.336144 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.336101    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc6jg\" (UniqueName: \"kubernetes.io/projected/a505761d-cc6a-41be-b089-2437e90c755a-kube-api-access-hc6jg\") pod \"a505761d-cc6a-41be-b089-2437e90c755a\" (UID: \"a505761d-cc6a-41be-b089-2437e90c755a\") "
Sep 11 19:59:31.336302 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.336204    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a505761d-cc6a-41be-b089-2437e90c755a-maas-api-tls\") pod \"a505761d-cc6a-41be-b089-2437e90c755a\" (UID: \"a505761d-cc6a-41be-b089-2437e90c755a\") "
Sep 11 19:59:31.336302 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.336251    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a505761d-cc6a-41be-b089-2437e90c755a-openshift-service-ca\") pod \"a505761d-cc6a-41be-b089-2437e90c755a\" (UID: \"a505761d-cc6a-41be-b089-2437e90c755a\") "
Sep 11 19:59:31.336404 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.336326    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a505761d-cc6a-41be-b089-2437e90c755a-metrics-certs\") pod \"a505761d-cc6a-41be-b089-2437e90c755a\" (UID: \"a505761d-cc6a-41be-b089-2437e90c755a\") "
Sep 11 19:59:31.336770 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.336731    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a505761d-cc6a-41be-b089-2437e90c755a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a505761d-cc6a-41be-b089-2437e90c755a" (UID: "a505761d-cc6a-41be-b089-2437e90c755a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:59:31.338476 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.338438    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a505761d-cc6a-41be-b089-2437e90c755a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a505761d-cc6a-41be-b089-2437e90c755a" (UID: "a505761d-cc6a-41be-b089-2437e90c755a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:59:31.338852 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.338816    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a505761d-cc6a-41be-b089-2437e90c755a-kube-api-access-hc6jg" (OuterVolumeSpecName: "kube-api-access-hc6jg") pod "a505761d-cc6a-41be-b089-2437e90c755a" (UID: "a505761d-cc6a-41be-b089-2437e90c755a"). InnerVolumeSpecName "kube-api-access-hc6jg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:59:31.338980 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.338876    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a505761d-cc6a-41be-b089-2437e90c755a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a505761d-cc6a-41be-b089-2437e90c755a" (UID: "a505761d-cc6a-41be-b089-2437e90c755a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:59:31.438104 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.438058    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a505761d-cc6a-41be-b089-2437e90c755a-metrics-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:31.438104 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.438103    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc6jg\" (UniqueName: \"kubernetes.io/projected/a505761d-cc6a-41be-b089-2437e90c755a-kube-api-access-hc6jg\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:31.438298 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.438119    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a505761d-cc6a-41be-b089-2437e90c755a-maas-api-tls\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:31.438298 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.438134    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a505761d-cc6a-41be-b089-2437e90c755a-openshift-service-ca\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:31.666420 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.666389    2558 generic.go:356] "Generic (PLEG): container finished" podID="a505761d-cc6a-41be-b089-2437e90c755a" containerID="ab56831cb8443cc521666f01e6ad84c3428d3a53d0d08613315178ea852dbcb1" exitCode=0
Sep 11 19:59:31.666617 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.666439    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"
Sep 11 19:59:31.666617 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.666472    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t" event={"ID":"a505761d-cc6a-41be-b089-2437e90c755a","Type":"ContainerDied","Data":"ab56831cb8443cc521666f01e6ad84c3428d3a53d0d08613315178ea852dbcb1"}
Sep 11 19:59:31.666617 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.666531    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t" event={"ID":"a505761d-cc6a-41be-b089-2437e90c755a","Type":"ContainerDied","Data":"a3f48018f7e84f6042ff251cf03d2156fd9d4e91d29241b443c29050bf2f6aee"}
Sep 11 19:59:31.666617 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.666560    2558 scope.go:117] "RemoveContainer" containerID="ab56831cb8443cc521666f01e6ad84c3428d3a53d0d08613315178ea852dbcb1"
Sep 11 19:59:31.676771 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.676748    2558 scope.go:117] "RemoveContainer" containerID="ab56831cb8443cc521666f01e6ad84c3428d3a53d0d08613315178ea852dbcb1"
Sep 11 19:59:31.677038 ip-10-0-138-43 kubenswrapper[2558]: E0911 19:59:31.677019    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab56831cb8443cc521666f01e6ad84c3428d3a53d0d08613315178ea852dbcb1\": container with ID starting with ab56831cb8443cc521666f01e6ad84c3428d3a53d0d08613315178ea852dbcb1 not found: ID does not exist" containerID="ab56831cb8443cc521666f01e6ad84c3428d3a53d0d08613315178ea852dbcb1"
Sep 11 19:59:31.677082 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.677047    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab56831cb8443cc521666f01e6ad84c3428d3a53d0d08613315178ea852dbcb1"} err="failed to get container status \"ab56831cb8443cc521666f01e6ad84c3428d3a53d0d08613315178ea852dbcb1\": rpc error: code = NotFound desc = could not find container \"ab56831cb8443cc521666f01e6ad84c3428d3a53d0d08613315178ea852dbcb1\": container with ID starting with ab56831cb8443cc521666f01e6ad84c3428d3a53d0d08613315178ea852dbcb1 not found: ID does not exist"
Sep 11 19:59:31.688318 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.688292    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"]
Sep 11 19:59:31.695061 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.695038    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d14d7e-79b895dfcc-vb27t"]
Sep 11 19:59:31.814919 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.814877    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="195a2bd1-6a70-4e0a-a728-27fcb3ead58b" path="/var/lib/kubelet/pods/195a2bd1-6a70-4e0a-a728-27fcb3ead58b/volumes"
Sep 11 19:59:31.815378 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:31.815356    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a505761d-cc6a-41be-b089-2437e90c755a" path="/var/lib/kubelet/pods/a505761d-cc6a-41be-b089-2437e90c755a/volumes"
Sep 11 19:59:48.617852 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:48.617815    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"]
Sep 11 19:59:48.618405 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:48.618013    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4" podUID="1dad3dae-b05e-4d88-bdf0-c31406c7ede3" containerName="maas-api" containerID="cri-o://c8301399abfff3bf0ff61ea7ec6388b45ae110f01dece49bc5428de3c6750e3f" gracePeriod=30
Sep 11 19:59:48.761219 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:48.761184    2558 generic.go:356] "Generic (PLEG): container finished" podID="1dad3dae-b05e-4d88-bdf0-c31406c7ede3" containerID="c8301399abfff3bf0ff61ea7ec6388b45ae110f01dece49bc5428de3c6750e3f" exitCode=0
Sep 11 19:59:48.761392 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:48.761252    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4" event={"ID":"1dad3dae-b05e-4d88-bdf0-c31406c7ede3","Type":"ContainerDied","Data":"c8301399abfff3bf0ff61ea7ec6388b45ae110f01dece49bc5428de3c6750e3f"}
Sep 11 19:59:48.888433 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:48.888405    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"
Sep 11 19:59:48.911241 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:48.911211    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-metrics-certs\") pod \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\" (UID: \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\") "
Sep 11 19:59:48.911408 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:48.911263    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-maas-api-tls\") pod \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\" (UID: \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\") "
Sep 11 19:59:48.911408 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:48.911303    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-openshift-service-ca\") pod \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\" (UID: \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\") "
Sep 11 19:59:48.912232 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:48.911909    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1dad3dae-b05e-4d88-bdf0-c31406c7ede3" (UID: "1dad3dae-b05e-4d88-bdf0-c31406c7ede3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 19:59:48.912232 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:48.911511    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nn7p\" (UniqueName: \"kubernetes.io/projected/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-kube-api-access-2nn7p\") pod \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\" (UID: \"1dad3dae-b05e-4d88-bdf0-c31406c7ede3\") "
Sep 11 19:59:48.913676 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:48.913430    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-openshift-service-ca\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:48.914036 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:48.914007    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1dad3dae-b05e-4d88-bdf0-c31406c7ede3" (UID: "1dad3dae-b05e-4d88-bdf0-c31406c7ede3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:59:48.914922 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:48.914886    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1dad3dae-b05e-4d88-bdf0-c31406c7ede3" (UID: "1dad3dae-b05e-4d88-bdf0-c31406c7ede3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 19:59:48.915258 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:48.915185    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-kube-api-access-2nn7p" (OuterVolumeSpecName: "kube-api-access-2nn7p") pod "1dad3dae-b05e-4d88-bdf0-c31406c7ede3" (UID: "1dad3dae-b05e-4d88-bdf0-c31406c7ede3"). InnerVolumeSpecName "kube-api-access-2nn7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 19:59:49.014929 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:49.014889    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nn7p\" (UniqueName: \"kubernetes.io/projected/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-kube-api-access-2nn7p\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:49.014929 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:49.014923    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-metrics-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:49.014929 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:49.014932    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1dad3dae-b05e-4d88-bdf0-c31406c7ede3-maas-api-tls\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 19:59:49.767733 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:49.767672    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4" event={"ID":"1dad3dae-b05e-4d88-bdf0-c31406c7ede3","Type":"ContainerDied","Data":"80fdf15f8d5fd9303cc406c98ad83efd0769d3ee78f9ece8528f2137023f36e5"}
Sep 11 19:59:49.768153 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:49.767734    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"
Sep 11 19:59:49.768153 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:49.767773    2558 scope.go:117] "RemoveContainer" containerID="c8301399abfff3bf0ff61ea7ec6388b45ae110f01dece49bc5428de3c6750e3f"
Sep 11 19:59:49.790528 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:49.790499    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"]
Sep 11 19:59:49.795412 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:49.795388    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-58f57d-bc4f8b8f4-4tss4"]
Sep 11 19:59:49.814248 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:49.814223    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dad3dae-b05e-4d88-bdf0-c31406c7ede3" path="/var/lib/kubelet/pods/1dad3dae-b05e-4d88-bdf0-c31406c7ede3/volumes"
Sep 11 19:59:55.446064 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:55.446013    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"]
Sep 11 19:59:55.447395 ip-10-0-138-43 kubenswrapper[2558]: I0911 19:59:55.447348    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz" podUID="17652cd9-6ad9-4318-bee9-a09da4162e94" containerName="istio-proxy" containerID="cri-o://f936ffe594add77eaeb8cbfa10176e35d3ea8ffae86ca5066173104593548219" gracePeriod=30
Sep 11 20:00:00.139589 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.139546    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819280-m658h"]
Sep 11 20:00:00.140157 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140135    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b95b4ef-0d68-4e4a-a326-fd35bf643b09" containerName="payload-pre-processing"
Sep 11 20:00:00.140242 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140160    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b95b4ef-0d68-4e4a-a326-fd35bf643b09" containerName="payload-pre-processing"
Sep 11 20:00:00.140242 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140177    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a895a8e7-aee0-4585-b288-d275d9c8f2ec" containerName="maas-api"
Sep 11 20:00:00.140242 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140185    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a895a8e7-aee0-4585-b288-d275d9c8f2ec" containerName="maas-api"
Sep 11 20:00:00.140242 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140202    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e738a24a-b4af-43ec-978f-e3f5fbc11a54" containerName="main"
Sep 11 20:00:00.140242 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140212    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e738a24a-b4af-43ec-978f-e3f5fbc11a54" containerName="main"
Sep 11 20:00:00.140242 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140222    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195a2bd1-6a70-4e0a-a728-27fcb3ead58b" containerName="istio-proxy"
Sep 11 20:00:00.140242 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140231    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="195a2bd1-6a70-4e0a-a728-27fcb3ead58b" containerName="istio-proxy"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140246    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a505761d-cc6a-41be-b089-2437e90c755a" containerName="maas-api"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140255    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a505761d-cc6a-41be-b089-2437e90c755a" containerName="maas-api"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140273    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2310873-9576-4655-ae15-faa08ed4fd7c" containerName="maas-api"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140282    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2310873-9576-4655-ae15-faa08ed4fd7c" containerName="maas-api"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140292    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e865ae0-9dab-49b1-ad48-732afc99bfff" containerName="payload-processing"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140300    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e865ae0-9dab-49b1-ad48-732afc99bfff" containerName="payload-processing"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140320    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dad3dae-b05e-4d88-bdf0-c31406c7ede3" containerName="maas-api"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140329    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dad3dae-b05e-4d88-bdf0-c31406c7ede3" containerName="maas-api"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140338    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3ab7c50-67ce-4b5d-a86c-5132e63f7188" containerName="main"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140346    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3ab7c50-67ce-4b5d-a86c-5132e63f7188" containerName="main"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140357    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1086d50-c963-4705-92da-51a1c144de7b" containerName="maas-api"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140366    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1086d50-c963-4705-92da-51a1c144de7b" containerName="maas-api"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140379    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ab47224-e82d-4d05-ad6c-bfa34e8dc0da" containerName="main"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140389    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ab47224-e82d-4d05-ad6c-bfa34e8dc0da" containerName="main"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140401    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="195a2bd1-6a70-4e0a-a728-27fcb3ead58b" containerName="istio-proxy"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140410    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="195a2bd1-6a70-4e0a-a728-27fcb3ead58b" containerName="istio-proxy"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140502    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b95b4ef-0d68-4e4a-a326-fd35bf643b09" containerName="payload-pre-processing"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140515    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dad3dae-b05e-4d88-bdf0-c31406c7ede3" containerName="maas-api"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140525    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1086d50-c963-4705-92da-51a1c144de7b" containerName="maas-api"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140537    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3ab7c50-67ce-4b5d-a86c-5132e63f7188" containerName="main"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140546    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="195a2bd1-6a70-4e0a-a728-27fcb3ead58b" containerName="istio-proxy"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140554    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e865ae0-9dab-49b1-ad48-732afc99bfff" containerName="payload-processing"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140563    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="195a2bd1-6a70-4e0a-a728-27fcb3ead58b" containerName="istio-proxy"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140573    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a895a8e7-aee0-4585-b288-d275d9c8f2ec" containerName="maas-api"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140581    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a505761d-cc6a-41be-b089-2437e90c755a" containerName="maas-api"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140590    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e738a24a-b4af-43ec-978f-e3f5fbc11a54" containerName="main"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140601    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ab47224-e82d-4d05-ad6c-bfa34e8dc0da" containerName="main"
Sep 11 20:00:00.140600 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.140609    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2310873-9576-4655-ae15-faa08ed4fd7c" containerName="maas-api"
Sep 11 20:00:00.144008 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.143989    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819280-m658h"
Sep 11 20:00:00.146926 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.146907    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\""
Sep 11 20:00:00.156340 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.156316    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819280-m658h"
Sep 11 20:00:00.156653 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.156624    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819280-m658h"]
Sep 11 20:00:00.435346 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.435316    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819280-m658h"]
Sep 11 20:00:00.437657 ip-10-0-138-43 kubenswrapper[2558]: W0911 20:00:00.437622    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda112ec37_694a_4a5b_990d_10885ef25e6f.slice/crio-fe49418fb903a4841cb9fb8f80b9de630964fb609f153b7bf6664b7bd60e2e02 WatchSource:0}: Error finding container fe49418fb903a4841cb9fb8f80b9de630964fb609f153b7bf6664b7bd60e2e02: Status 404 returned error can't find the container with id fe49418fb903a4841cb9fb8f80b9de630964fb609f153b7bf6664b7bd60e2e02
Sep 11 20:00:00.706675 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.706650    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 20:00:00.831519 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.831489    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-envoy\") pod \"17652cd9-6ad9-4318-bee9-a09da4162e94\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") "
Sep 11 20:00:00.831519 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.831532    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-workload-socket\") pod \"17652cd9-6ad9-4318-bee9-a09da4162e94\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") "
Sep 11 20:00:00.831844 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.831575    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sm8px\" (UniqueName: \"kubernetes.io/projected/17652cd9-6ad9-4318-bee9-a09da4162e94-kube-api-access-sm8px\") pod \"17652cd9-6ad9-4318-bee9-a09da4162e94\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") "
Sep 11 20:00:00.831844 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.831626    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-credential-socket\") pod \"17652cd9-6ad9-4318-bee9-a09da4162e94\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") "
Sep 11 20:00:00.831844 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.831654    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-podinfo\") pod \"17652cd9-6ad9-4318-bee9-a09da4162e94\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") "
Sep 11 20:00:00.831844 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.831679    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-data\") pod \"17652cd9-6ad9-4318-bee9-a09da4162e94\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") "
Sep 11 20:00:00.831844 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.831806    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-workload-certs\") pod \"17652cd9-6ad9-4318-bee9-a09da4162e94\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") "
Sep 11 20:00:00.831844 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.831843    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-token\") pod \"17652cd9-6ad9-4318-bee9-a09da4162e94\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") "
Sep 11 20:00:00.832155 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.831892    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17652cd9-6ad9-4318-bee9-a09da4162e94-istiod-ca-cert\") pod \"17652cd9-6ad9-4318-bee9-a09da4162e94\" (UID: \"17652cd9-6ad9-4318-bee9-a09da4162e94\") "
Sep 11 20:00:00.832155 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.832012    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-data" (OuterVolumeSpecName: "istio-data") pod "17652cd9-6ad9-4318-bee9-a09da4162e94" (UID: "17652cd9-6ad9-4318-bee9-a09da4162e94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:00.832155 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.832106    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "17652cd9-6ad9-4318-bee9-a09da4162e94" (UID: "17652cd9-6ad9-4318-bee9-a09da4162e94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:00.832323 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.832238    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "17652cd9-6ad9-4318-bee9-a09da4162e94" (UID: "17652cd9-6ad9-4318-bee9-a09da4162e94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:00.832323 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.832299    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "17652cd9-6ad9-4318-bee9-a09da4162e94" (UID: "17652cd9-6ad9-4318-bee9-a09da4162e94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:00.832425 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.832349    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17652cd9-6ad9-4318-bee9-a09da4162e94-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "17652cd9-6ad9-4318-bee9-a09da4162e94" (UID: "17652cd9-6ad9-4318-bee9-a09da4162e94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:00:00.832764 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.832735    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-workload-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:00:00.832885 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.832767    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17652cd9-6ad9-4318-bee9-a09da4162e94-istiod-ca-cert\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:00:00.832885 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.832782    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-workload-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:00:00.832885 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.832798    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-credential-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:00:00.832885 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.832813    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-data\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:00:00.834186 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.834161    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "17652cd9-6ad9-4318-bee9-a09da4162e94" (UID: "17652cd9-6ad9-4318-bee9-a09da4162e94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 20:00:00.834309 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.834215    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "17652cd9-6ad9-4318-bee9-a09da4162e94" (UID: "17652cd9-6ad9-4318-bee9-a09da4162e94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:00:00.834309 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.834234    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17652cd9-6ad9-4318-bee9-a09da4162e94-kube-api-access-sm8px" (OuterVolumeSpecName: "kube-api-access-sm8px") pod "17652cd9-6ad9-4318-bee9-a09da4162e94" (UID: "17652cd9-6ad9-4318-bee9-a09da4162e94"). InnerVolumeSpecName "kube-api-access-sm8px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:00:00.834564 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.834540    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-token" (OuterVolumeSpecName: "istio-token") pod "17652cd9-6ad9-4318-bee9-a09da4162e94" (UID: "17652cd9-6ad9-4318-bee9-a09da4162e94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:00:00.837607 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.837576    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819280-m658h" event={"ID":"a112ec37-694a-4a5b-990d-10885ef25e6f","Type":"ContainerStarted","Data":"fe49418fb903a4841cb9fb8f80b9de630964fb609f153b7bf6664b7bd60e2e02"}
Sep 11 20:00:00.839161 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.839140    2558 generic.go:356] "Generic (PLEG): container finished" podID="17652cd9-6ad9-4318-bee9-a09da4162e94" containerID="f936ffe594add77eaeb8cbfa10176e35d3ea8ffae86ca5066173104593548219" exitCode=0
Sep 11 20:00:00.839256 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.839190    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz" event={"ID":"17652cd9-6ad9-4318-bee9-a09da4162e94","Type":"ContainerDied","Data":"f936ffe594add77eaeb8cbfa10176e35d3ea8ffae86ca5066173104593548219"}
Sep 11 20:00:00.839256 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.839195    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"
Sep 11 20:00:00.839256 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.839228    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz" event={"ID":"17652cd9-6ad9-4318-bee9-a09da4162e94","Type":"ContainerDied","Data":"86e3585490e5d9519d6ea158ca105e867e0ba8912e74f3966f0dc4feedfe9149"}
Sep 11 20:00:00.839256 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.839248    2558 scope.go:117] "RemoveContainer" containerID="f936ffe594add77eaeb8cbfa10176e35d3ea8ffae86ca5066173104593548219"
Sep 11 20:00:00.848231 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.848214    2558 scope.go:117] "RemoveContainer" containerID="c78d728f36119988833e0a2f8fdef8378130b0493471d45d69d3be292278d5b6"
Sep 11 20:00:00.856168 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.856148    2558 scope.go:117] "RemoveContainer" containerID="f936ffe594add77eaeb8cbfa10176e35d3ea8ffae86ca5066173104593548219"
Sep 11 20:00:00.856481 ip-10-0-138-43 kubenswrapper[2558]: E0911 20:00:00.856460    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f936ffe594add77eaeb8cbfa10176e35d3ea8ffae86ca5066173104593548219\": container with ID starting with f936ffe594add77eaeb8cbfa10176e35d3ea8ffae86ca5066173104593548219 not found: ID does not exist" containerID="f936ffe594add77eaeb8cbfa10176e35d3ea8ffae86ca5066173104593548219"
Sep 11 20:00:00.856546 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.856489    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f936ffe594add77eaeb8cbfa10176e35d3ea8ffae86ca5066173104593548219"} err="failed to get container status \"f936ffe594add77eaeb8cbfa10176e35d3ea8ffae86ca5066173104593548219\": rpc error: code = NotFound desc = could not find container \"f936ffe594add77eaeb8cbfa10176e35d3ea8ffae86ca5066173104593548219\": container with ID starting with f936ffe594add77eaeb8cbfa10176e35d3ea8ffae86ca5066173104593548219 not found: ID does not exist"
Sep 11 20:00:00.856546 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.856508    2558 scope.go:117] "RemoveContainer" containerID="c78d728f36119988833e0a2f8fdef8378130b0493471d45d69d3be292278d5b6"
Sep 11 20:00:00.856831 ip-10-0-138-43 kubenswrapper[2558]: E0911 20:00:00.856807    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c78d728f36119988833e0a2f8fdef8378130b0493471d45d69d3be292278d5b6\": container with ID starting with c78d728f36119988833e0a2f8fdef8378130b0493471d45d69d3be292278d5b6 not found: ID does not exist" containerID="c78d728f36119988833e0a2f8fdef8378130b0493471d45d69d3be292278d5b6"
Sep 11 20:00:00.856879 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.856838    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c78d728f36119988833e0a2f8fdef8378130b0493471d45d69d3be292278d5b6"} err="failed to get container status \"c78d728f36119988833e0a2f8fdef8378130b0493471d45d69d3be292278d5b6\": rpc error: code = NotFound desc = could not find container \"c78d728f36119988833e0a2f8fdef8378130b0493471d45d69d3be292278d5b6\": container with ID starting with c78d728f36119988833e0a2f8fdef8378130b0493471d45d69d3be292278d5b6 not found: ID does not exist"
Sep 11 20:00:00.862694 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.862670    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"]
Sep 11 20:00:00.873729 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.871897    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-58f57d-openshift-default-65ff98674b-4jmfz"]
Sep 11 20:00:00.934282 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.934231    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-token\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:00:00.934282 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.934274    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-envoy\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:00:00.934282 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.934286    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm8px\" (UniqueName: \"kubernetes.io/projected/17652cd9-6ad9-4318-bee9-a09da4162e94-kube-api-access-sm8px\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:00:00.934282 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:00.934296    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17652cd9-6ad9-4318-bee9-a09da4162e94-istio-podinfo\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:00:01.815594 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:01.815558    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17652cd9-6ad9-4318-bee9-a09da4162e94" path="/var/lib/kubelet/pods/17652cd9-6ad9-4318-bee9-a09da4162e94/volumes"
Sep 11 20:00:03.861736 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:03.861683    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819280-m658h" event={"ID":"a112ec37-694a-4a5b-990d-10885ef25e6f","Type":"ContainerStarted","Data":"5e868141f9884bb0d5a7f7b8dfabd67d8893990c4c876c9a5385a9185bb9edcf"}
Sep 11 20:00:03.875872 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:03.875820    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819280-m658h" podStartSLOduration=0.83341176 podStartE2EDuration="3.875805223s" totalImagesPullingTime="3.042393463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:03.874305614 +0000 UTC m=+2020.491475106" watchObservedRunningTime="2026-09-11 20:00:03.875805223 +0000 UTC m=+2020.492974714"
Sep 11 20:00:04.868490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:04.868454    2558 generic.go:356] "Generic (PLEG): container finished" podID="a112ec37-694a-4a5b-990d-10885ef25e6f" containerID="5e868141f9884bb0d5a7f7b8dfabd67d8893990c4c876c9a5385a9185bb9edcf" exitCode=0
Sep 11 20:00:04.869193 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:04.868539    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819280-m658h" event={"ID":"a112ec37-694a-4a5b-990d-10885ef25e6f","Type":"ContainerDied","Data":"5e868141f9884bb0d5a7f7b8dfabd67d8893990c4c876c9a5385a9185bb9edcf"}
Sep 11 20:00:05.993682 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:05.993652    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819280-m658h"
Sep 11 20:00:06.881051 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:06.881012    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819280-m658h" event={"ID":"a112ec37-694a-4a5b-990d-10885ef25e6f","Type":"ContainerDied","Data":"fe49418fb903a4841cb9fb8f80b9de630964fb609f153b7bf6664b7bd60e2e02"}
Sep 11 20:00:06.881221 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:06.881072    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe49418fb903a4841cb9fb8f80b9de630964fb609f153b7bf6664b7bd60e2e02"
Sep 11 20:00:06.881221 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:06.881023    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819280-m658h"
Sep 11 20:00:12.569616 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.569574    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"]
Sep 11 20:00:12.570094 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.570076    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a112ec37-694a-4a5b-990d-10885ef25e6f" containerName="cleanup"
Sep 11 20:00:12.570151 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.570097    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a112ec37-694a-4a5b-990d-10885ef25e6f" containerName="cleanup"
Sep 11 20:00:12.570151 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.570119    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17652cd9-6ad9-4318-bee9-a09da4162e94" containerName="istio-proxy"
Sep 11 20:00:12.570151 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.570125    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="17652cd9-6ad9-4318-bee9-a09da4162e94" containerName="istio-proxy"
Sep 11 20:00:12.570151 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.570134    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17652cd9-6ad9-4318-bee9-a09da4162e94" containerName="istio-proxy"
Sep 11 20:00:12.570151 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.570140    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="17652cd9-6ad9-4318-bee9-a09da4162e94" containerName="istio-proxy"
Sep 11 20:00:12.570334 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.570205    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="17652cd9-6ad9-4318-bee9-a09da4162e94" containerName="istio-proxy"
Sep 11 20:00:12.570334 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.570220    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="17652cd9-6ad9-4318-bee9-a09da4162e94" containerName="istio-proxy"
Sep 11 20:00:12.570334 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.570233    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a112ec37-694a-4a5b-990d-10885ef25e6f" containerName="cleanup"
Sep 11 20:00:12.573534 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.573510    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:12.575911 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.575889    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-58c78c\"/\"openshift-service-ca.crt\""
Sep 11 20:00:12.576599 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.576556    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-58c78c\"/\"sub-test-model-58c78c-kserve-self-signed-certs\""
Sep 11 20:00:12.576599 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.576589    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-58c78c\"/\"kube-root-ca.crt\""
Sep 11 20:00:12.576799 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.576776    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-58c78c\"/\"default-dockercfg-pct5c\""
Sep 11 20:00:12.583150 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.583123    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"]
Sep 11 20:00:12.749925 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.749878    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-tmp-dir\") pod \"sub-test-model-58c78c-kserve-76c88b6755-bc8v7\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:12.749925 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.749919    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-dshm\") pod \"sub-test-model-58c78c-kserve-76c88b6755-bc8v7\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:12.750214 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.749943    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-home\") pod \"sub-test-model-58c78c-kserve-76c88b6755-bc8v7\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:12.750214 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.749959    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-tls-certs\") pod \"sub-test-model-58c78c-kserve-76c88b6755-bc8v7\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:12.750214 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.750055    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-model-cache\") pod \"sub-test-model-58c78c-kserve-76c88b6755-bc8v7\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:12.750214 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.750111    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fstl4\" (UniqueName: \"kubernetes.io/projected/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-kube-api-access-fstl4\") pod \"sub-test-model-58c78c-kserve-76c88b6755-bc8v7\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:12.850894 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.850792    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fstl4\" (UniqueName: \"kubernetes.io/projected/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-kube-api-access-fstl4\") pod \"sub-test-model-58c78c-kserve-76c88b6755-bc8v7\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:12.850894 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.850853    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-tmp-dir\") pod \"sub-test-model-58c78c-kserve-76c88b6755-bc8v7\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:12.851100 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.851032    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-dshm\") pod \"sub-test-model-58c78c-kserve-76c88b6755-bc8v7\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:12.851100 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.851084    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-home\") pod \"sub-test-model-58c78c-kserve-76c88b6755-bc8v7\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:12.851169 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.851113    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-tls-certs\") pod \"sub-test-model-58c78c-kserve-76c88b6755-bc8v7\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:12.851233 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.851217    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-model-cache\") pod \"sub-test-model-58c78c-kserve-76c88b6755-bc8v7\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:12.851283 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.851254    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-tmp-dir\") pod \"sub-test-model-58c78c-kserve-76c88b6755-bc8v7\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:12.851377 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.851357    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-home\") pod \"sub-test-model-58c78c-kserve-76c88b6755-bc8v7\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:12.851568 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.851543    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-model-cache\") pod \"sub-test-model-58c78c-kserve-76c88b6755-bc8v7\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:12.853474 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.853454    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-dshm\") pod \"sub-test-model-58c78c-kserve-76c88b6755-bc8v7\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:12.853780 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.853760    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-tls-certs\") pod \"sub-test-model-58c78c-kserve-76c88b6755-bc8v7\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:12.859040 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.859019    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fstl4\" (UniqueName: \"kubernetes.io/projected/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-kube-api-access-fstl4\") pod \"sub-test-model-58c78c-kserve-76c88b6755-bc8v7\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:12.885059 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:12.885018    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:13.158541 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:13.158511    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"]
Sep 11 20:00:13.160049 ip-10-0-138-43 kubenswrapper[2558]: W0911 20:00:13.160018    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14f3d016_47d7_46ae_b67f_c4f42ff5dbcd.slice/crio-2eade0398c5fd45007d67a80d1c00ae84ca8b079c51836afc5eff9a9ad0c639d WatchSource:0}: Error finding container 2eade0398c5fd45007d67a80d1c00ae84ca8b079c51836afc5eff9a9ad0c639d: Status 404 returned error can't find the container with id 2eade0398c5fd45007d67a80d1c00ae84ca8b079c51836afc5eff9a9ad0c639d
Sep 11 20:00:13.918990 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:13.918949    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7" event={"ID":"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd","Type":"ContainerStarted","Data":"81a6d7739ee89b7744e9ecd20cafdf85a91007bf10e61113fce36e9b7c5a3a2e"}
Sep 11 20:00:13.918990 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:13.918993    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7" event={"ID":"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd","Type":"ContainerStarted","Data":"2eade0398c5fd45007d67a80d1c00ae84ca8b079c51836afc5eff9a9ad0c639d"}
Sep 11 20:00:13.952415 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:13.952362    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7" podStartSLOduration=1.9523473930000002 podStartE2EDuration="1.952347393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:13.950144547 +0000 UTC m=+2030.567314039" watchObservedRunningTime="2026-09-11 20:00:13.952347393 +0000 UTC m=+2030.569516921"
Sep 11 20:00:22.885791 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:22.885748    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:22.886314 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:22.885811    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:22.898936 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:22.898904    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:00:22.974765 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:00:22.974730    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:01:33.945319 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:33.945284    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"]
Sep 11 20:01:41.567507 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.567473    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"]
Sep 11 20:01:41.570059 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.570042    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:41.573046 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.573015    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-58c78c\"/\"rate-test-model-58c78c-kserve-self-signed-certs\""
Sep 11 20:01:41.590005 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.589974    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"]
Sep 11 20:01:41.613031 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.612996    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-tmp-dir\") pod \"rate-test-model-58c78c-kserve-65dcdfc64c-84k8h\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:41.613184 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.613043    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cff41771-3e7e-418c-8997-4ecaa3b40ac8-tls-certs\") pod \"rate-test-model-58c78c-kserve-65dcdfc64c-84k8h\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:41.613184 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.613068    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-model-cache\") pod \"rate-test-model-58c78c-kserve-65dcdfc64c-84k8h\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:41.613184 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.613167    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-dshm\") pod \"rate-test-model-58c78c-kserve-65dcdfc64c-84k8h\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:41.613355 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.613286    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgk7h\" (UniqueName: \"kubernetes.io/projected/cff41771-3e7e-418c-8997-4ecaa3b40ac8-kube-api-access-bgk7h\") pod \"rate-test-model-58c78c-kserve-65dcdfc64c-84k8h\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:41.613410 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.613356    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-home\") pod \"rate-test-model-58c78c-kserve-65dcdfc64c-84k8h\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:41.714460 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.714421    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-dshm\") pod \"rate-test-model-58c78c-kserve-65dcdfc64c-84k8h\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:41.714647 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.714487    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgk7h\" (UniqueName: \"kubernetes.io/projected/cff41771-3e7e-418c-8997-4ecaa3b40ac8-kube-api-access-bgk7h\") pod \"rate-test-model-58c78c-kserve-65dcdfc64c-84k8h\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:41.714647 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.714602    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-home\") pod \"rate-test-model-58c78c-kserve-65dcdfc64c-84k8h\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:41.714815 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.714725    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-tmp-dir\") pod \"rate-test-model-58c78c-kserve-65dcdfc64c-84k8h\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:41.715014 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.714968    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cff41771-3e7e-418c-8997-4ecaa3b40ac8-tls-certs\") pod \"rate-test-model-58c78c-kserve-65dcdfc64c-84k8h\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:41.715388 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.715108    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-home\") pod \"rate-test-model-58c78c-kserve-65dcdfc64c-84k8h\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:41.715487 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.715190    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-tmp-dir\") pod \"rate-test-model-58c78c-kserve-65dcdfc64c-84k8h\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:41.715624 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.715601    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-model-cache\") pod \"rate-test-model-58c78c-kserve-65dcdfc64c-84k8h\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:41.715826 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.715764    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-model-cache\") pod \"rate-test-model-58c78c-kserve-65dcdfc64c-84k8h\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:41.717306 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.717276    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-dshm\") pod \"rate-test-model-58c78c-kserve-65dcdfc64c-84k8h\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:41.718206 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.718184    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cff41771-3e7e-418c-8997-4ecaa3b40ac8-tls-certs\") pod \"rate-test-model-58c78c-kserve-65dcdfc64c-84k8h\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:41.722802 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.722780    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgk7h\" (UniqueName: \"kubernetes.io/projected/cff41771-3e7e-418c-8997-4ecaa3b40ac8-kube-api-access-bgk7h\") pod \"rate-test-model-58c78c-kserve-65dcdfc64c-84k8h\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") " pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:41.880238 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:41.880156    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:42.155581 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:42.155555    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"]
Sep 11 20:01:42.157294 ip-10-0-138-43 kubenswrapper[2558]: W0911 20:01:42.157266    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcff41771_3e7e_418c_8997_4ecaa3b40ac8.slice/crio-1557af6bc9033ebf9625721cd753ae295a8796f959c3214425a11b9a7313e58a WatchSource:0}: Error finding container 1557af6bc9033ebf9625721cd753ae295a8796f959c3214425a11b9a7313e58a: Status 404 returned error can't find the container with id 1557af6bc9033ebf9625721cd753ae295a8796f959c3214425a11b9a7313e58a
Sep 11 20:01:42.364938 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:42.364900    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h" event={"ID":"cff41771-3e7e-418c-8997-4ecaa3b40ac8","Type":"ContainerStarted","Data":"49ad01dfdc845ebbffa97b59b4b843776c2e488057651bb34657031e535dd153"}
Sep 11 20:01:42.365114 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:42.364951    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h" event={"ID":"cff41771-3e7e-418c-8997-4ecaa3b40ac8","Type":"ContainerStarted","Data":"1557af6bc9033ebf9625721cd753ae295a8796f959c3214425a11b9a7313e58a"}
Sep 11 20:01:42.383613 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:42.383569    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h" podStartSLOduration=1.383553698 podStartE2EDuration="1.383553698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:42.381134988 +0000 UTC m=+2118.998304480" watchObservedRunningTime="2026-09-11 20:01:42.383553698 +0000 UTC m=+2119.000723188"
Sep 11 20:01:48.964060 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:48.964014    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7" podUID="14f3d016-47d7-46ae-b67f-c4f42ff5dbcd" containerName="main" containerID="cri-o://81a6d7739ee89b7744e9ecd20cafdf85a91007bf10e61113fce36e9b7c5a3a2e" gracePeriod=45
Sep 11 20:01:49.216734 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.216658    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:01:49.287016 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.286980    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-dshm\") pod \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") "
Sep 11 20:01:49.287186 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.287024    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fstl4\" (UniqueName: \"kubernetes.io/projected/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-kube-api-access-fstl4\") pod \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") "
Sep 11 20:01:49.287186 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.287089    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-model-cache\") pod \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") "
Sep 11 20:01:49.287186 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.287125    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-tmp-dir\") pod \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") "
Sep 11 20:01:49.287186 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.287142    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-home\") pod \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") "
Sep 11 20:01:49.287405 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.287187    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-tls-certs\") pod \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\" (UID: \"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd\") "
Sep 11 20:01:49.287405 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.287345    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "14f3d016-47d7-46ae-b67f-c4f42ff5dbcd" (UID: "14f3d016-47d7-46ae-b67f-c4f42ff5dbcd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:49.287405 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.287381    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-model-cache" (OuterVolumeSpecName: "model-cache") pod "14f3d016-47d7-46ae-b67f-c4f42ff5dbcd" (UID: "14f3d016-47d7-46ae-b67f-c4f42ff5dbcd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:49.287534 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.287458    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-home" (OuterVolumeSpecName: "home") pod "14f3d016-47d7-46ae-b67f-c4f42ff5dbcd" (UID: "14f3d016-47d7-46ae-b67f-c4f42ff5dbcd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:49.287631 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.287609    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-model-cache\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:01:49.287631 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.287632    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-tmp-dir\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:01:49.287827 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.287640    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-home\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:01:49.289363 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.289339    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14f3d016-47d7-46ae-b67f-c4f42ff5dbcd" (UID: "14f3d016-47d7-46ae-b67f-c4f42ff5dbcd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:01:49.289639 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.289613    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-dshm" (OuterVolumeSpecName: "dshm") pod "14f3d016-47d7-46ae-b67f-c4f42ff5dbcd" (UID: "14f3d016-47d7-46ae-b67f-c4f42ff5dbcd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:01:49.289754 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.289635    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-kube-api-access-fstl4" (OuterVolumeSpecName: "kube-api-access-fstl4") pod "14f3d016-47d7-46ae-b67f-c4f42ff5dbcd" (UID: "14f3d016-47d7-46ae-b67f-c4f42ff5dbcd"). InnerVolumeSpecName "kube-api-access-fstl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:01:49.388550 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.388504    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-tls-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:01:49.388550 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.388544    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-dshm\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:01:49.388550 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.388554    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fstl4\" (UniqueName: \"kubernetes.io/projected/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd-kube-api-access-fstl4\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:01:49.400824 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.400795    2558 generic.go:356] "Generic (PLEG): container finished" podID="14f3d016-47d7-46ae-b67f-c4f42ff5dbcd" containerID="81a6d7739ee89b7744e9ecd20cafdf85a91007bf10e61113fce36e9b7c5a3a2e" exitCode=0
Sep 11 20:01:49.400979 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.400844    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"
Sep 11 20:01:49.400979 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.400883    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7" event={"ID":"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd","Type":"ContainerDied","Data":"81a6d7739ee89b7744e9ecd20cafdf85a91007bf10e61113fce36e9b7c5a3a2e"}
Sep 11 20:01:49.400979 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.400931    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7" event={"ID":"14f3d016-47d7-46ae-b67f-c4f42ff5dbcd","Type":"ContainerDied","Data":"2eade0398c5fd45007d67a80d1c00ae84ca8b079c51836afc5eff9a9ad0c639d"}
Sep 11 20:01:49.400979 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.400947    2558 scope.go:117] "RemoveContainer" containerID="81a6d7739ee89b7744e9ecd20cafdf85a91007bf10e61113fce36e9b7c5a3a2e"
Sep 11 20:01:49.410499 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.410472    2558 scope.go:117] "RemoveContainer" containerID="81a6d7739ee89b7744e9ecd20cafdf85a91007bf10e61113fce36e9b7c5a3a2e"
Sep 11 20:01:49.410789 ip-10-0-138-43 kubenswrapper[2558]: E0911 20:01:49.410771    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81a6d7739ee89b7744e9ecd20cafdf85a91007bf10e61113fce36e9b7c5a3a2e\": container with ID starting with 81a6d7739ee89b7744e9ecd20cafdf85a91007bf10e61113fce36e9b7c5a3a2e not found: ID does not exist" containerID="81a6d7739ee89b7744e9ecd20cafdf85a91007bf10e61113fce36e9b7c5a3a2e"
Sep 11 20:01:49.410844 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.410801    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81a6d7739ee89b7744e9ecd20cafdf85a91007bf10e61113fce36e9b7c5a3a2e"} err="failed to get container status \"81a6d7739ee89b7744e9ecd20cafdf85a91007bf10e61113fce36e9b7c5a3a2e\": rpc error: code = NotFound desc = could not find container \"81a6d7739ee89b7744e9ecd20cafdf85a91007bf10e61113fce36e9b7c5a3a2e\": container with ID starting with 81a6d7739ee89b7744e9ecd20cafdf85a91007bf10e61113fce36e9b7c5a3a2e not found: ID does not exist"
Sep 11 20:01:49.424231 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.424206    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"]
Sep 11 20:01:49.427567 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.427547    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-58c78c/sub-test-model-58c78c-kserve-76c88b6755-bc8v7"]
Sep 11 20:01:49.814049 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:49.814016    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14f3d016-47d7-46ae-b67f-c4f42ff5dbcd" path="/var/lib/kubelet/pods/14f3d016-47d7-46ae-b67f-c4f42ff5dbcd/volumes"
Sep 11 20:01:51.880683 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:51.880642    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:51.881086 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:51.880694    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:51.893358 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:51.893330    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:01:52.428341 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:01:52.428310    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:02:43.157950 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:43.157915    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"]
Sep 11 20:02:44.890281 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:44.890250    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"]
Sep 11 20:02:44.890912 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:44.890892    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14f3d016-47d7-46ae-b67f-c4f42ff5dbcd" containerName="main"
Sep 11 20:02:44.891009 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:44.890914    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="14f3d016-47d7-46ae-b67f-c4f42ff5dbcd" containerName="main"
Sep 11 20:02:44.891009 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:44.891001    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="14f3d016-47d7-46ae-b67f-c4f42ff5dbcd" containerName="main"
Sep 11 20:02:44.893007 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:44.892985    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:44.895331 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:44.895307    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-81ca1f-openshift-default-dockercfg-kskbx\""
Sep 11 20:02:44.904789 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:44.904761    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"]
Sep 11 20:02:45.042229 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.042192    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-workload-socket\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.042425 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.042250    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-envoy\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.042425 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.042279    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-podinfo\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.042425 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.042299    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c279f8b-c969-4989-9104-d53f6826f7cb-istiod-ca-cert\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.042425 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.042331    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-token\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.042425 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.042413    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vv7d\" (UniqueName: \"kubernetes.io/projected/4c279f8b-c969-4989-9104-d53f6826f7cb-kube-api-access-5vv7d\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.042631 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.042432    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-workload-certs\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.042631 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.042453    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-credential-socket\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.042631 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.042481    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-data\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.144045 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.143954    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-token\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.144194 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.144117    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vv7d\" (UniqueName: \"kubernetes.io/projected/4c279f8b-c969-4989-9104-d53f6826f7cb-kube-api-access-5vv7d\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.144194 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.144163    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-workload-certs\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.144268 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.144196    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-credential-socket\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.144268 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.144244    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-data\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.144349 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.144279    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-workload-socket\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.144397 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.144360    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-envoy\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.144442 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.144403    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-podinfo\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.144442 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.144432    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c279f8b-c969-4989-9104-d53f6826f7cb-istiod-ca-cert\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.144645 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.144620    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-workload-certs\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.144645 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.144638    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-data\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.144834 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.144682    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-credential-socket\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.144897 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.144846    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-workload-socket\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.145136 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.145109    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c279f8b-c969-4989-9104-d53f6826f7cb-istiod-ca-cert\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.146801 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.146778    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-envoy\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.146971 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.146948    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-podinfo\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.152134 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.152105    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-token\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.152253 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.152233    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vv7d\" (UniqueName: \"kubernetes.io/projected/4c279f8b-c969-4989-9104-d53f6826f7cb-kube-api-access-5vv7d\") pod \"e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") " pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.204648 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.204618    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:45.477860 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.477827    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"]
Sep 11 20:02:45.479032 ip-10-0-138-43 kubenswrapper[2558]: W0911 20:02:45.479004    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c279f8b_c969_4989_9104_d53f6826f7cb.slice/crio-61a6c21b97965a891dd66505f5099b68e7d3ad4118b4ab3733432dbfd8417858 WatchSource:0}: Error finding container 61a6c21b97965a891dd66505f5099b68e7d3ad4118b4ab3733432dbfd8417858: Status 404 returned error can't find the container with id 61a6c21b97965a891dd66505f5099b68e7d3ad4118b4ab3733432dbfd8417858
Sep 11 20:02:45.481317 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.481282    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 20:02:45.481417 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.481362    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 20:02:45.481417 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.481408    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 20:02:45.688614 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.688573    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9" event={"ID":"4c279f8b-c969-4989-9104-d53f6826f7cb","Type":"ContainerStarted","Data":"f1bdf838c4331762b0faf7008eaf0f752e0fda6b4c0e94541bbca2c59a2c17d2"}
Sep 11 20:02:45.688810 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.688672    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9" event={"ID":"4c279f8b-c969-4989-9104-d53f6826f7cb","Type":"ContainerStarted","Data":"61a6c21b97965a891dd66505f5099b68e7d3ad4118b4ab3733432dbfd8417858"}
Sep 11 20:02:45.709331 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:45.709273    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9" podStartSLOduration=1.7092518110000001 podStartE2EDuration="1.709251811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:02:45.707355406 +0000 UTC m=+2182.324524909" watchObservedRunningTime="2026-09-11 20:02:45.709251811 +0000 UTC m=+2182.326421302"
Sep 11 20:02:46.205246 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:46.205208    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:46.209948 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:46.209922    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:46.694083 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:46.694051    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:46.695136 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:46.695117    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:02:49.432781 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:49.432741    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"]
Sep 11 20:02:49.435836 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:49.435812    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"
Sep 11 20:02:49.437853 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:49.437832    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-81ca1f-serving-cert\""
Sep 11 20:02:49.446056 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:49.446031    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"]
Sep 11 20:02:49.593979 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:49.593942    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jthlp\" (UniqueName: \"kubernetes.io/projected/45ebeb41-95d0-40da-a784-e5256504a37e-kube-api-access-jthlp\") pod \"maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh\" (UID: \"45ebeb41-95d0-40da-a784-e5256504a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"
Sep 11 20:02:49.594155 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:49.594055    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ebeb41-95d0-40da-a784-e5256504a37e-metrics-certs\") pod \"maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh\" (UID: \"45ebeb41-95d0-40da-a784-e5256504a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"
Sep 11 20:02:49.594155 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:49.594132    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ebeb41-95d0-40da-a784-e5256504a37e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh\" (UID: \"45ebeb41-95d0-40da-a784-e5256504a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"
Sep 11 20:02:49.594237 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:49.594184    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ebeb41-95d0-40da-a784-e5256504a37e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh\" (UID: \"45ebeb41-95d0-40da-a784-e5256504a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"
Sep 11 20:02:49.695178 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:49.695090    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ebeb41-95d0-40da-a784-e5256504a37e-metrics-certs\") pod \"maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh\" (UID: \"45ebeb41-95d0-40da-a784-e5256504a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"
Sep 11 20:02:49.695178 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:49.695156    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ebeb41-95d0-40da-a784-e5256504a37e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh\" (UID: \"45ebeb41-95d0-40da-a784-e5256504a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"
Sep 11 20:02:49.695357 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:49.695182    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ebeb41-95d0-40da-a784-e5256504a37e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh\" (UID: \"45ebeb41-95d0-40da-a784-e5256504a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"
Sep 11 20:02:49.695357 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:49.695220    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jthlp\" (UniqueName: \"kubernetes.io/projected/45ebeb41-95d0-40da-a784-e5256504a37e-kube-api-access-jthlp\") pod \"maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh\" (UID: \"45ebeb41-95d0-40da-a784-e5256504a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"
Sep 11 20:02:49.695974 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:49.695943    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ebeb41-95d0-40da-a784-e5256504a37e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh\" (UID: \"45ebeb41-95d0-40da-a784-e5256504a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"
Sep 11 20:02:49.697675 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:49.697652    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ebeb41-95d0-40da-a784-e5256504a37e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh\" (UID: \"45ebeb41-95d0-40da-a784-e5256504a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"
Sep 11 20:02:49.697675 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:49.697660    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ebeb41-95d0-40da-a784-e5256504a37e-metrics-certs\") pod \"maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh\" (UID: \"45ebeb41-95d0-40da-a784-e5256504a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"
Sep 11 20:02:49.704026 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:49.703997    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jthlp\" (UniqueName: \"kubernetes.io/projected/45ebeb41-95d0-40da-a784-e5256504a37e-kube-api-access-jthlp\") pod \"maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh\" (UID: \"45ebeb41-95d0-40da-a784-e5256504a37e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"
Sep 11 20:02:49.748960 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:49.748928    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"
Sep 11 20:02:50.024317 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:50.024288    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"]
Sep 11 20:02:50.025186 ip-10-0-138-43 kubenswrapper[2558]: W0911 20:02:50.025160    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ebeb41_95d0_40da_a784_e5256504a37e.slice/crio-d41949bd48061e11f1dfd15078bc5506929829c8c5ddaa0ca1d6a999dabd7574 WatchSource:0}: Error finding container d41949bd48061e11f1dfd15078bc5506929829c8c5ddaa0ca1d6a999dabd7574: Status 404 returned error can't find the container with id d41949bd48061e11f1dfd15078bc5506929829c8c5ddaa0ca1d6a999dabd7574
Sep 11 20:02:50.026973 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:50.026953    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 20:02:50.716497 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:50.716466    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh" event={"ID":"45ebeb41-95d0-40da-a784-e5256504a37e","Type":"ContainerStarted","Data":"926d458a9b4ce185e64ba3945b7e6986c44be470be0501f02772f04e548931d5"}
Sep 11 20:02:50.716925 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:50.716517    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh" event={"ID":"45ebeb41-95d0-40da-a784-e5256504a37e","Type":"ContainerStarted","Data":"d41949bd48061e11f1dfd15078bc5506929829c8c5ddaa0ca1d6a999dabd7574"}
Sep 11 20:02:50.716925 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:50.716624    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"
Sep 11 20:02:50.736159 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:50.736110    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh" podStartSLOduration=1.262068119 podStartE2EDuration="1.736095331s" totalImagesPullingTime="474.027212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:02:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:02:50.733124821 +0000 UTC m=+2187.350294313" watchObservedRunningTime="2026-09-11 20:02:50.736095331 +0000 UTC m=+2187.353264822"
Sep 11 20:02:56.726762 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:56.726732    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"
Sep 11 20:02:58.176931 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.176886    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h" podUID="cff41771-3e7e-418c-8997-4ecaa3b40ac8" containerName="main" containerID="cri-o://49ad01dfdc845ebbffa97b59b4b843776c2e488057651bb34657031e535dd153" gracePeriod=45
Sep 11 20:02:58.484200 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.484172    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:02:58.587944 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.587907    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-model-cache\") pod \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") "
Sep 11 20:02:58.588147 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.588031    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-tmp-dir\") pod \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") "
Sep 11 20:02:58.588147 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.588085    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cff41771-3e7e-418c-8997-4ecaa3b40ac8-tls-certs\") pod \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") "
Sep 11 20:02:58.588275 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.588146    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-home\") pod \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") "
Sep 11 20:02:58.588275 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.588151    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-model-cache" (OuterVolumeSpecName: "model-cache") pod "cff41771-3e7e-418c-8997-4ecaa3b40ac8" (UID: "cff41771-3e7e-418c-8997-4ecaa3b40ac8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:02:58.588275 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.588173    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-dshm\") pod \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") "
Sep 11 20:02:58.588275 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.588242    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgk7h\" (UniqueName: \"kubernetes.io/projected/cff41771-3e7e-418c-8997-4ecaa3b40ac8-kube-api-access-bgk7h\") pod \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\" (UID: \"cff41771-3e7e-418c-8997-4ecaa3b40ac8\") "
Sep 11 20:02:58.588275 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.588246    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cff41771-3e7e-418c-8997-4ecaa3b40ac8" (UID: "cff41771-3e7e-418c-8997-4ecaa3b40ac8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:02:58.588521 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.588293    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-home" (OuterVolumeSpecName: "home") pod "cff41771-3e7e-418c-8997-4ecaa3b40ac8" (UID: "cff41771-3e7e-418c-8997-4ecaa3b40ac8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:02:58.588585 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.588570    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-tmp-dir\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:02:58.588639 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.588592    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-home\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:02:58.588639 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.588606    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-model-cache\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:02:58.590094 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.590067    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cff41771-3e7e-418c-8997-4ecaa3b40ac8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cff41771-3e7e-418c-8997-4ecaa3b40ac8" (UID: "cff41771-3e7e-418c-8997-4ecaa3b40ac8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:02:58.590223 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.590200    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff41771-3e7e-418c-8997-4ecaa3b40ac8-kube-api-access-bgk7h" (OuterVolumeSpecName: "kube-api-access-bgk7h") pod "cff41771-3e7e-418c-8997-4ecaa3b40ac8" (UID: "cff41771-3e7e-418c-8997-4ecaa3b40ac8"). InnerVolumeSpecName "kube-api-access-bgk7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:02:58.590288 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.590205    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-dshm" (OuterVolumeSpecName: "dshm") pod "cff41771-3e7e-418c-8997-4ecaa3b40ac8" (UID: "cff41771-3e7e-418c-8997-4ecaa3b40ac8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:02:58.689193 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.689158    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgk7h\" (UniqueName: \"kubernetes.io/projected/cff41771-3e7e-418c-8997-4ecaa3b40ac8-kube-api-access-bgk7h\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:02:58.689193 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.689188    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cff41771-3e7e-418c-8997-4ecaa3b40ac8-tls-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:02:58.689193 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.689199    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cff41771-3e7e-418c-8997-4ecaa3b40ac8-dshm\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:02:58.768888 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.768802    2558 generic.go:356] "Generic (PLEG): container finished" podID="cff41771-3e7e-418c-8997-4ecaa3b40ac8" containerID="49ad01dfdc845ebbffa97b59b4b843776c2e488057651bb34657031e535dd153" exitCode=0
Sep 11 20:02:58.768888 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.768843    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"
Sep 11 20:02:58.769090 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.768849    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h" event={"ID":"cff41771-3e7e-418c-8997-4ecaa3b40ac8","Type":"ContainerDied","Data":"49ad01dfdc845ebbffa97b59b4b843776c2e488057651bb34657031e535dd153"}
Sep 11 20:02:58.769090 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.768953    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h" event={"ID":"cff41771-3e7e-418c-8997-4ecaa3b40ac8","Type":"ContainerDied","Data":"1557af6bc9033ebf9625721cd753ae295a8796f959c3214425a11b9a7313e58a"}
Sep 11 20:02:58.769090 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.768974    2558 scope.go:117] "RemoveContainer" containerID="49ad01dfdc845ebbffa97b59b4b843776c2e488057651bb34657031e535dd153"
Sep 11 20:02:58.778182 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.778167    2558 scope.go:117] "RemoveContainer" containerID="49ad01dfdc845ebbffa97b59b4b843776c2e488057651bb34657031e535dd153"
Sep 11 20:02:58.778451 ip-10-0-138-43 kubenswrapper[2558]: E0911 20:02:58.778430    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ad01dfdc845ebbffa97b59b4b843776c2e488057651bb34657031e535dd153\": container with ID starting with 49ad01dfdc845ebbffa97b59b4b843776c2e488057651bb34657031e535dd153 not found: ID does not exist" containerID="49ad01dfdc845ebbffa97b59b4b843776c2e488057651bb34657031e535dd153"
Sep 11 20:02:58.778526 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.778458    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ad01dfdc845ebbffa97b59b4b843776c2e488057651bb34657031e535dd153"} err="failed to get container status \"49ad01dfdc845ebbffa97b59b4b843776c2e488057651bb34657031e535dd153\": rpc error: code = NotFound desc = could not find container \"49ad01dfdc845ebbffa97b59b4b843776c2e488057651bb34657031e535dd153\": container with ID starting with 49ad01dfdc845ebbffa97b59b4b843776c2e488057651bb34657031e535dd153 not found: ID does not exist"
Sep 11 20:02:58.795302 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.795277    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"]
Sep 11 20:02:58.797797 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:58.797776    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-58c78c/rate-test-model-58c78c-kserve-65dcdfc64c-84k8h"]
Sep 11 20:02:59.813909 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:02:59.813875    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cff41771-3e7e-418c-8997-4ecaa3b40ac8" path="/var/lib/kubelet/pods/cff41771-3e7e-418c-8997-4ecaa3b40ac8/volumes"
Sep 11 20:03:02.735903 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:02.735861    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 11 20:03:02.736285 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:02.735942    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9" podUID="4c279f8b-c969-4989-9104-d53f6826f7cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 11 20:03:03.803025 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:03.802993    2558 generic.go:356] "Generic (PLEG): container finished" podID="4c279f8b-c969-4989-9104-d53f6826f7cb" containerID="f1bdf838c4331762b0faf7008eaf0f752e0fda6b4c0e94541bbca2c59a2c17d2" exitCode=0
Sep 11 20:03:03.803468 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:03.803042    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9" event={"ID":"4c279f8b-c969-4989-9104-d53f6826f7cb","Type":"ContainerDied","Data":"f1bdf838c4331762b0faf7008eaf0f752e0fda6b4c0e94541bbca2c59a2c17d2"}
Sep 11 20:03:03.803468 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:03.803422    2558 scope.go:117] "RemoveContainer" containerID="f1bdf838c4331762b0faf7008eaf0f752e0fda6b4c0e94541bbca2c59a2c17d2"
Sep 11 20:03:03.804499 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:03.804462    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 20:03:03.804553 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:03.804532    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 20:03:03.804602 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:03.804564    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236200Ki","pods":"250"}
Sep 11 20:03:04.815845 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:04.815800    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9" event={"ID":"4c279f8b-c969-4989-9104-d53f6826f7cb","Type":"ContainerStarted","Data":"8209a026de1ca31773ccd4993a636bf8ab27ba6736e5aaa2981ac7304db2c9a9"}
Sep 11 20:03:05.205805 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:05.205764    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:03:05.206002 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:05.205977    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 11 20:03:05.206050 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:05.206029    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9" podUID="4c279f8b-c969-4989-9104-d53f6826f7cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 11 20:03:06.205967 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:06.205930    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 11 20:03:06.206361 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:06.205990    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9" podUID="4c279f8b-c969-4989-9104-d53f6826f7cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 11 20:03:07.205544 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:07.205504    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 11 20:03:07.205764 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:07.205562    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9" podUID="4c279f8b-c969-4989-9104-d53f6826f7cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 11 20:03:08.205694 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:08.205658    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 11 20:03:08.206131 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:08.205734    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9" podUID="4c279f8b-c969-4989-9104-d53f6826f7cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 11 20:03:09.206066 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:09.206027    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 11 20:03:09.206469 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:09.206082    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9" podUID="4c279f8b-c969-4989-9104-d53f6826f7cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 11 20:03:10.224496 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:10.224447    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:03:10.224990 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:10.224888    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:03:10.225858 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:10.225834    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:03:19.002793 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.002744    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"]
Sep 11 20:03:19.003301 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.003185    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cff41771-3e7e-418c-8997-4ecaa3b40ac8" containerName="main"
Sep 11 20:03:19.003301 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.003198    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="cff41771-3e7e-418c-8997-4ecaa3b40ac8" containerName="main"
Sep 11 20:03:19.003301 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.003269    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="cff41771-3e7e-418c-8997-4ecaa3b40ac8" containerName="main"
Sep 11 20:03:19.005617 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.005593    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"
Sep 11 20:03:19.008208 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.008179    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5df0d4-serving-cert\""
Sep 11 20:03:19.018439 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.018402    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"]
Sep 11 20:03:19.104815 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.104775    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6ltn\" (UniqueName: \"kubernetes.io/projected/7863fcb4-6750-4e9c-acc7-08d7cbce6665-kube-api-access-m6ltn\") pod \"maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r\" (UID: \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"
Sep 11 20:03:19.104973 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.104869    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7863fcb4-6750-4e9c-acc7-08d7cbce6665-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r\" (UID: \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"
Sep 11 20:03:19.105040 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.104969    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7863fcb4-6750-4e9c-acc7-08d7cbce6665-metrics-certs\") pod \"maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r\" (UID: \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"
Sep 11 20:03:19.105040 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.105013    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7863fcb4-6750-4e9c-acc7-08d7cbce6665-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r\" (UID: \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"
Sep 11 20:03:19.205532 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.205487    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7863fcb4-6750-4e9c-acc7-08d7cbce6665-metrics-certs\") pod \"maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r\" (UID: \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"
Sep 11 20:03:19.205532 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.205541    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7863fcb4-6750-4e9c-acc7-08d7cbce6665-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r\" (UID: \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"
Sep 11 20:03:19.205852 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.205581    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6ltn\" (UniqueName: \"kubernetes.io/projected/7863fcb4-6750-4e9c-acc7-08d7cbce6665-kube-api-access-m6ltn\") pod \"maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r\" (UID: \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"
Sep 11 20:03:19.205852 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.205635    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7863fcb4-6750-4e9c-acc7-08d7cbce6665-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r\" (UID: \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"
Sep 11 20:03:19.205852 ip-10-0-138-43 kubenswrapper[2558]: E0911 20:03:19.205726    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-serving-cert: secret "maas-api-e2e-ipp-b-5df0d4-serving-cert" not found
Sep 11 20:03:19.205852 ip-10-0-138-43 kubenswrapper[2558]: E0911 20:03:19.205798    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7863fcb4-6750-4e9c-acc7-08d7cbce6665-maas-api-tls podName:7863fcb4-6750-4e9c-acc7-08d7cbce6665 nodeName:}" failed. No retries permitted until 2026-09-11 20:03:19.705779736 +0000 UTC m=+2216.322949215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7863fcb4-6750-4e9c-acc7-08d7cbce6665-maas-api-tls") pod "maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r" (UID: "7863fcb4-6750-4e9c-acc7-08d7cbce6665") : secret "maas-api-e2e-ipp-b-5df0d4-serving-cert" not found
Sep 11 20:03:19.206350 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.206330    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7863fcb4-6750-4e9c-acc7-08d7cbce6665-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r\" (UID: \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"
Sep 11 20:03:19.208051 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.208025    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7863fcb4-6750-4e9c-acc7-08d7cbce6665-metrics-certs\") pod \"maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r\" (UID: \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"
Sep 11 20:03:19.215059 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.215015    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6ltn\" (UniqueName: \"kubernetes.io/projected/7863fcb4-6750-4e9c-acc7-08d7cbce6665-kube-api-access-m6ltn\") pod \"maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r\" (UID: \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"
Sep 11 20:03:19.711410 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.711374    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7863fcb4-6750-4e9c-acc7-08d7cbce6665-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r\" (UID: \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"
Sep 11 20:03:19.713932 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.713905    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7863fcb4-6750-4e9c-acc7-08d7cbce6665-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r\" (UID: \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"
Sep 11 20:03:19.778085 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.778049    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7"]
Sep 11 20:03:19.781202 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.781177    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7"
Sep 11 20:03:19.784081 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.784053    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5df0d4-dockercfg-wc4j2\""
Sep 11 20:03:19.784217 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.784094    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5df0d4\""
Sep 11 20:03:19.794248 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.794219    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7"]
Sep 11 20:03:19.913028 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.912984    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcmxs\" (UniqueName: \"kubernetes.io/projected/40c1021e-2e62-4adb-82a1-082cfec7a572-kube-api-access-wcmxs\") pod \"payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7\" (UID: \"40c1021e-2e62-4adb-82a1-082cfec7a572\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7"
Sep 11 20:03:19.913200 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.913102    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c1021e-2e62-4adb-82a1-082cfec7a572-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7\" (UID: \"40c1021e-2e62-4adb-82a1-082cfec7a572\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7"
Sep 11 20:03:19.917874 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:19.917832    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"
Sep 11 20:03:20.014397 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:20.014366    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcmxs\" (UniqueName: \"kubernetes.io/projected/40c1021e-2e62-4adb-82a1-082cfec7a572-kube-api-access-wcmxs\") pod \"payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7\" (UID: \"40c1021e-2e62-4adb-82a1-082cfec7a572\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7"
Sep 11 20:03:20.014787 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:20.014463    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c1021e-2e62-4adb-82a1-082cfec7a572-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7\" (UID: \"40c1021e-2e62-4adb-82a1-082cfec7a572\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7"
Sep 11 20:03:20.015185 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:20.015166    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c1021e-2e62-4adb-82a1-082cfec7a572-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7\" (UID: \"40c1021e-2e62-4adb-82a1-082cfec7a572\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7"
Sep 11 20:03:20.024220 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:20.024188    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcmxs\" (UniqueName: \"kubernetes.io/projected/40c1021e-2e62-4adb-82a1-082cfec7a572-kube-api-access-wcmxs\") pod \"payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7\" (UID: \"40c1021e-2e62-4adb-82a1-082cfec7a572\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7"
Sep 11 20:03:20.095308 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:20.095266    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7"
Sep 11 20:03:20.190412 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:20.190385    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"]
Sep 11 20:03:20.193311 ip-10-0-138-43 kubenswrapper[2558]: W0911 20:03:20.193277    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7863fcb4_6750_4e9c_acc7_08d7cbce6665.slice/crio-e5d9a103be73ee49391bde0b7179ca8f34a6895f0909a6214ca5eebedd611447 WatchSource:0}: Error finding container e5d9a103be73ee49391bde0b7179ca8f34a6895f0909a6214ca5eebedd611447: Status 404 returned error can't find the container with id e5d9a103be73ee49391bde0b7179ca8f34a6895f0909a6214ca5eebedd611447
Sep 11 20:03:20.367647 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:20.367620    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7"]
Sep 11 20:03:20.369724 ip-10-0-138-43 kubenswrapper[2558]: W0911 20:03:20.369673    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c1021e_2e62_4adb_82a1_082cfec7a572.slice/crio-0c4a50d50951a73b976542bca74e31dca9ef187178189eecfb2a702025e68627 WatchSource:0}: Error finding container 0c4a50d50951a73b976542bca74e31dca9ef187178189eecfb2a702025e68627: Status 404 returned error can't find the container with id 0c4a50d50951a73b976542bca74e31dca9ef187178189eecfb2a702025e68627
Sep 11 20:03:20.906227 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:20.906187    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r" event={"ID":"7863fcb4-6750-4e9c-acc7-08d7cbce6665","Type":"ContainerStarted","Data":"a21376345c5cc2eb80d76e8414a9b709a368c35290e26be5926f9fab507e044f"}
Sep 11 20:03:20.906445 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:20.906250    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r" event={"ID":"7863fcb4-6750-4e9c-acc7-08d7cbce6665","Type":"ContainerStarted","Data":"e5d9a103be73ee49391bde0b7179ca8f34a6895f0909a6214ca5eebedd611447"}
Sep 11 20:03:20.906445 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:20.906337    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"
Sep 11 20:03:20.908044 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:20.908022    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7" event={"ID":"40c1021e-2e62-4adb-82a1-082cfec7a572","Type":"ContainerStarted","Data":"c2b236443f3494b857de4eb09ea7daf812590212f3ccc3413c1d7379433c98ae"}
Sep 11 20:03:20.908183 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:20.908069    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7" event={"ID":"40c1021e-2e62-4adb-82a1-082cfec7a572","Type":"ContainerStarted","Data":"0c4a50d50951a73b976542bca74e31dca9ef187178189eecfb2a702025e68627"}
Sep 11 20:03:20.908183 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:20.908102    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7"
Sep 11 20:03:20.924801 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:20.924749    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r" podStartSLOduration=2.583788563 podStartE2EDuration="2.924702707s" totalImagesPullingTime="340.914144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:03:20.922387723 +0000 UTC m=+2217.539557240" watchObservedRunningTime="2026-09-11 20:03:20.924702707 +0000 UTC m=+2217.541872197"
Sep 11 20:03:20.940246 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:20.940182    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7" podStartSLOduration=1.940160279 podStartE2EDuration="1.940160279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:03:20.936366211 +0000 UTC m=+2217.553535724" watchObservedRunningTime="2026-09-11 20:03:20.940160279 +0000 UTC m=+2217.557329771"
Sep 11 20:03:26.919073 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:26.919042    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"
Sep 11 20:03:31.916591 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:31.916555    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7"
Sep 11 20:03:51.672193 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:51.672115    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xptvp_cb31f235-193f-4244-9c95-06e347d981ee/payload-processing/0.log"
Sep 11 20:03:51.790625 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:03:51.790587    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7_40c1021e-2e62-4adb-82a1-082cfec7a572/payload-processing/0.log"
Sep 11 20:04:53.236724 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:04:53.236682    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xptvp_cb31f235-193f-4244-9c95-06e347d981ee/payload-processing/0.log"
Sep 11 20:04:53.357797 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:04:53.357767    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7_40c1021e-2e62-4adb-82a1-082cfec7a572/payload-processing/0.log"
Sep 11 20:06:04.884933 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:04.884883    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"]
Sep 11 20:06:04.885999 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:04.885961    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh" podUID="45ebeb41-95d0-40da-a784-e5256504a37e" containerName="maas-api" containerID="cri-o://926d458a9b4ce185e64ba3945b7e6986c44be470be0501f02772f04e548931d5" gracePeriod=30
Sep 11 20:06:05.157762 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.157736    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"
Sep 11 20:06:05.280110 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.280072    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ebeb41-95d0-40da-a784-e5256504a37e-openshift-service-ca\") pod \"45ebeb41-95d0-40da-a784-e5256504a37e\" (UID: \"45ebeb41-95d0-40da-a784-e5256504a37e\") "
Sep 11 20:06:05.280325 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.280207    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ebeb41-95d0-40da-a784-e5256504a37e-metrics-certs\") pod \"45ebeb41-95d0-40da-a784-e5256504a37e\" (UID: \"45ebeb41-95d0-40da-a784-e5256504a37e\") "
Sep 11 20:06:05.280325 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.280310    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jthlp\" (UniqueName: \"kubernetes.io/projected/45ebeb41-95d0-40da-a784-e5256504a37e-kube-api-access-jthlp\") pod \"45ebeb41-95d0-40da-a784-e5256504a37e\" (UID: \"45ebeb41-95d0-40da-a784-e5256504a37e\") "
Sep 11 20:06:05.280465 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.280341    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ebeb41-95d0-40da-a784-e5256504a37e-maas-api-tls\") pod \"45ebeb41-95d0-40da-a784-e5256504a37e\" (UID: \"45ebeb41-95d0-40da-a784-e5256504a37e\") "
Sep 11 20:06:05.280602 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.280567    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ebeb41-95d0-40da-a784-e5256504a37e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "45ebeb41-95d0-40da-a784-e5256504a37e" (UID: "45ebeb41-95d0-40da-a784-e5256504a37e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:06:05.280771 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.280740    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ebeb41-95d0-40da-a784-e5256504a37e-openshift-service-ca\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:06:05.282558 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.282524    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ebeb41-95d0-40da-a784-e5256504a37e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "45ebeb41-95d0-40da-a784-e5256504a37e" (UID: "45ebeb41-95d0-40da-a784-e5256504a37e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:06:05.282558 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.282546    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ebeb41-95d0-40da-a784-e5256504a37e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "45ebeb41-95d0-40da-a784-e5256504a37e" (UID: "45ebeb41-95d0-40da-a784-e5256504a37e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:06:05.282888 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.282868    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ebeb41-95d0-40da-a784-e5256504a37e-kube-api-access-jthlp" (OuterVolumeSpecName: "kube-api-access-jthlp") pod "45ebeb41-95d0-40da-a784-e5256504a37e" (UID: "45ebeb41-95d0-40da-a784-e5256504a37e"). InnerVolumeSpecName "kube-api-access-jthlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:06:05.381329 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.381298    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45ebeb41-95d0-40da-a784-e5256504a37e-metrics-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:06:05.381329 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.381326    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jthlp\" (UniqueName: \"kubernetes.io/projected/45ebeb41-95d0-40da-a784-e5256504a37e-kube-api-access-jthlp\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:06:05.381329 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.381337    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45ebeb41-95d0-40da-a784-e5256504a37e-maas-api-tls\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:06:05.768616 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.768581    2558 generic.go:356] "Generic (PLEG): container finished" podID="45ebeb41-95d0-40da-a784-e5256504a37e" containerID="926d458a9b4ce185e64ba3945b7e6986c44be470be0501f02772f04e548931d5" exitCode=0
Sep 11 20:06:05.768830 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.768640    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"
Sep 11 20:06:05.768830 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.768671    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh" event={"ID":"45ebeb41-95d0-40da-a784-e5256504a37e","Type":"ContainerDied","Data":"926d458a9b4ce185e64ba3945b7e6986c44be470be0501f02772f04e548931d5"}
Sep 11 20:06:05.768830 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.768763    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh" event={"ID":"45ebeb41-95d0-40da-a784-e5256504a37e","Type":"ContainerDied","Data":"d41949bd48061e11f1dfd15078bc5506929829c8c5ddaa0ca1d6a999dabd7574"}
Sep 11 20:06:05.768830 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.768795    2558 scope.go:117] "RemoveContainer" containerID="926d458a9b4ce185e64ba3945b7e6986c44be470be0501f02772f04e548931d5"
Sep 11 20:06:05.778933 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.778915    2558 scope.go:117] "RemoveContainer" containerID="926d458a9b4ce185e64ba3945b7e6986c44be470be0501f02772f04e548931d5"
Sep 11 20:06:05.779213 ip-10-0-138-43 kubenswrapper[2558]: E0911 20:06:05.779194    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"926d458a9b4ce185e64ba3945b7e6986c44be470be0501f02772f04e548931d5\": container with ID starting with 926d458a9b4ce185e64ba3945b7e6986c44be470be0501f02772f04e548931d5 not found: ID does not exist" containerID="926d458a9b4ce185e64ba3945b7e6986c44be470be0501f02772f04e548931d5"
Sep 11 20:06:05.779283 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.779223    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926d458a9b4ce185e64ba3945b7e6986c44be470be0501f02772f04e548931d5"} err="failed to get container status \"926d458a9b4ce185e64ba3945b7e6986c44be470be0501f02772f04e548931d5\": rpc error: code = NotFound desc = could not find container \"926d458a9b4ce185e64ba3945b7e6986c44be470be0501f02772f04e548931d5\": container with ID starting with 926d458a9b4ce185e64ba3945b7e6986c44be470be0501f02772f04e548931d5 not found: ID does not exist"
Sep 11 20:06:05.798193 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.798158    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"]
Sep 11 20:06:05.803888 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.803858    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-81ca1f-69989996c4-hlpkh"]
Sep 11 20:06:05.814735 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:05.814693    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45ebeb41-95d0-40da-a784-e5256504a37e" path="/var/lib/kubelet/pods/45ebeb41-95d0-40da-a784-e5256504a37e/volumes"
Sep 11 20:06:53.847220 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:53.847122    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"]
Sep 11 20:06:53.847776 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:53.847448    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9" podUID="4c279f8b-c969-4989-9104-d53f6826f7cb" containerName="istio-proxy" containerID="cri-o://8209a026de1ca31773ccd4993a636bf8ab27ba6736e5aaa2981ac7304db2c9a9" gracePeriod=30
Sep 11 20:06:55.227574 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:55.227535    2558 patch_prober.go:28] interesting pod/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 20:06:55.228016 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:55.227599    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9" podUID="4c279f8b-c969-4989-9104-d53f6826f7cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 20:06:59.047465 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.047427    2558 generic.go:356] "Generic (PLEG): container finished" podID="4c279f8b-c969-4989-9104-d53f6826f7cb" containerID="8209a026de1ca31773ccd4993a636bf8ab27ba6736e5aaa2981ac7304db2c9a9" exitCode=0
Sep 11 20:06:59.047894 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.047496    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9" event={"ID":"4c279f8b-c969-4989-9104-d53f6826f7cb","Type":"ContainerDied","Data":"8209a026de1ca31773ccd4993a636bf8ab27ba6736e5aaa2981ac7304db2c9a9"}
Sep 11 20:06:59.047894 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.047566    2558 scope.go:117] "RemoveContainer" containerID="f1bdf838c4331762b0faf7008eaf0f752e0fda6b4c0e94541bbca2c59a2c17d2"
Sep 11 20:06:59.108114 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.108088    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:06:59.131798 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.131700    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-workload-certs\") pod \"4c279f8b-c969-4989-9104-d53f6826f7cb\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") "
Sep 11 20:06:59.131798 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.131771    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-envoy\") pod \"4c279f8b-c969-4989-9104-d53f6826f7cb\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") "
Sep 11 20:06:59.132015 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.131864    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vv7d\" (UniqueName: \"kubernetes.io/projected/4c279f8b-c969-4989-9104-d53f6826f7cb-kube-api-access-5vv7d\") pod \"4c279f8b-c969-4989-9104-d53f6826f7cb\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") "
Sep 11 20:06:59.132015 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.131896    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-podinfo\") pod \"4c279f8b-c969-4989-9104-d53f6826f7cb\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") "
Sep 11 20:06:59.132015 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.131955    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-workload-socket\") pod \"4c279f8b-c969-4989-9104-d53f6826f7cb\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") "
Sep 11 20:06:59.132015 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.131979    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c279f8b-c969-4989-9104-d53f6826f7cb-istiod-ca-cert\") pod \"4c279f8b-c969-4989-9104-d53f6826f7cb\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") "
Sep 11 20:06:59.132231 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.132037    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-data\") pod \"4c279f8b-c969-4989-9104-d53f6826f7cb\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") "
Sep 11 20:06:59.132231 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.132066    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-credential-socket\") pod \"4c279f8b-c969-4989-9104-d53f6826f7cb\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") "
Sep 11 20:06:59.132231 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.132096    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-token\") pod \"4c279f8b-c969-4989-9104-d53f6826f7cb\" (UID: \"4c279f8b-c969-4989-9104-d53f6826f7cb\") "
Sep 11 20:06:59.132231 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.132121    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4c279f8b-c969-4989-9104-d53f6826f7cb" (UID: "4c279f8b-c969-4989-9104-d53f6826f7cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:06:59.132445 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.132403    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-workload-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:06:59.132445 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.132432    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-data" (OuterVolumeSpecName: "istio-data") pod "4c279f8b-c969-4989-9104-d53f6826f7cb" (UID: "4c279f8b-c969-4989-9104-d53f6826f7cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:06:59.132560 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.132451    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c279f8b-c969-4989-9104-d53f6826f7cb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4c279f8b-c969-4989-9104-d53f6826f7cb" (UID: "4c279f8b-c969-4989-9104-d53f6826f7cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:06:59.132666 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.132645    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4c279f8b-c969-4989-9104-d53f6826f7cb" (UID: "4c279f8b-c969-4989-9104-d53f6826f7cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:06:59.132748 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.132669    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4c279f8b-c969-4989-9104-d53f6826f7cb" (UID: "4c279f8b-c969-4989-9104-d53f6826f7cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:06:59.134806 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.134770    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-token" (OuterVolumeSpecName: "istio-token") pod "4c279f8b-c969-4989-9104-d53f6826f7cb" (UID: "4c279f8b-c969-4989-9104-d53f6826f7cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:06:59.134806 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.134780    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c279f8b-c969-4989-9104-d53f6826f7cb-kube-api-access-5vv7d" (OuterVolumeSpecName: "kube-api-access-5vv7d") pod "4c279f8b-c969-4989-9104-d53f6826f7cb" (UID: "4c279f8b-c969-4989-9104-d53f6826f7cb"). InnerVolumeSpecName "kube-api-access-5vv7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:06:59.134991 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.134851    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4c279f8b-c969-4989-9104-d53f6826f7cb" (UID: "4c279f8b-c969-4989-9104-d53f6826f7cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 20:06:59.134991 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.134886    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4c279f8b-c969-4989-9104-d53f6826f7cb" (UID: "4c279f8b-c969-4989-9104-d53f6826f7cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:06:59.233552 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.233513    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-envoy\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:06:59.233552 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.233546    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vv7d\" (UniqueName: \"kubernetes.io/projected/4c279f8b-c969-4989-9104-d53f6826f7cb-kube-api-access-5vv7d\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:06:59.233802 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.233561    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-podinfo\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:06:59.233802 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.233574    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-workload-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:06:59.233802 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.233586    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c279f8b-c969-4989-9104-d53f6826f7cb-istiod-ca-cert\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:06:59.233802 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.233597    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-data\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:06:59.233802 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.233607    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c279f8b-c969-4989-9104-d53f6826f7cb-credential-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:06:59.233802 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:06:59.233618    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c279f8b-c969-4989-9104-d53f6826f7cb-istio-token\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:00.054411 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:00.054377    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"
Sep 11 20:07:00.054411 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:00.054401    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9" event={"ID":"4c279f8b-c969-4989-9104-d53f6826f7cb","Type":"ContainerDied","Data":"61a6c21b97965a891dd66505f5099b68e7d3ad4118b4ab3733432dbfd8417858"}
Sep 11 20:07:00.054993 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:00.054467    2558 scope.go:117] "RemoveContainer" containerID="8209a026de1ca31773ccd4993a636bf8ab27ba6736e5aaa2981ac7304db2c9a9"
Sep 11 20:07:00.072004 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:00.071923    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"]
Sep 11 20:07:00.076073 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:00.076040    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-81ca1f-openshift-default-5f596b5bb6-8x7z9"]
Sep 11 20:07:01.814392 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:01.814357    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c279f8b-c969-4989-9104-d53f6826f7cb" path="/var/lib/kubelet/pods/4c279f8b-c969-4989-9104-d53f6826f7cb/volumes"
Sep 11 20:07:04.818699 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:04.818653    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"]
Sep 11 20:07:04.819233 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:04.818931    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r" podUID="7863fcb4-6750-4e9c-acc7-08d7cbce6665" containerName="maas-api" containerID="cri-o://a21376345c5cc2eb80d76e8414a9b709a368c35290e26be5926f9fab507e044f" gracePeriod=30
Sep 11 20:07:04.897672 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:04.897623    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7"]
Sep 11 20:07:04.898021 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:04.897962    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7" podUID="40c1021e-2e62-4adb-82a1-082cfec7a572" containerName="payload-processing" containerID="cri-o://c2b236443f3494b857de4eb09ea7daf812590212f3ccc3413c1d7379433c98ae" gracePeriod=30
Sep 11 20:07:05.087735 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.087068    2558 generic.go:356] "Generic (PLEG): container finished" podID="7863fcb4-6750-4e9c-acc7-08d7cbce6665" containerID="a21376345c5cc2eb80d76e8414a9b709a368c35290e26be5926f9fab507e044f" exitCode=0
Sep 11 20:07:05.087735 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.087164    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r" event={"ID":"7863fcb4-6750-4e9c-acc7-08d7cbce6665","Type":"ContainerDied","Data":"a21376345c5cc2eb80d76e8414a9b709a368c35290e26be5926f9fab507e044f"}
Sep 11 20:07:05.089587 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.089554    2558 generic.go:356] "Generic (PLEG): container finished" podID="40c1021e-2e62-4adb-82a1-082cfec7a572" containerID="c2b236443f3494b857de4eb09ea7daf812590212f3ccc3413c1d7379433c98ae" exitCode=0
Sep 11 20:07:05.089697 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.089627    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7" event={"ID":"40c1021e-2e62-4adb-82a1-082cfec7a572","Type":"ContainerDied","Data":"c2b236443f3494b857de4eb09ea7daf812590212f3ccc3413c1d7379433c98ae"}
Sep 11 20:07:05.147557 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.147495    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"
Sep 11 20:07:05.183586 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.183557    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7"
Sep 11 20:07:05.196216 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.196188    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7863fcb4-6750-4e9c-acc7-08d7cbce6665-openshift-service-ca\") pod \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\" (UID: \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\") "
Sep 11 20:07:05.196385 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.196229    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7863fcb4-6750-4e9c-acc7-08d7cbce6665-maas-api-tls\") pod \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\" (UID: \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\") "
Sep 11 20:07:05.196385 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.196309    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6ltn\" (UniqueName: \"kubernetes.io/projected/7863fcb4-6750-4e9c-acc7-08d7cbce6665-kube-api-access-m6ltn\") pod \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\" (UID: \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\") "
Sep 11 20:07:05.196385 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.196347    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7863fcb4-6750-4e9c-acc7-08d7cbce6665-metrics-certs\") pod \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\" (UID: \"7863fcb4-6750-4e9c-acc7-08d7cbce6665\") "
Sep 11 20:07:05.196636 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.196605    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7863fcb4-6750-4e9c-acc7-08d7cbce6665-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7863fcb4-6750-4e9c-acc7-08d7cbce6665" (UID: "7863fcb4-6750-4e9c-acc7-08d7cbce6665"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:07:05.196788 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.196756    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7863fcb4-6750-4e9c-acc7-08d7cbce6665-openshift-service-ca\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:05.198448 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.198412    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7863fcb4-6750-4e9c-acc7-08d7cbce6665-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7863fcb4-6750-4e9c-acc7-08d7cbce6665" (UID: "7863fcb4-6750-4e9c-acc7-08d7cbce6665"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:07:05.198538 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.198486    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7863fcb4-6750-4e9c-acc7-08d7cbce6665-kube-api-access-m6ltn" (OuterVolumeSpecName: "kube-api-access-m6ltn") pod "7863fcb4-6750-4e9c-acc7-08d7cbce6665" (UID: "7863fcb4-6750-4e9c-acc7-08d7cbce6665"). InnerVolumeSpecName "kube-api-access-m6ltn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:07:05.198856 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.198835    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7863fcb4-6750-4e9c-acc7-08d7cbce6665-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7863fcb4-6750-4e9c-acc7-08d7cbce6665" (UID: "7863fcb4-6750-4e9c-acc7-08d7cbce6665"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:07:05.297996 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.297959    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c1021e-2e62-4adb-82a1-082cfec7a572-plugins-config-volume\") pod \"40c1021e-2e62-4adb-82a1-082cfec7a572\" (UID: \"40c1021e-2e62-4adb-82a1-082cfec7a572\") "
Sep 11 20:07:05.298171 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.298005    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcmxs\" (UniqueName: \"kubernetes.io/projected/40c1021e-2e62-4adb-82a1-082cfec7a572-kube-api-access-wcmxs\") pod \"40c1021e-2e62-4adb-82a1-082cfec7a572\" (UID: \"40c1021e-2e62-4adb-82a1-082cfec7a572\") "
Sep 11 20:07:05.298281 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.298264    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7863fcb4-6750-4e9c-acc7-08d7cbce6665-maas-api-tls\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:05.298281 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.298278    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6ltn\" (UniqueName: \"kubernetes.io/projected/7863fcb4-6750-4e9c-acc7-08d7cbce6665-kube-api-access-m6ltn\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:05.298392 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.298287    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7863fcb4-6750-4e9c-acc7-08d7cbce6665-metrics-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:05.298392 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.298315    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c1021e-2e62-4adb-82a1-082cfec7a572-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "40c1021e-2e62-4adb-82a1-082cfec7a572" (UID: "40c1021e-2e62-4adb-82a1-082cfec7a572"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:07:05.300476 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.300445    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c1021e-2e62-4adb-82a1-082cfec7a572-kube-api-access-wcmxs" (OuterVolumeSpecName: "kube-api-access-wcmxs") pod "40c1021e-2e62-4adb-82a1-082cfec7a572" (UID: "40c1021e-2e62-4adb-82a1-082cfec7a572"). InnerVolumeSpecName "kube-api-access-wcmxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:07:05.398912 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.398826    2558 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40c1021e-2e62-4adb-82a1-082cfec7a572-plugins-config-volume\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:05.398912 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:05.398856    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcmxs\" (UniqueName: \"kubernetes.io/projected/40c1021e-2e62-4adb-82a1-082cfec7a572-kube-api-access-wcmxs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:06.097416 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:06.097332    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"
Sep 11 20:07:06.097861 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:06.097331    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r" event={"ID":"7863fcb4-6750-4e9c-acc7-08d7cbce6665","Type":"ContainerDied","Data":"e5d9a103be73ee49391bde0b7179ca8f34a6895f0909a6214ca5eebedd611447"}
Sep 11 20:07:06.097861 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:06.097470    2558 scope.go:117] "RemoveContainer" containerID="a21376345c5cc2eb80d76e8414a9b709a368c35290e26be5926f9fab507e044f"
Sep 11 20:07:06.098999 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:06.098981    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7"
Sep 11 20:07:06.099111 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:06.099000    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7" event={"ID":"40c1021e-2e62-4adb-82a1-082cfec7a572","Type":"ContainerDied","Data":"0c4a50d50951a73b976542bca74e31dca9ef187178189eecfb2a702025e68627"}
Sep 11 20:07:06.106860 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:06.106756    2558 scope.go:117] "RemoveContainer" containerID="c2b236443f3494b857de4eb09ea7daf812590212f3ccc3413c1d7379433c98ae"
Sep 11 20:07:06.128786 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:06.128755    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7"]
Sep 11 20:07:06.136284 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:06.136255    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5df0d4-6bbd9f77cc-9jfm7"]
Sep 11 20:07:06.146514 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:06.146487    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"]
Sep 11 20:07:06.150841 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:06.150816    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df0d4-777fb687cd-7vn5r"]
Sep 11 20:07:07.814087 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:07.814056    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40c1021e-2e62-4adb-82a1-082cfec7a572" path="/var/lib/kubelet/pods/40c1021e-2e62-4adb-82a1-082cfec7a572/volumes"
Sep 11 20:07:07.814524 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:07.814460    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7863fcb4-6750-4e9c-acc7-08d7cbce6665" path="/var/lib/kubelet/pods/7863fcb4-6750-4e9c-acc7-08d7cbce6665/volumes"
Sep 11 20:07:29.332630 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:29.332595    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"]
Sep 11 20:07:29.333152 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:29.332881    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g" podUID="d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" containerName="istio-proxy" containerID="cri-o://1f4df180bba4a9f368e2075e9fa994e7c8589a9a0694e9b62a3c04cc9ff35cef" gracePeriod=30
Sep 11 20:07:32.522916 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:32.522880    2558 patch_prober.go:28] interesting pod/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 20:07:32.523314 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:32.522938    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g" podUID="d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 20:07:34.591440 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.591409    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 20:07:34.694121 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.694083    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-token\") pod \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") "
Sep 11 20:07:34.694281 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.694147    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-envoy\") pod \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") "
Sep 11 20:07:34.694281 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.694169    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-workload-socket\") pod \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") "
Sep 11 20:07:34.694281 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.694204    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-podinfo\") pod \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") "
Sep 11 20:07:34.694281 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.694224    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-credential-socket\") pod \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") "
Sep 11 20:07:34.694281 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.694240    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-workload-certs\") pod \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") "
Sep 11 20:07:34.694281 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.694270    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ljlg\" (UniqueName: \"kubernetes.io/projected/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-kube-api-access-7ljlg\") pod \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") "
Sep 11 20:07:34.694586 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.694313    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-data\") pod \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") "
Sep 11 20:07:34.694586 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.694354    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istiod-ca-cert\") pod \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\" (UID: \"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7\") "
Sep 11 20:07:34.694741 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.694654    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" (UID: "d5a0e675-d5b8-4b37-b40a-cd57e627b4b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:34.694741 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.694673    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" (UID: "d5a0e675-d5b8-4b37-b40a-cd57e627b4b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:34.694899 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.694870    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" (UID: "d5a0e675-d5b8-4b37-b40a-cd57e627b4b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:34.695033 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.694984    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" (UID: "d5a0e675-d5b8-4b37-b40a-cd57e627b4b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:07:34.695033 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.695008    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-data" (OuterVolumeSpecName: "istio-data") pod "d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" (UID: "d5a0e675-d5b8-4b37-b40a-cd57e627b4b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:34.696877 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.696856    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" (UID: "d5a0e675-d5b8-4b37-b40a-cd57e627b4b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:34.696969 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.696895    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-token" (OuterVolumeSpecName: "istio-token") pod "d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" (UID: "d5a0e675-d5b8-4b37-b40a-cd57e627b4b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:07:34.697021 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.696960    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" (UID: "d5a0e675-d5b8-4b37-b40a-cd57e627b4b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 20:07:34.697021 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.696973    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-kube-api-access-7ljlg" (OuterVolumeSpecName: "kube-api-access-7ljlg") pod "d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" (UID: "d5a0e675-d5b8-4b37-b40a-cd57e627b4b7"). InnerVolumeSpecName "kube-api-access-7ljlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:07:34.795123 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.795088    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-token\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:34.795299 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.795139    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-envoy\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:34.795299 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.795149    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-workload-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:34.795299 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.795158    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-podinfo\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:34.795299 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.795166    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-credential-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:34.795299 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.795175    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-workload-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:34.795299 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.795184    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ljlg\" (UniqueName: \"kubernetes.io/projected/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-kube-api-access-7ljlg\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:34.795299 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.795191    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istio-data\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:34.795299 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:34.795199    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7-istiod-ca-cert\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:35.255914 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:35.255880    2558 generic.go:356] "Generic (PLEG): container finished" podID="d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" containerID="1f4df180bba4a9f368e2075e9fa994e7c8589a9a0694e9b62a3c04cc9ff35cef" exitCode=0
Sep 11 20:07:35.256101 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:35.255932    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"
Sep 11 20:07:35.256101 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:35.255953    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g" event={"ID":"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7","Type":"ContainerDied","Data":"1f4df180bba4a9f368e2075e9fa994e7c8589a9a0694e9b62a3c04cc9ff35cef"}
Sep 11 20:07:35.256101 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:35.255997    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g" event={"ID":"d5a0e675-d5b8-4b37-b40a-cd57e627b4b7","Type":"ContainerDied","Data":"3e79b09caceb06340e7009a01dc8c9a395f49464d381414c3c8be9f83b5e1bf9"}
Sep 11 20:07:35.256101 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:35.256019    2558 scope.go:117] "RemoveContainer" containerID="1f4df180bba4a9f368e2075e9fa994e7c8589a9a0694e9b62a3c04cc9ff35cef"
Sep 11 20:07:35.265571 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:35.265553    2558 scope.go:117] "RemoveContainer" containerID="0c724a2aa1d16e2858302dcb758fdeecc0a6f35800b0e1e785b8cb57428e199f"
Sep 11 20:07:35.275463 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:35.275442    2558 scope.go:117] "RemoveContainer" containerID="1f4df180bba4a9f368e2075e9fa994e7c8589a9a0694e9b62a3c04cc9ff35cef"
Sep 11 20:07:35.275825 ip-10-0-138-43 kubenswrapper[2558]: E0911 20:07:35.275802    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4df180bba4a9f368e2075e9fa994e7c8589a9a0694e9b62a3c04cc9ff35cef\": container with ID starting with 1f4df180bba4a9f368e2075e9fa994e7c8589a9a0694e9b62a3c04cc9ff35cef not found: ID does not exist" containerID="1f4df180bba4a9f368e2075e9fa994e7c8589a9a0694e9b62a3c04cc9ff35cef"
Sep 11 20:07:35.275901 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:35.275832    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4df180bba4a9f368e2075e9fa994e7c8589a9a0694e9b62a3c04cc9ff35cef"} err="failed to get container status \"1f4df180bba4a9f368e2075e9fa994e7c8589a9a0694e9b62a3c04cc9ff35cef\": rpc error: code = NotFound desc = could not find container \"1f4df180bba4a9f368e2075e9fa994e7c8589a9a0694e9b62a3c04cc9ff35cef\": container with ID starting with 1f4df180bba4a9f368e2075e9fa994e7c8589a9a0694e9b62a3c04cc9ff35cef not found: ID does not exist"
Sep 11 20:07:35.275901 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:35.275850    2558 scope.go:117] "RemoveContainer" containerID="0c724a2aa1d16e2858302dcb758fdeecc0a6f35800b0e1e785b8cb57428e199f"
Sep 11 20:07:35.276190 ip-10-0-138-43 kubenswrapper[2558]: E0911 20:07:35.276170    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c724a2aa1d16e2858302dcb758fdeecc0a6f35800b0e1e785b8cb57428e199f\": container with ID starting with 0c724a2aa1d16e2858302dcb758fdeecc0a6f35800b0e1e785b8cb57428e199f not found: ID does not exist" containerID="0c724a2aa1d16e2858302dcb758fdeecc0a6f35800b0e1e785b8cb57428e199f"
Sep 11 20:07:35.276255 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:35.276197    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c724a2aa1d16e2858302dcb758fdeecc0a6f35800b0e1e785b8cb57428e199f"} err="failed to get container status \"0c724a2aa1d16e2858302dcb758fdeecc0a6f35800b0e1e785b8cb57428e199f\": rpc error: code = NotFound desc = could not find container \"0c724a2aa1d16e2858302dcb758fdeecc0a6f35800b0e1e785b8cb57428e199f\": container with ID starting with 0c724a2aa1d16e2858302dcb758fdeecc0a6f35800b0e1e785b8cb57428e199f not found: ID does not exist"
Sep 11 20:07:35.281631 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:35.281607    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"]
Sep 11 20:07:35.288179 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:35.288157    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-58c78c-openshift-default-66c648d89b-f2c2g"]
Sep 11 20:07:35.815218 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:35.815183    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" path="/var/lib/kubelet/pods/d5a0e675-d5b8-4b37-b40a-cd57e627b4b7/volumes"
Sep 11 20:07:46.988871 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:46.988831    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"]
Sep 11 20:07:46.989292 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:46.989063    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9" podUID="6eaede7b-f137-4263-942b-468e5fd15cc9" containerName="istio-proxy" containerID="cri-o://8747c4b39aa66482dab0ee6e7e2b4ba85100b9adbdc18f946197c635cda05960" gracePeriod=30
Sep 11 20:07:47.819793 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:47.819753    2558 patch_prober.go:28] interesting pod/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 20:07:47.819971 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:47.819836    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9" podUID="6eaede7b-f137-4263-942b-468e5fd15cc9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 20:07:49.479682 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:49.479646    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56b74cbc4d-v298d"]
Sep 11 20:07:49.480153 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:49.479876    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-56b74cbc4d-v298d" podUID="9f382032-fd78-40b4-b17f-3e0b530d1444" containerName="manager" containerID="cri-o://e9b1f0ccfaba82ec3ff415f7c7be7adeffb3885fff509c1f6146b65823626024" gracePeriod=30
Sep 11 20:07:49.732039 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:49.731967    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56b74cbc4d-v298d"
Sep 11 20:07:49.845222 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:49.845179    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkqtr\" (UniqueName: \"kubernetes.io/projected/9f382032-fd78-40b4-b17f-3e0b530d1444-kube-api-access-wkqtr\") pod \"9f382032-fd78-40b4-b17f-3e0b530d1444\" (UID: \"9f382032-fd78-40b4-b17f-3e0b530d1444\") "
Sep 11 20:07:49.845391 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:49.845230    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f382032-fd78-40b4-b17f-3e0b530d1444-webhook-cert\") pod \"9f382032-fd78-40b4-b17f-3e0b530d1444\" (UID: \"9f382032-fd78-40b4-b17f-3e0b530d1444\") "
Sep 11 20:07:49.845391 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:49.845268    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f382032-fd78-40b4-b17f-3e0b530d1444-metrics-certs\") pod \"9f382032-fd78-40b4-b17f-3e0b530d1444\" (UID: \"9f382032-fd78-40b4-b17f-3e0b530d1444\") "
Sep 11 20:07:49.847413 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:49.847381    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f382032-fd78-40b4-b17f-3e0b530d1444-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9f382032-fd78-40b4-b17f-3e0b530d1444" (UID: "9f382032-fd78-40b4-b17f-3e0b530d1444"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:07:49.847553 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:49.847410    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f382032-fd78-40b4-b17f-3e0b530d1444-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9f382032-fd78-40b4-b17f-3e0b530d1444" (UID: "9f382032-fd78-40b4-b17f-3e0b530d1444"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:07:49.847553 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:49.847420    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f382032-fd78-40b4-b17f-3e0b530d1444-kube-api-access-wkqtr" (OuterVolumeSpecName: "kube-api-access-wkqtr") pod "9f382032-fd78-40b4-b17f-3e0b530d1444" (UID: "9f382032-fd78-40b4-b17f-3e0b530d1444"). InnerVolumeSpecName "kube-api-access-wkqtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:07:49.946781 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:49.946741    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkqtr\" (UniqueName: \"kubernetes.io/projected/9f382032-fd78-40b4-b17f-3e0b530d1444-kube-api-access-wkqtr\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:49.946781 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:49.946775    2558 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f382032-fd78-40b4-b17f-3e0b530d1444-webhook-cert\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:49.946781 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:49.946787    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f382032-fd78-40b4-b17f-3e0b530d1444-metrics-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:50.331207 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:50.331171    2558 generic.go:356] "Generic (PLEG): container finished" podID="9f382032-fd78-40b4-b17f-3e0b530d1444" containerID="e9b1f0ccfaba82ec3ff415f7c7be7adeffb3885fff509c1f6146b65823626024" exitCode=0
Sep 11 20:07:50.331387 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:50.331225    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56b74cbc4d-v298d"
Sep 11 20:07:50.331387 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:50.331263    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56b74cbc4d-v298d" event={"ID":"9f382032-fd78-40b4-b17f-3e0b530d1444","Type":"ContainerDied","Data":"e9b1f0ccfaba82ec3ff415f7c7be7adeffb3885fff509c1f6146b65823626024"}
Sep 11 20:07:50.331387 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:50.331324    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56b74cbc4d-v298d" event={"ID":"9f382032-fd78-40b4-b17f-3e0b530d1444","Type":"ContainerDied","Data":"0f6050295cd7b31f15a1a97de982272a0b476ea1bdc23f04378355f40ad57a27"}
Sep 11 20:07:50.331387 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:50.331345    2558 scope.go:117] "RemoveContainer" containerID="e9b1f0ccfaba82ec3ff415f7c7be7adeffb3885fff509c1f6146b65823626024"
Sep 11 20:07:50.340907 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:50.340888    2558 scope.go:117] "RemoveContainer" containerID="e9b1f0ccfaba82ec3ff415f7c7be7adeffb3885fff509c1f6146b65823626024"
Sep 11 20:07:50.341187 ip-10-0-138-43 kubenswrapper[2558]: E0911 20:07:50.341166    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9b1f0ccfaba82ec3ff415f7c7be7adeffb3885fff509c1f6146b65823626024\": container with ID starting with e9b1f0ccfaba82ec3ff415f7c7be7adeffb3885fff509c1f6146b65823626024 not found: ID does not exist" containerID="e9b1f0ccfaba82ec3ff415f7c7be7adeffb3885fff509c1f6146b65823626024"
Sep 11 20:07:50.341240 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:50.341196    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b1f0ccfaba82ec3ff415f7c7be7adeffb3885fff509c1f6146b65823626024"} err="failed to get container status \"e9b1f0ccfaba82ec3ff415f7c7be7adeffb3885fff509c1f6146b65823626024\": rpc error: code = NotFound desc = could not find container \"e9b1f0ccfaba82ec3ff415f7c7be7adeffb3885fff509c1f6146b65823626024\": container with ID starting with e9b1f0ccfaba82ec3ff415f7c7be7adeffb3885fff509c1f6146b65823626024 not found: ID does not exist"
Sep 11 20:07:50.353308 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:50.353281    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56b74cbc4d-v298d"]
Sep 11 20:07:50.356999 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:50.356972    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-56b74cbc4d-v298d"]
Sep 11 20:07:51.813501 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:51.813460    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f382032-fd78-40b4-b17f-3e0b530d1444" path="/var/lib/kubelet/pods/9f382032-fd78-40b4-b17f-3e0b530d1444/volumes"
Sep 11 20:07:52.253995 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.253968    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 20:07:52.352017 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.351982    2558 generic.go:356] "Generic (PLEG): container finished" podID="6eaede7b-f137-4263-942b-468e5fd15cc9" containerID="8747c4b39aa66482dab0ee6e7e2b4ba85100b9adbdc18f946197c635cda05960" exitCode=0
Sep 11 20:07:52.352195 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.352032    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"
Sep 11 20:07:52.352195 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.352074    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9" event={"ID":"6eaede7b-f137-4263-942b-468e5fd15cc9","Type":"ContainerDied","Data":"8747c4b39aa66482dab0ee6e7e2b4ba85100b9adbdc18f946197c635cda05960"}
Sep 11 20:07:52.352195 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.352127    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9" event={"ID":"6eaede7b-f137-4263-942b-468e5fd15cc9","Type":"ContainerDied","Data":"459deee8eff461b38067a3779fd5cbba89e83057e1d32db0a094a877b37ba3cc"}
Sep 11 20:07:52.352195 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.352155    2558 scope.go:117] "RemoveContainer" containerID="8747c4b39aa66482dab0ee6e7e2b4ba85100b9adbdc18f946197c635cda05960"
Sep 11 20:07:52.360656 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.360635    2558 scope.go:117] "RemoveContainer" containerID="c3d2d43249f8553977b8dacb8e9d3b2788fd27062a5763f30084a48bae51c5c4"
Sep 11 20:07:52.368637 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.368621    2558 scope.go:117] "RemoveContainer" containerID="8747c4b39aa66482dab0ee6e7e2b4ba85100b9adbdc18f946197c635cda05960"
Sep 11 20:07:52.368929 ip-10-0-138-43 kubenswrapper[2558]: E0911 20:07:52.368910    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8747c4b39aa66482dab0ee6e7e2b4ba85100b9adbdc18f946197c635cda05960\": container with ID starting with 8747c4b39aa66482dab0ee6e7e2b4ba85100b9adbdc18f946197c635cda05960 not found: ID does not exist" containerID="8747c4b39aa66482dab0ee6e7e2b4ba85100b9adbdc18f946197c635cda05960"
Sep 11 20:07:52.368990 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.368936    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8747c4b39aa66482dab0ee6e7e2b4ba85100b9adbdc18f946197c635cda05960"} err="failed to get container status \"8747c4b39aa66482dab0ee6e7e2b4ba85100b9adbdc18f946197c635cda05960\": rpc error: code = NotFound desc = could not find container \"8747c4b39aa66482dab0ee6e7e2b4ba85100b9adbdc18f946197c635cda05960\": container with ID starting with 8747c4b39aa66482dab0ee6e7e2b4ba85100b9adbdc18f946197c635cda05960 not found: ID does not exist"
Sep 11 20:07:52.368990 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.368952    2558 scope.go:117] "RemoveContainer" containerID="c3d2d43249f8553977b8dacb8e9d3b2788fd27062a5763f30084a48bae51c5c4"
Sep 11 20:07:52.369236 ip-10-0-138-43 kubenswrapper[2558]: E0911 20:07:52.369216    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d2d43249f8553977b8dacb8e9d3b2788fd27062a5763f30084a48bae51c5c4\": container with ID starting with c3d2d43249f8553977b8dacb8e9d3b2788fd27062a5763f30084a48bae51c5c4 not found: ID does not exist" containerID="c3d2d43249f8553977b8dacb8e9d3b2788fd27062a5763f30084a48bae51c5c4"
Sep 11 20:07:52.369279 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.369243    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d2d43249f8553977b8dacb8e9d3b2788fd27062a5763f30084a48bae51c5c4"} err="failed to get container status \"c3d2d43249f8553977b8dacb8e9d3b2788fd27062a5763f30084a48bae51c5c4\": rpc error: code = NotFound desc = could not find container \"c3d2d43249f8553977b8dacb8e9d3b2788fd27062a5763f30084a48bae51c5c4\": container with ID starting with c3d2d43249f8553977b8dacb8e9d3b2788fd27062a5763f30084a48bae51c5c4 not found: ID does not exist"
Sep 11 20:07:52.371535 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.371517    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-credential-socket\") pod \"6eaede7b-f137-4263-942b-468e5fd15cc9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") "
Sep 11 20:07:52.371577 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.371545    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eaede7b-f137-4263-942b-468e5fd15cc9-istiod-ca-cert\") pod \"6eaede7b-f137-4263-942b-468e5fd15cc9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") "
Sep 11 20:07:52.371577 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.371564    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdp29\" (UniqueName: \"kubernetes.io/projected/6eaede7b-f137-4263-942b-468e5fd15cc9-kube-api-access-xdp29\") pod \"6eaede7b-f137-4263-942b-468e5fd15cc9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") "
Sep 11 20:07:52.371678 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.371593    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-envoy\") pod \"6eaede7b-f137-4263-942b-468e5fd15cc9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") "
Sep 11 20:07:52.371678 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.371609    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-podinfo\") pod \"6eaede7b-f137-4263-942b-468e5fd15cc9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") "
Sep 11 20:07:52.371678 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.371673    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-workload-certs\") pod \"6eaede7b-f137-4263-942b-468e5fd15cc9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") "
Sep 11 20:07:52.371889 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.371702    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-token\") pod \"6eaede7b-f137-4263-942b-468e5fd15cc9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") "
Sep 11 20:07:52.371889 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.371815    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-workload-socket\") pod \"6eaede7b-f137-4263-942b-468e5fd15cc9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") "
Sep 11 20:07:52.371889 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.371834    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6eaede7b-f137-4263-942b-468e5fd15cc9" (UID: "6eaede7b-f137-4263-942b-468e5fd15cc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:52.371889 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.371886    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-data\") pod \"6eaede7b-f137-4263-942b-468e5fd15cc9\" (UID: \"6eaede7b-f137-4263-942b-468e5fd15cc9\") "
Sep 11 20:07:52.372151 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.371909    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eaede7b-f137-4263-942b-468e5fd15cc9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6eaede7b-f137-4263-942b-468e5fd15cc9" (UID: "6eaede7b-f137-4263-942b-468e5fd15cc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:07:52.372151 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.372009    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6eaede7b-f137-4263-942b-468e5fd15cc9" (UID: "6eaede7b-f137-4263-942b-468e5fd15cc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:52.372261 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.372204    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-data" (OuterVolumeSpecName: "istio-data") pod "6eaede7b-f137-4263-942b-468e5fd15cc9" (UID: "6eaede7b-f137-4263-942b-468e5fd15cc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:52.372322 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.372267    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-workload-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:52.372322 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.372290    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-credential-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:52.372322 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.372306    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eaede7b-f137-4263-942b-468e5fd15cc9-istiod-ca-cert\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:52.372484 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.372325    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6eaede7b-f137-4263-942b-468e5fd15cc9" (UID: "6eaede7b-f137-4263-942b-468e5fd15cc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:52.374042 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.374021    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6eaede7b-f137-4263-942b-468e5fd15cc9" (UID: "6eaede7b-f137-4263-942b-468e5fd15cc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:07:52.374120 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.374100    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eaede7b-f137-4263-942b-468e5fd15cc9-kube-api-access-xdp29" (OuterVolumeSpecName: "kube-api-access-xdp29") pod "6eaede7b-f137-4263-942b-468e5fd15cc9" (UID: "6eaede7b-f137-4263-942b-468e5fd15cc9"). InnerVolumeSpecName "kube-api-access-xdp29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:07:52.374297 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.374278    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6eaede7b-f137-4263-942b-468e5fd15cc9" (UID: "6eaede7b-f137-4263-942b-468e5fd15cc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 20:07:52.374376 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.374333    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-token" (OuterVolumeSpecName: "istio-token") pod "6eaede7b-f137-4263-942b-468e5fd15cc9" (UID: "6eaede7b-f137-4263-942b-468e5fd15cc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:07:52.472989 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.472951    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-token\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:52.472989 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.472984    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-workload-socket\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:52.472989 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.472994    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-data\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:52.473235 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.473004    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdp29\" (UniqueName: \"kubernetes.io/projected/6eaede7b-f137-4263-942b-468e5fd15cc9-kube-api-access-xdp29\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:52.473235 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.473014    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-envoy\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:52.473235 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.473022    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eaede7b-f137-4263-942b-468e5fd15cc9-istio-podinfo\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:07:52.675485 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.675450    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"]
Sep 11 20:07:52.679136 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:52.679111    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae268e-openshift-default-7d76ddbd4-qm2w9"]
Sep 11 20:07:53.815182 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:07:53.815139    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eaede7b-f137-4263-942b-468e5fd15cc9" path="/var/lib/kubelet/pods/6eaede7b-f137-4263-942b-468e5fd15cc9/volumes"
Sep 11 20:08:51.114689 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.114649    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56b74cbc4d-w6tww"]
Sep 11 20:08:51.115123 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115049    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40c1021e-2e62-4adb-82a1-082cfec7a572" containerName="payload-processing"
Sep 11 20:08:51.115123 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115066    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="40c1021e-2e62-4adb-82a1-082cfec7a572" containerName="payload-processing"
Sep 11 20:08:51.115123 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115085    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" containerName="istio-proxy"
Sep 11 20:08:51.115123 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115095    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" containerName="istio-proxy"
Sep 11 20:08:51.115123 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115103    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" containerName="istio-proxy"
Sep 11 20:08:51.115123 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115109    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" containerName="istio-proxy"
Sep 11 20:08:51.115123 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115115    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eaede7b-f137-4263-942b-468e5fd15cc9" containerName="istio-proxy"
Sep 11 20:08:51.115123 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115120    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eaede7b-f137-4263-942b-468e5fd15cc9" containerName="istio-proxy"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115134    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c279f8b-c969-4989-9104-d53f6826f7cb" containerName="istio-proxy"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115139    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c279f8b-c969-4989-9104-d53f6826f7cb" containerName="istio-proxy"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115145    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45ebeb41-95d0-40da-a784-e5256504a37e" containerName="maas-api"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115150    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="45ebeb41-95d0-40da-a784-e5256504a37e" containerName="maas-api"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115159    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f382032-fd78-40b4-b17f-3e0b530d1444" containerName="manager"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115164    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f382032-fd78-40b4-b17f-3e0b530d1444" containerName="manager"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115170    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eaede7b-f137-4263-942b-468e5fd15cc9" containerName="istio-proxy"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115175    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eaede7b-f137-4263-942b-468e5fd15cc9" containerName="istio-proxy"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115182    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c279f8b-c969-4989-9104-d53f6826f7cb" containerName="istio-proxy"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115187    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c279f8b-c969-4989-9104-d53f6826f7cb" containerName="istio-proxy"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115192    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7863fcb4-6750-4e9c-acc7-08d7cbce6665" containerName="maas-api"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115197    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7863fcb4-6750-4e9c-acc7-08d7cbce6665" containerName="maas-api"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115257    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="45ebeb41-95d0-40da-a784-e5256504a37e" containerName="maas-api"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115265    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7863fcb4-6750-4e9c-acc7-08d7cbce6665" containerName="maas-api"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115271    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f382032-fd78-40b4-b17f-3e0b530d1444" containerName="manager"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115277    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" containerName="istio-proxy"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115282    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eaede7b-f137-4263-942b-468e5fd15cc9" containerName="istio-proxy"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115288    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5a0e675-d5b8-4b37-b40a-cd57e627b4b7" containerName="istio-proxy"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115294    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c279f8b-c969-4989-9104-d53f6826f7cb" containerName="istio-proxy"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115302    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c279f8b-c969-4989-9104-d53f6826f7cb" containerName="istio-proxy"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115311    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="40c1021e-2e62-4adb-82a1-082cfec7a572" containerName="payload-processing"
Sep 11 20:08:51.115490 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.115319    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eaede7b-f137-4263-942b-468e5fd15cc9" containerName="istio-proxy"
Sep 11 20:08:51.118366 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.118350    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56b74cbc4d-w6tww"
Sep 11 20:08:51.120823 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.120794    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 20:08:51.120823 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.120818    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 20:08:51.121015 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.120822    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-km7bf\""
Sep 11 20:08:51.121015 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.120826    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 20:08:51.131168 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.131144    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56b74cbc4d-w6tww"]
Sep 11 20:08:51.230013 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.229975    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/beac7486-f820-4057-8637-e1c647460219-webhook-cert\") pod \"maas-controller-56b74cbc4d-w6tww\" (UID: \"beac7486-f820-4057-8637-e1c647460219\") " pod="opendatahub/maas-controller-56b74cbc4d-w6tww"
Sep 11 20:08:51.230196 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.230039    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxk78\" (UniqueName: \"kubernetes.io/projected/beac7486-f820-4057-8637-e1c647460219-kube-api-access-dxk78\") pod \"maas-controller-56b74cbc4d-w6tww\" (UID: \"beac7486-f820-4057-8637-e1c647460219\") " pod="opendatahub/maas-controller-56b74cbc4d-w6tww"
Sep 11 20:08:51.230196 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.230098    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beac7486-f820-4057-8637-e1c647460219-metrics-certs\") pod \"maas-controller-56b74cbc4d-w6tww\" (UID: \"beac7486-f820-4057-8637-e1c647460219\") " pod="opendatahub/maas-controller-56b74cbc4d-w6tww"
Sep 11 20:08:51.331404 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.331358    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/beac7486-f820-4057-8637-e1c647460219-webhook-cert\") pod \"maas-controller-56b74cbc4d-w6tww\" (UID: \"beac7486-f820-4057-8637-e1c647460219\") " pod="opendatahub/maas-controller-56b74cbc4d-w6tww"
Sep 11 20:08:51.331586 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.331441    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxk78\" (UniqueName: \"kubernetes.io/projected/beac7486-f820-4057-8637-e1c647460219-kube-api-access-dxk78\") pod \"maas-controller-56b74cbc4d-w6tww\" (UID: \"beac7486-f820-4057-8637-e1c647460219\") " pod="opendatahub/maas-controller-56b74cbc4d-w6tww"
Sep 11 20:08:51.331586 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.331471    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beac7486-f820-4057-8637-e1c647460219-metrics-certs\") pod \"maas-controller-56b74cbc4d-w6tww\" (UID: \"beac7486-f820-4057-8637-e1c647460219\") " pod="opendatahub/maas-controller-56b74cbc4d-w6tww"
Sep 11 20:08:51.333832 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.333800    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/beac7486-f820-4057-8637-e1c647460219-webhook-cert\") pod \"maas-controller-56b74cbc4d-w6tww\" (UID: \"beac7486-f820-4057-8637-e1c647460219\") " pod="opendatahub/maas-controller-56b74cbc4d-w6tww"
Sep 11 20:08:51.333966 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.333943    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beac7486-f820-4057-8637-e1c647460219-metrics-certs\") pod \"maas-controller-56b74cbc4d-w6tww\" (UID: \"beac7486-f820-4057-8637-e1c647460219\") " pod="opendatahub/maas-controller-56b74cbc4d-w6tww"
Sep 11 20:08:51.340065 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.340043    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxk78\" (UniqueName: \"kubernetes.io/projected/beac7486-f820-4057-8637-e1c647460219-kube-api-access-dxk78\") pod \"maas-controller-56b74cbc4d-w6tww\" (UID: \"beac7486-f820-4057-8637-e1c647460219\") " pod="opendatahub/maas-controller-56b74cbc4d-w6tww"
Sep 11 20:08:51.428112 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.428064    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56b74cbc4d-w6tww"
Sep 11 20:08:51.702095 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.702069    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56b74cbc4d-w6tww"]
Sep 11 20:08:51.704135 ip-10-0-138-43 kubenswrapper[2558]: W0911 20:08:51.704110    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeac7486_f820_4057_8637_e1c647460219.slice/crio-289176b3dcb65e22544e9ab51b2774c4eaa3a36b129c278ee385a62a1573252b WatchSource:0}: Error finding container 289176b3dcb65e22544e9ab51b2774c4eaa3a36b129c278ee385a62a1573252b: Status 404 returned error can't find the container with id 289176b3dcb65e22544e9ab51b2774c4eaa3a36b129c278ee385a62a1573252b
Sep 11 20:08:51.705574 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:51.705549    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 20:08:52.647513 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:52.647422    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56b74cbc4d-w6tww" event={"ID":"beac7486-f820-4057-8637-e1c647460219","Type":"ContainerStarted","Data":"112adffca602be5cf14a95296b7629220912de4871cf1d6e23e0eee5ee96d7b3"}
Sep 11 20:08:52.647513 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:52.647476    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56b74cbc4d-w6tww" event={"ID":"beac7486-f820-4057-8637-e1c647460219","Type":"ContainerStarted","Data":"289176b3dcb65e22544e9ab51b2774c4eaa3a36b129c278ee385a62a1573252b"}
Sep 11 20:08:52.647513 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:52.647510    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56b74cbc4d-w6tww"
Sep 11 20:08:52.669652 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:08:52.669598    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-56b74cbc4d-w6tww" podStartSLOduration=1.156234927 podStartE2EDuration="1.669581621s" totalImagesPullingTime="513.346694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:08:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:08:52.667042072 +0000 UTC m=+2549.284211564" watchObservedRunningTime="2026-09-11 20:08:52.669581621 +0000 UTC m=+2549.286751134"
Sep 11 20:09:03.661821 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:03.661789    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56b74cbc4d-w6tww"
Sep 11 20:09:12.947002 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:12.946957    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56b74cbc4d-w6tww"]
Sep 11 20:09:12.947517 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:12.947237    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-56b74cbc4d-w6tww" podUID="beac7486-f820-4057-8637-e1c647460219" containerName="manager" containerID="cri-o://112adffca602be5cf14a95296b7629220912de4871cf1d6e23e0eee5ee96d7b3" gracePeriod=30
Sep 11 20:09:13.191853 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:13.191829    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56b74cbc4d-w6tww"
Sep 11 20:09:13.351766 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:13.351660    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxk78\" (UniqueName: \"kubernetes.io/projected/beac7486-f820-4057-8637-e1c647460219-kube-api-access-dxk78\") pod \"beac7486-f820-4057-8637-e1c647460219\" (UID: \"beac7486-f820-4057-8637-e1c647460219\") "
Sep 11 20:09:13.351766 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:13.351742    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beac7486-f820-4057-8637-e1c647460219-metrics-certs\") pod \"beac7486-f820-4057-8637-e1c647460219\" (UID: \"beac7486-f820-4057-8637-e1c647460219\") "
Sep 11 20:09:13.352069 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:13.351777    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/beac7486-f820-4057-8637-e1c647460219-webhook-cert\") pod \"beac7486-f820-4057-8637-e1c647460219\" (UID: \"beac7486-f820-4057-8637-e1c647460219\") "
Sep 11 20:09:13.353846 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:13.353818    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beac7486-f820-4057-8637-e1c647460219-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "beac7486-f820-4057-8637-e1c647460219" (UID: "beac7486-f820-4057-8637-e1c647460219"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:09:13.353974 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:13.353849    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beac7486-f820-4057-8637-e1c647460219-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "beac7486-f820-4057-8637-e1c647460219" (UID: "beac7486-f820-4057-8637-e1c647460219"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:09:13.353974 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:13.353852    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beac7486-f820-4057-8637-e1c647460219-kube-api-access-dxk78" (OuterVolumeSpecName: "kube-api-access-dxk78") pod "beac7486-f820-4057-8637-e1c647460219" (UID: "beac7486-f820-4057-8637-e1c647460219"). InnerVolumeSpecName "kube-api-access-dxk78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:09:13.452922 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:13.452890    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxk78\" (UniqueName: \"kubernetes.io/projected/beac7486-f820-4057-8637-e1c647460219-kube-api-access-dxk78\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:09:13.452922 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:13.452920    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beac7486-f820-4057-8637-e1c647460219-metrics-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:09:13.452922 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:13.452930    2558 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/beac7486-f820-4057-8637-e1c647460219-webhook-cert\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:09:13.758738 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:13.758690    2558 generic.go:356] "Generic (PLEG): container finished" podID="beac7486-f820-4057-8637-e1c647460219" containerID="112adffca602be5cf14a95296b7629220912de4871cf1d6e23e0eee5ee96d7b3" exitCode=0
Sep 11 20:09:13.758904 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:13.758753    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56b74cbc4d-w6tww"
Sep 11 20:09:13.758904 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:13.758761    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56b74cbc4d-w6tww" event={"ID":"beac7486-f820-4057-8637-e1c647460219","Type":"ContainerDied","Data":"112adffca602be5cf14a95296b7629220912de4871cf1d6e23e0eee5ee96d7b3"}
Sep 11 20:09:13.758904 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:13.758811    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56b74cbc4d-w6tww" event={"ID":"beac7486-f820-4057-8637-e1c647460219","Type":"ContainerDied","Data":"289176b3dcb65e22544e9ab51b2774c4eaa3a36b129c278ee385a62a1573252b"}
Sep 11 20:09:13.758904 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:13.758836    2558 scope.go:117] "RemoveContainer" containerID="112adffca602be5cf14a95296b7629220912de4871cf1d6e23e0eee5ee96d7b3"
Sep 11 20:09:13.767947 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:13.767929    2558 scope.go:117] "RemoveContainer" containerID="112adffca602be5cf14a95296b7629220912de4871cf1d6e23e0eee5ee96d7b3"
Sep 11 20:09:13.768193 ip-10-0-138-43 kubenswrapper[2558]: E0911 20:09:13.768174    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"112adffca602be5cf14a95296b7629220912de4871cf1d6e23e0eee5ee96d7b3\": container with ID starting with 112adffca602be5cf14a95296b7629220912de4871cf1d6e23e0eee5ee96d7b3 not found: ID does not exist" containerID="112adffca602be5cf14a95296b7629220912de4871cf1d6e23e0eee5ee96d7b3"
Sep 11 20:09:13.768239 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:13.768200    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"112adffca602be5cf14a95296b7629220912de4871cf1d6e23e0eee5ee96d7b3"} err="failed to get container status \"112adffca602be5cf14a95296b7629220912de4871cf1d6e23e0eee5ee96d7b3\": rpc error: code = NotFound desc = could not find container \"112adffca602be5cf14a95296b7629220912de4871cf1d6e23e0eee5ee96d7b3\": container with ID starting with 112adffca602be5cf14a95296b7629220912de4871cf1d6e23e0eee5ee96d7b3 not found: ID does not exist"
Sep 11 20:09:13.780558 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:13.780534    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56b74cbc4d-w6tww"]
Sep 11 20:09:13.786100 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:13.786075    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-56b74cbc4d-w6tww"]
Sep 11 20:09:13.813512 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:13.813488    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beac7486-f820-4057-8637-e1c647460219" path="/var/lib/kubelet/pods/beac7486-f820-4057-8637-e1c647460219/volumes"
Sep 11 20:09:14.544815 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:14.544779    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56b74cbc4d-q84gm"]
Sep 11 20:09:14.545183 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:14.545152    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beac7486-f820-4057-8637-e1c647460219" containerName="manager"
Sep 11 20:09:14.545183 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:14.545162    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="beac7486-f820-4057-8637-e1c647460219" containerName="manager"
Sep 11 20:09:14.545270 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:14.545218    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="beac7486-f820-4057-8637-e1c647460219" containerName="manager"
Sep 11 20:09:14.549463 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:14.549447    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56b74cbc4d-q84gm"
Sep 11 20:09:14.551808 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:14.551780    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-km7bf\""
Sep 11 20:09:14.552468 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:14.552453    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 20:09:14.552981 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:14.552956    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 20:09:14.553098 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:14.552969    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 20:09:14.561658 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:14.561637    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56b74cbc4d-q84gm"]
Sep 11 20:09:14.663318 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:14.663282    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9e9a871-2036-4541-8279-d6ebf7c554e1-webhook-cert\") pod \"maas-controller-56b74cbc4d-q84gm\" (UID: \"d9e9a871-2036-4541-8279-d6ebf7c554e1\") " pod="opendatahub/maas-controller-56b74cbc4d-q84gm"
Sep 11 20:09:14.663318 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:14.663321    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e9a871-2036-4541-8279-d6ebf7c554e1-metrics-certs\") pod \"maas-controller-56b74cbc4d-q84gm\" (UID: \"d9e9a871-2036-4541-8279-d6ebf7c554e1\") " pod="opendatahub/maas-controller-56b74cbc4d-q84gm"
Sep 11 20:09:14.663539 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:14.663345    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phbqs\" (UniqueName: \"kubernetes.io/projected/d9e9a871-2036-4541-8279-d6ebf7c554e1-kube-api-access-phbqs\") pod \"maas-controller-56b74cbc4d-q84gm\" (UID: \"d9e9a871-2036-4541-8279-d6ebf7c554e1\") " pod="opendatahub/maas-controller-56b74cbc4d-q84gm"
Sep 11 20:09:14.763865 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:14.763837    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9e9a871-2036-4541-8279-d6ebf7c554e1-webhook-cert\") pod \"maas-controller-56b74cbc4d-q84gm\" (UID: \"d9e9a871-2036-4541-8279-d6ebf7c554e1\") " pod="opendatahub/maas-controller-56b74cbc4d-q84gm"
Sep 11 20:09:14.764067 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:14.763869    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e9a871-2036-4541-8279-d6ebf7c554e1-metrics-certs\") pod \"maas-controller-56b74cbc4d-q84gm\" (UID: \"d9e9a871-2036-4541-8279-d6ebf7c554e1\") " pod="opendatahub/maas-controller-56b74cbc4d-q84gm"
Sep 11 20:09:14.764067 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:14.763896    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phbqs\" (UniqueName: \"kubernetes.io/projected/d9e9a871-2036-4541-8279-d6ebf7c554e1-kube-api-access-phbqs\") pod \"maas-controller-56b74cbc4d-q84gm\" (UID: \"d9e9a871-2036-4541-8279-d6ebf7c554e1\") " pod="opendatahub/maas-controller-56b74cbc4d-q84gm"
Sep 11 20:09:14.766434 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:14.766396    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9e9a871-2036-4541-8279-d6ebf7c554e1-metrics-certs\") pod \"maas-controller-56b74cbc4d-q84gm\" (UID: \"d9e9a871-2036-4541-8279-d6ebf7c554e1\") " pod="opendatahub/maas-controller-56b74cbc4d-q84gm"
Sep 11 20:09:14.766563 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:14.766453    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9e9a871-2036-4541-8279-d6ebf7c554e1-webhook-cert\") pod \"maas-controller-56b74cbc4d-q84gm\" (UID: \"d9e9a871-2036-4541-8279-d6ebf7c554e1\") " pod="opendatahub/maas-controller-56b74cbc4d-q84gm"
Sep 11 20:09:14.773300 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:14.773272    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phbqs\" (UniqueName: \"kubernetes.io/projected/d9e9a871-2036-4541-8279-d6ebf7c554e1-kube-api-access-phbqs\") pod \"maas-controller-56b74cbc4d-q84gm\" (UID: \"d9e9a871-2036-4541-8279-d6ebf7c554e1\") " pod="opendatahub/maas-controller-56b74cbc4d-q84gm"
Sep 11 20:09:14.860040 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:14.859936    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56b74cbc4d-q84gm"
Sep 11 20:09:15.134171 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:15.134144    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56b74cbc4d-q84gm"]
Sep 11 20:09:15.136391 ip-10-0-138-43 kubenswrapper[2558]: W0911 20:09:15.136364    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e9a871_2036_4541_8279_d6ebf7c554e1.slice/crio-7c7edd11287ea41eaf6a309247b01bc966234a0bd6185f295e01ad8d050d63bd WatchSource:0}: Error finding container 7c7edd11287ea41eaf6a309247b01bc966234a0bd6185f295e01ad8d050d63bd: Status 404 returned error can't find the container with id 7c7edd11287ea41eaf6a309247b01bc966234a0bd6185f295e01ad8d050d63bd
Sep 11 20:09:15.771301 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:15.771214    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56b74cbc4d-q84gm" event={"ID":"d9e9a871-2036-4541-8279-d6ebf7c554e1","Type":"ContainerStarted","Data":"67f0829573cd71fe188699b77f83f794939c6365afed8eb919dfddd24dcfb7ee"}
Sep 11 20:09:15.771756 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:15.771300    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56b74cbc4d-q84gm" event={"ID":"d9e9a871-2036-4541-8279-d6ebf7c554e1","Type":"ContainerStarted","Data":"7c7edd11287ea41eaf6a309247b01bc966234a0bd6185f295e01ad8d050d63bd"}
Sep 11 20:09:15.771756 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:15.771365    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56b74cbc4d-q84gm"
Sep 11 20:09:15.790579 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:15.790522    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-56b74cbc4d-q84gm" podStartSLOduration=1.434363954 podStartE2EDuration="1.790504462s" totalImagesPullingTime="356.140508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:09:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:09:15.789531691 +0000 UTC m=+2572.406701194" watchObservedRunningTime="2026-09-11 20:09:15.790504462 +0000 UTC m=+2572.407673954"
Sep 11 20:09:26.778587 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:09:26.778554    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56b74cbc4d-q84gm"
Sep 11 20:12:27.759025 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.758979    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"]
Sep 11 20:12:27.762515 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.762491    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:27.764845 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.764819    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-92f4f9a0-kserve-self-signed-certs\""
Sep 11 20:12:27.772153 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.771965    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"]
Sep 11 20:12:27.877560 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.877512    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-model-cache\") pod \"e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") " pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:27.877761 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.877634    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4xg8\" (UniqueName: \"kubernetes.io/projected/fa76dfa6-b322-4dac-b31b-da244d897ca4-kube-api-access-j4xg8\") pod \"e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") " pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:27.877761 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.877731    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-dshm\") pod \"e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") " pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:27.877872 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.877774    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-tmp-dir\") pod \"e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") " pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:27.877872 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.877802    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa76dfa6-b322-4dac-b31b-da244d897ca4-tls-certs\") pod \"e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") " pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:27.877872 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.877842    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-home\") pod \"e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") " pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:27.978813 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.978776    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-model-cache\") pod \"e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") " pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:27.979009 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.978831    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4xg8\" (UniqueName: \"kubernetes.io/projected/fa76dfa6-b322-4dac-b31b-da244d897ca4-kube-api-access-j4xg8\") pod \"e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") " pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:27.979009 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.978877    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-dshm\") pod \"e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") " pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:27.979009 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.978896    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-tmp-dir\") pod \"e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") " pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:27.979009 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.978917    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa76dfa6-b322-4dac-b31b-da244d897ca4-tls-certs\") pod \"e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") " pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:27.979009 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.978939    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-home\") pod \"e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") " pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:27.979290 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.979264    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-model-cache\") pod \"e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") " pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:27.979290 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.979279    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-tmp-dir\") pod \"e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") " pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:27.979395 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.979343    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-home\") pod \"e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") " pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:27.981338 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.981307    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-dshm\") pod \"e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") " pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:27.981450 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.981372    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa76dfa6-b322-4dac-b31b-da244d897ca4-tls-certs\") pod \"e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") " pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:27.996584 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:27.996554    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4xg8\" (UniqueName: \"kubernetes.io/projected/fa76dfa6-b322-4dac-b31b-da244d897ca4-kube-api-access-j4xg8\") pod \"e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") " pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:28.074571 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:28.074494    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:28.345737 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:28.345692    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"]
Sep 11 20:12:28.346902 ip-10-0-138-43 kubenswrapper[2558]: W0911 20:12:28.346877    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa76dfa6_b322_4dac_b31b_da244d897ca4.slice/crio-6d71ca36f80bcf33d38de8646e5e208c5a3c1b542169419a4ea1b6b30712bdfa WatchSource:0}: Error finding container 6d71ca36f80bcf33d38de8646e5e208c5a3c1b542169419a4ea1b6b30712bdfa: Status 404 returned error can't find the container with id 6d71ca36f80bcf33d38de8646e5e208c5a3c1b542169419a4ea1b6b30712bdfa
Sep 11 20:12:28.706601 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:28.706559    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244" event={"ID":"fa76dfa6-b322-4dac-b31b-da244d897ca4","Type":"ContainerStarted","Data":"20e2437dfe57303258154bce5b69032a3180b8323b7fb8911b78a6cc370581c1"}
Sep 11 20:12:28.706817 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:28.706612    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244" event={"ID":"fa76dfa6-b322-4dac-b31b-da244d897ca4","Type":"ContainerStarted","Data":"6d71ca36f80bcf33d38de8646e5e208c5a3c1b542169419a4ea1b6b30712bdfa"}
Sep 11 20:12:28.725257 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:28.725198    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244" podStartSLOduration=1.7251801580000001 podStartE2EDuration="1.725180158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:12:28.722432389 +0000 UTC m=+2765.339601900" watchObservedRunningTime="2026-09-11 20:12:28.725180158 +0000 UTC m=+2765.342349641"
Sep 11 20:12:38.075720 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:38.075663    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:38.076217 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:38.075763    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:38.088562 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:38.088527    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:38.769839 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:38.769807    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:12:46.536492 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:12:46.536397    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"]
Sep 11 20:13:01.556450 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.556376    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244" podUID="fa76dfa6-b322-4dac-b31b-da244d897ca4" containerName="main" containerID="cri-o://20e2437dfe57303258154bce5b69032a3180b8323b7fb8911b78a6cc370581c1" gracePeriod=45
Sep 11 20:13:01.804931 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.804899    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:13:01.875217 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.875117    2558 generic.go:356] "Generic (PLEG): container finished" podID="fa76dfa6-b322-4dac-b31b-da244d897ca4" containerID="20e2437dfe57303258154bce5b69032a3180b8323b7fb8911b78a6cc370581c1" exitCode=0
Sep 11 20:13:01.875217 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.875168    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"
Sep 11 20:13:01.875217 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.875176    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244" event={"ID":"fa76dfa6-b322-4dac-b31b-da244d897ca4","Type":"ContainerDied","Data":"20e2437dfe57303258154bce5b69032a3180b8323b7fb8911b78a6cc370581c1"}
Sep 11 20:13:01.875217 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.875212    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244" event={"ID":"fa76dfa6-b322-4dac-b31b-da244d897ca4","Type":"ContainerDied","Data":"6d71ca36f80bcf33d38de8646e5e208c5a3c1b542169419a4ea1b6b30712bdfa"}
Sep 11 20:13:01.875529 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.875229    2558 scope.go:117] "RemoveContainer" containerID="20e2437dfe57303258154bce5b69032a3180b8323b7fb8911b78a6cc370581c1"
Sep 11 20:13:01.883741 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.883721    2558 scope.go:117] "RemoveContainer" containerID="20e2437dfe57303258154bce5b69032a3180b8323b7fb8911b78a6cc370581c1"
Sep 11 20:13:01.884032 ip-10-0-138-43 kubenswrapper[2558]: E0911 20:13:01.883998    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20e2437dfe57303258154bce5b69032a3180b8323b7fb8911b78a6cc370581c1\": container with ID starting with 20e2437dfe57303258154bce5b69032a3180b8323b7fb8911b78a6cc370581c1 not found: ID does not exist" containerID="20e2437dfe57303258154bce5b69032a3180b8323b7fb8911b78a6cc370581c1"
Sep 11 20:13:01.884095 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.884039    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20e2437dfe57303258154bce5b69032a3180b8323b7fb8911b78a6cc370581c1"} err="failed to get container status \"20e2437dfe57303258154bce5b69032a3180b8323b7fb8911b78a6cc370581c1\": rpc error: code = NotFound desc = could not find container \"20e2437dfe57303258154bce5b69032a3180b8323b7fb8911b78a6cc370581c1\": container with ID starting with 20e2437dfe57303258154bce5b69032a3180b8323b7fb8911b78a6cc370581c1 not found: ID does not exist"
Sep 11 20:13:01.889306 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.889284    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-home\") pod \"fa76dfa6-b322-4dac-b31b-da244d897ca4\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") "
Sep 11 20:13:01.889414 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.889333    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-dshm\") pod \"fa76dfa6-b322-4dac-b31b-da244d897ca4\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") "
Sep 11 20:13:01.889414 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.889373    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4xg8\" (UniqueName: \"kubernetes.io/projected/fa76dfa6-b322-4dac-b31b-da244d897ca4-kube-api-access-j4xg8\") pod \"fa76dfa6-b322-4dac-b31b-da244d897ca4\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") "
Sep 11 20:13:01.889517 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.889429    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa76dfa6-b322-4dac-b31b-da244d897ca4-tls-certs\") pod \"fa76dfa6-b322-4dac-b31b-da244d897ca4\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") "
Sep 11 20:13:01.889572 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.889514    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-model-cache\") pod \"fa76dfa6-b322-4dac-b31b-da244d897ca4\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") "
Sep 11 20:13:01.889572 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.889553    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-home" (OuterVolumeSpecName: "home") pod "fa76dfa6-b322-4dac-b31b-da244d897ca4" (UID: "fa76dfa6-b322-4dac-b31b-da244d897ca4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:13:01.889680 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.889564    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-tmp-dir\") pod \"fa76dfa6-b322-4dac-b31b-da244d897ca4\" (UID: \"fa76dfa6-b322-4dac-b31b-da244d897ca4\") "
Sep 11 20:13:01.889829 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.889803    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa76dfa6-b322-4dac-b31b-da244d897ca4" (UID: "fa76dfa6-b322-4dac-b31b-da244d897ca4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:13:01.889915 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.889820    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa76dfa6-b322-4dac-b31b-da244d897ca4" (UID: "fa76dfa6-b322-4dac-b31b-da244d897ca4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:13:01.890360 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.890123    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-tmp-dir\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:13:01.890360 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.890150    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-home\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:13:01.890360 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.890164    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-model-cache\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:13:01.891654 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.891617    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa76dfa6-b322-4dac-b31b-da244d897ca4-kube-api-access-j4xg8" (OuterVolumeSpecName: "kube-api-access-j4xg8") pod "fa76dfa6-b322-4dac-b31b-da244d897ca4" (UID: "fa76dfa6-b322-4dac-b31b-da244d897ca4"). InnerVolumeSpecName "kube-api-access-j4xg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:13:01.891942 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.891923    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-dshm" (OuterVolumeSpecName: "dshm") pod "fa76dfa6-b322-4dac-b31b-da244d897ca4" (UID: "fa76dfa6-b322-4dac-b31b-da244d897ca4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 20:13:01.892022 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.891937    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa76dfa6-b322-4dac-b31b-da244d897ca4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa76dfa6-b322-4dac-b31b-da244d897ca4" (UID: "fa76dfa6-b322-4dac-b31b-da244d897ca4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:13:01.991016 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.990980    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa76dfa6-b322-4dac-b31b-da244d897ca4-dshm\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:13:01.991016 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.991009    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4xg8\" (UniqueName: \"kubernetes.io/projected/fa76dfa6-b322-4dac-b31b-da244d897ca4-kube-api-access-j4xg8\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:13:01.991016 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:01.991021    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa76dfa6-b322-4dac-b31b-da244d897ca4-tls-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:13:02.202023 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:02.201992    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"]
Sep 11 20:13:02.211556 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:02.211520    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-92f4f9a0-kserve-9858d9bd7-jw244"]
Sep 11 20:13:03.813757 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:03.813699    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa76dfa6-b322-4dac-b31b-da244d897ca4" path="/var/lib/kubelet/pods/fa76dfa6-b322-4dac-b31b-da244d897ca4/volumes"
Sep 11 20:13:09.736701 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:09.736661    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t"]
Sep 11 20:13:09.737309 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:09.737277    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa76dfa6-b322-4dac-b31b-da244d897ca4" containerName="main"
Sep 11 20:13:09.737309 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:09.737298    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa76dfa6-b322-4dac-b31b-da244d897ca4" containerName="main"
Sep 11 20:13:09.737446 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:09.737416    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa76dfa6-b322-4dac-b31b-da244d897ca4" containerName="main"
Sep 11 20:13:09.740910 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:09.740889    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t"
Sep 11 20:13:09.752178 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:09.752154    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t"]
Sep 11 20:13:09.871652 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:09.871619    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89109992-a947-4371-9e0e-ca8728bcb81a-openshift-service-ca\") pod \"maas-api-786f4dddd4-24b2t\" (UID: \"89109992-a947-4371-9e0e-ca8728bcb81a\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t"
Sep 11 20:13:09.871847 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:09.871734    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89109992-a947-4371-9e0e-ca8728bcb81a-metrics-certs\") pod \"maas-api-786f4dddd4-24b2t\" (UID: \"89109992-a947-4371-9e0e-ca8728bcb81a\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t"
Sep 11 20:13:09.871847 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:09.871800    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89109992-a947-4371-9e0e-ca8728bcb81a-maas-api-tls\") pod \"maas-api-786f4dddd4-24b2t\" (UID: \"89109992-a947-4371-9e0e-ca8728bcb81a\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t"
Sep 11 20:13:09.871929 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:09.871888    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xtsz\" (UniqueName: \"kubernetes.io/projected/89109992-a947-4371-9e0e-ca8728bcb81a-kube-api-access-6xtsz\") pod \"maas-api-786f4dddd4-24b2t\" (UID: \"89109992-a947-4371-9e0e-ca8728bcb81a\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t"
Sep 11 20:13:09.973371 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:09.973329    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89109992-a947-4371-9e0e-ca8728bcb81a-openshift-service-ca\") pod \"maas-api-786f4dddd4-24b2t\" (UID: \"89109992-a947-4371-9e0e-ca8728bcb81a\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t"
Sep 11 20:13:09.973548 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:09.973384    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89109992-a947-4371-9e0e-ca8728bcb81a-metrics-certs\") pod \"maas-api-786f4dddd4-24b2t\" (UID: \"89109992-a947-4371-9e0e-ca8728bcb81a\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t"
Sep 11 20:13:09.973548 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:09.973408    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89109992-a947-4371-9e0e-ca8728bcb81a-maas-api-tls\") pod \"maas-api-786f4dddd4-24b2t\" (UID: \"89109992-a947-4371-9e0e-ca8728bcb81a\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t"
Sep 11 20:13:09.973548 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:09.973448    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xtsz\" (UniqueName: \"kubernetes.io/projected/89109992-a947-4371-9e0e-ca8728bcb81a-kube-api-access-6xtsz\") pod \"maas-api-786f4dddd4-24b2t\" (UID: \"89109992-a947-4371-9e0e-ca8728bcb81a\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t"
Sep 11 20:13:09.974166 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:09.974135    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89109992-a947-4371-9e0e-ca8728bcb81a-openshift-service-ca\") pod \"maas-api-786f4dddd4-24b2t\" (UID: \"89109992-a947-4371-9e0e-ca8728bcb81a\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t"
Sep 11 20:13:09.975993 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:09.975962    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89109992-a947-4371-9e0e-ca8728bcb81a-maas-api-tls\") pod \"maas-api-786f4dddd4-24b2t\" (UID: \"89109992-a947-4371-9e0e-ca8728bcb81a\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t"
Sep 11 20:13:09.976098 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:09.976028    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89109992-a947-4371-9e0e-ca8728bcb81a-metrics-certs\") pod \"maas-api-786f4dddd4-24b2t\" (UID: \"89109992-a947-4371-9e0e-ca8728bcb81a\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t"
Sep 11 20:13:09.982551 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:09.982531    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xtsz\" (UniqueName: \"kubernetes.io/projected/89109992-a947-4371-9e0e-ca8728bcb81a-kube-api-access-6xtsz\") pod \"maas-api-786f4dddd4-24b2t\" (UID: \"89109992-a947-4371-9e0e-ca8728bcb81a\") " pod="odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t"
Sep 11 20:13:10.053613 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:10.053529    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t"
Sep 11 20:13:10.326489 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:10.326460    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t"]
Sep 11 20:13:10.328112 ip-10-0-138-43 kubenswrapper[2558]: W0911 20:13:10.328088    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89109992_a947_4371_9e0e_ca8728bcb81a.slice/crio-a4e601112aa2980cd8d2ec1363635c0201c150bd1990f4e37a2491338033e380 WatchSource:0}: Error finding container a4e601112aa2980cd8d2ec1363635c0201c150bd1990f4e37a2491338033e380: Status 404 returned error can't find the container with id a4e601112aa2980cd8d2ec1363635c0201c150bd1990f4e37a2491338033e380
Sep 11 20:13:10.920861 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:10.920822    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t" event={"ID":"89109992-a947-4371-9e0e-ca8728bcb81a","Type":"ContainerStarted","Data":"1a5dfc7dfb7b647955378c36ea1ff29ba938015bb3a3e8a8a9f0c736882c9206"}
Sep 11 20:13:10.921264 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:10.920878    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t" event={"ID":"89109992-a947-4371-9e0e-ca8728bcb81a","Type":"ContainerStarted","Data":"a4e601112aa2980cd8d2ec1363635c0201c150bd1990f4e37a2491338033e380"}
Sep 11 20:13:10.921264 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:10.920936    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t"
Sep 11 20:13:10.938239 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:10.938180    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t" podStartSLOduration=1.472023847 podStartE2EDuration="1.938163911s" totalImagesPullingTime="466.140064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:10.935550139 +0000 UTC m=+2807.552719633" watchObservedRunningTime="2026-09-11 20:13:10.938163911 +0000 UTC m=+2807.555333402"
Sep 11 20:13:16.931849 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:16.931814    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-786f4dddd4-24b2t"
Sep 11 20:13:24.696721 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:24.696666    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"]
Sep 11 20:13:24.697187 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:24.696953    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx" podUID="8224af77-d328-4adb-a8aa-b53aa915e55b" containerName="maas-api" containerID="cri-o://153b8e1823bc0ebaaff70fc5ac8821003f1dc408ad7ae647a69b603bee4eda0d" gracePeriod=30
Sep 11 20:13:25.952409 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:25.952383    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"
Sep 11 20:13:25.997504 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:25.997471    2558 generic.go:356] "Generic (PLEG): container finished" podID="8224af77-d328-4adb-a8aa-b53aa915e55b" containerID="153b8e1823bc0ebaaff70fc5ac8821003f1dc408ad7ae647a69b603bee4eda0d" exitCode=0
Sep 11 20:13:25.997698 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:25.997529    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx" event={"ID":"8224af77-d328-4adb-a8aa-b53aa915e55b","Type":"ContainerDied","Data":"153b8e1823bc0ebaaff70fc5ac8821003f1dc408ad7ae647a69b603bee4eda0d"}
Sep 11 20:13:25.997698 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:25.997532    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"
Sep 11 20:13:25.997698 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:25.997573    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx" event={"ID":"8224af77-d328-4adb-a8aa-b53aa915e55b","Type":"ContainerDied","Data":"601f0b74258f75b9c4a83d3276664f6eb024c241152a79ec821425b495ccfa18"}
Sep 11 20:13:25.997698 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:25.997597    2558 scope.go:117] "RemoveContainer" containerID="153b8e1823bc0ebaaff70fc5ac8821003f1dc408ad7ae647a69b603bee4eda0d"
Sep 11 20:13:26.006993 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:26.006973    2558 scope.go:117] "RemoveContainer" containerID="153b8e1823bc0ebaaff70fc5ac8821003f1dc408ad7ae647a69b603bee4eda0d"
Sep 11 20:13:26.007308 ip-10-0-138-43 kubenswrapper[2558]: E0911 20:13:26.007284    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153b8e1823bc0ebaaff70fc5ac8821003f1dc408ad7ae647a69b603bee4eda0d\": container with ID starting with 153b8e1823bc0ebaaff70fc5ac8821003f1dc408ad7ae647a69b603bee4eda0d not found: ID does not exist" containerID="153b8e1823bc0ebaaff70fc5ac8821003f1dc408ad7ae647a69b603bee4eda0d"
Sep 11 20:13:26.007388 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:26.007318    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153b8e1823bc0ebaaff70fc5ac8821003f1dc408ad7ae647a69b603bee4eda0d"} err="failed to get container status \"153b8e1823bc0ebaaff70fc5ac8821003f1dc408ad7ae647a69b603bee4eda0d\": rpc error: code = NotFound desc = could not find container \"153b8e1823bc0ebaaff70fc5ac8821003f1dc408ad7ae647a69b603bee4eda0d\": container with ID starting with 153b8e1823bc0ebaaff70fc5ac8821003f1dc408ad7ae647a69b603bee4eda0d not found: ID does not exist"
Sep 11 20:13:26.130589 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:26.130494    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5qsg\" (UniqueName: \"kubernetes.io/projected/8224af77-d328-4adb-a8aa-b53aa915e55b-kube-api-access-d5qsg\") pod \"8224af77-d328-4adb-a8aa-b53aa915e55b\" (UID: \"8224af77-d328-4adb-a8aa-b53aa915e55b\") "
Sep 11 20:13:26.130589 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:26.130567    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8224af77-d328-4adb-a8aa-b53aa915e55b-metrics-certs\") pod \"8224af77-d328-4adb-a8aa-b53aa915e55b\" (UID: \"8224af77-d328-4adb-a8aa-b53aa915e55b\") "
Sep 11 20:13:26.130840 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:26.130606    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8224af77-d328-4adb-a8aa-b53aa915e55b-openshift-service-ca\") pod \"8224af77-d328-4adb-a8aa-b53aa915e55b\" (UID: \"8224af77-d328-4adb-a8aa-b53aa915e55b\") "
Sep 11 20:13:26.130840 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:26.130678    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8224af77-d328-4adb-a8aa-b53aa915e55b-maas-api-tls\") pod \"8224af77-d328-4adb-a8aa-b53aa915e55b\" (UID: \"8224af77-d328-4adb-a8aa-b53aa915e55b\") "
Sep 11 20:13:26.131055 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:26.131023    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8224af77-d328-4adb-a8aa-b53aa915e55b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8224af77-d328-4adb-a8aa-b53aa915e55b" (UID: "8224af77-d328-4adb-a8aa-b53aa915e55b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 20:13:26.132795 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:26.132764    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8224af77-d328-4adb-a8aa-b53aa915e55b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8224af77-d328-4adb-a8aa-b53aa915e55b" (UID: "8224af77-d328-4adb-a8aa-b53aa915e55b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:13:26.132892 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:26.132787    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8224af77-d328-4adb-a8aa-b53aa915e55b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8224af77-d328-4adb-a8aa-b53aa915e55b" (UID: "8224af77-d328-4adb-a8aa-b53aa915e55b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 20:13:26.132892 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:26.132789    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8224af77-d328-4adb-a8aa-b53aa915e55b-kube-api-access-d5qsg" (OuterVolumeSpecName: "kube-api-access-d5qsg") pod "8224af77-d328-4adb-a8aa-b53aa915e55b" (UID: "8224af77-d328-4adb-a8aa-b53aa915e55b"). InnerVolumeSpecName "kube-api-access-d5qsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 20:13:26.232577 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:26.232541    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8224af77-d328-4adb-a8aa-b53aa915e55b-maas-api-tls\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:13:26.232577 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:26.232572    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5qsg\" (UniqueName: \"kubernetes.io/projected/8224af77-d328-4adb-a8aa-b53aa915e55b-kube-api-access-d5qsg\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:13:26.232577 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:26.232582    2558 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8224af77-d328-4adb-a8aa-b53aa915e55b-metrics-certs\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:13:26.232848 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:26.232590    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8224af77-d328-4adb-a8aa-b53aa915e55b-openshift-service-ca\") on node \"ip-10-0-138-43.ec2.internal\" DevicePath \"\""
Sep 11 20:13:26.319660 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:26.319630    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"]
Sep 11 20:13:26.323069 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:26.323042    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-ccb8f7bdd-r55jx"]
Sep 11 20:13:27.814486 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:27.814451    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8224af77-d328-4adb-a8aa-b53aa915e55b" path="/var/lib/kubelet/pods/8224af77-d328-4adb-a8aa-b53aa915e55b/volumes"
Sep 11 20:13:44.627719 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:44.627682    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-bvhv7_d537e8db-07f2-4b1d-88cf-be3abd543e71/manager/0.log"
Sep 11 20:13:44.873022 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:44.872968    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56b74cbc4d-q84gm_d9e9a871-2036-4541-8279-d6ebf7c554e1/manager/0.log"
Sep 11 20:13:45.329644 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:45.329617    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c47b89ccd-ndv2n_4d22b8c9-3e1c-445b-9c2c-e873033b0761/copy-manifests/0.log"
Sep 11 20:13:45.337592 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:45.337567    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c47b89ccd-ndv2n_4d22b8c9-3e1c-445b-9c2c-e873033b0761/manager/0.log"
Sep 11 20:13:45.821100 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:45.821065    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-786f4dddd4-24b2t_89109992-a947-4371-9e0e-ca8728bcb81a/maas-api/0.log"
Sep 11 20:13:46.159603 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:46.159515    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819280-m658h_a112ec37-694a-4a5b-990d-10885ef25e6f/cleanup/0.log"
Sep 11 20:13:46.268023 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:46.267991    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-5cf57_fbf3e31c-3f2a-474d-995c-83efc6611826/postgres/0.log"
Sep 11 20:13:47.249439 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:47.249407    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-bd2rp_d6c77b1e-ad5d-439f-801d-a40432e32cb6/manager/0.log"
Sep 11 20:13:48.493905 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:48.493872    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-pqmwd_c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba/istio-proxy/1.log"
Sep 11 20:13:48.615333 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:48.615298    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wkf6k_33f7cec5-65d3-4c19-a677-50f6e7969695/payload-pre-processing/0.log"
Sep 11 20:13:48.734086 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:48.734051    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xptvp_cb31f235-193f-4244-9c95-06e347d981ee/payload-processing/0.log"
Sep 11 20:13:49.187720 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:49.187675    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-j9mjz_cbbffc01-0831-4480-834c-8fb2cc60a333/main/0.log"
Sep 11 20:13:49.296176 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:49.296146    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-z4zxz_f7bf84f6-aff6-414e-aedb-123f5e4022dd/main/0.log"
Sep 11 20:13:49.625275 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:49.625191    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fgsrvw_6eb64051-b17b-41df-a9de-2ae4708c59db/main/0.log"
Sep 11 20:13:49.735748 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:49.735695    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-lqzgf_ee29cea2-3b01-42f8-b6e1-afa4e9a872c9/main/0.log"
Sep 11 20:13:57.784746 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:57.784693    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pfnzl_58780500-db7c-40c3-b4ab-b484237c4ef2/global-pull-secret-syncer/0.log"
Sep 11 20:13:57.900815 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:57.900785    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l6ktp_d4ca207f-ccf3-4f01-bd0d-cd4cc9148997/konnectivity-agent/0.log"
Sep 11 20:13:58.001915 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:13:58.001876    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-43.ec2.internal_af89a8d622e91b5bb6c931160e64948c/haproxy/0.log"
Sep 11 20:14:02.749455 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:02.749416    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-bd2rp_d6c77b1e-ad5d-439f-801d-a40432e32cb6/manager/0.log"
Sep 11 20:14:04.904914 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:04.904883    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b4fxt_5bd3b61e-bae5-4fbb-a921-ffd2f6c74675/node-exporter/0.log"
Sep 11 20:14:04.926586 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:04.926550    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b4fxt_5bd3b61e-bae5-4fbb-a921-ffd2f6c74675/kube-rbac-proxy/0.log"
Sep 11 20:14:04.947920 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:04.947893    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b4fxt_5bd3b61e-bae5-4fbb-a921-ffd2f6c74675/init-textfile/0.log"
Sep 11 20:14:05.106358 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:05.106259    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-mkv72_d094cc80-b4b7-4f94-bd57-cb159354ea85/kube-rbac-proxy-main/0.log"
Sep 11 20:14:05.128990 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:05.128964    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-mkv72_d094cc80-b4b7-4f94-bd57-cb159354ea85/kube-rbac-proxy-self/0.log"
Sep 11 20:14:05.149466 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:05.149438    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-mkv72_d094cc80-b4b7-4f94-bd57-cb159354ea85/openshift-state-metrics/0.log"
Sep 11 20:14:05.192877 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:05.188198    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c8b0a80-0b86-4c1c-a5c9-642029c131ee/prometheus/0.log"
Sep 11 20:14:05.210985 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:05.210954    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c8b0a80-0b86-4c1c-a5c9-642029c131ee/config-reloader/0.log"
Sep 11 20:14:05.229617 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:05.229590    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c8b0a80-0b86-4c1c-a5c9-642029c131ee/thanos-sidecar/0.log"
Sep 11 20:14:05.253358 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:05.253332    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c8b0a80-0b86-4c1c-a5c9-642029c131ee/kube-rbac-proxy-web/0.log"
Sep 11 20:14:05.276309 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:05.276287    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c8b0a80-0b86-4c1c-a5c9-642029c131ee/kube-rbac-proxy/0.log"
Sep 11 20:14:05.296370 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:05.296345    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c8b0a80-0b86-4c1c-a5c9-642029c131ee/kube-rbac-proxy-thanos/0.log"
Sep 11 20:14:05.317230 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:05.317206    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5c8b0a80-0b86-4c1c-a5c9-642029c131ee/init-config-reloader/0.log"
Sep 11 20:14:05.346054 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:05.346028    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-gn2ff_7cd4631c-9b63-41a2-9fd0-16e7a88bf834/prometheus-operator/0.log"
Sep 11 20:14:05.361672 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:05.361592    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-gn2ff_7cd4631c-9b63-41a2-9fd0-16e7a88bf834/kube-rbac-proxy/0.log"
Sep 11 20:14:05.393019 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:05.392994    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-s9wws_870f9d8f-89f3-4184-b005-ddd6c7e06f6e/prometheus-operator-admission-webhook/0.log"
Sep 11 20:14:08.120961 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.120924    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"]
Sep 11 20:14:08.121330 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.121292    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8224af77-d328-4adb-a8aa-b53aa915e55b" containerName="maas-api"
Sep 11 20:14:08.121330 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.121304    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8224af77-d328-4adb-a8aa-b53aa915e55b" containerName="maas-api"
Sep 11 20:14:08.121419 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.121373    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8224af77-d328-4adb-a8aa-b53aa915e55b" containerName="maas-api"
Sep 11 20:14:08.124302 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.124285    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"
Sep 11 20:14:08.126251 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.126226    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kx729\"/\"openshift-service-ca.crt\""
Sep 11 20:14:08.126379 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.126299    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kx729\"/\"default-dockercfg-hfjpd\""
Sep 11 20:14:08.126994 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.126980    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kx729\"/\"kube-root-ca.crt\""
Sep 11 20:14:08.131516 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.131494    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"]
Sep 11 20:14:08.214832 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.214800    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f8713f7e-9dd1-424e-baeb-ac7d1ad33503-proc\") pod \"perf-node-gather-daemonset-nvmdw\" (UID: \"f8713f7e-9dd1-424e-baeb-ac7d1ad33503\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"
Sep 11 20:14:08.215011 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.214882    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8713f7e-9dd1-424e-baeb-ac7d1ad33503-sys\") pod \"perf-node-gather-daemonset-nvmdw\" (UID: \"f8713f7e-9dd1-424e-baeb-ac7d1ad33503\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"
Sep 11 20:14:08.215011 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.214904    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kmm4\" (UniqueName: \"kubernetes.io/projected/f8713f7e-9dd1-424e-baeb-ac7d1ad33503-kube-api-access-7kmm4\") pod \"perf-node-gather-daemonset-nvmdw\" (UID: \"f8713f7e-9dd1-424e-baeb-ac7d1ad33503\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"
Sep 11 20:14:08.215011 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.214942    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f8713f7e-9dd1-424e-baeb-ac7d1ad33503-lib-modules\") pod \"perf-node-gather-daemonset-nvmdw\" (UID: \"f8713f7e-9dd1-424e-baeb-ac7d1ad33503\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"
Sep 11 20:14:08.215011 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.214997    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f8713f7e-9dd1-424e-baeb-ac7d1ad33503-podres\") pod \"perf-node-gather-daemonset-nvmdw\" (UID: \"f8713f7e-9dd1-424e-baeb-ac7d1ad33503\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"
Sep 11 20:14:08.316125 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.316085    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8713f7e-9dd1-424e-baeb-ac7d1ad33503-sys\") pod \"perf-node-gather-daemonset-nvmdw\" (UID: \"f8713f7e-9dd1-424e-baeb-ac7d1ad33503\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"
Sep 11 20:14:08.316125 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.316126    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kmm4\" (UniqueName: \"kubernetes.io/projected/f8713f7e-9dd1-424e-baeb-ac7d1ad33503-kube-api-access-7kmm4\") pod \"perf-node-gather-daemonset-nvmdw\" (UID: \"f8713f7e-9dd1-424e-baeb-ac7d1ad33503\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"
Sep 11 20:14:08.316318 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.316171    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f8713f7e-9dd1-424e-baeb-ac7d1ad33503-lib-modules\") pod \"perf-node-gather-daemonset-nvmdw\" (UID: \"f8713f7e-9dd1-424e-baeb-ac7d1ad33503\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"
Sep 11 20:14:08.316318 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.316191    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f8713f7e-9dd1-424e-baeb-ac7d1ad33503-podres\") pod \"perf-node-gather-daemonset-nvmdw\" (UID: \"f8713f7e-9dd1-424e-baeb-ac7d1ad33503\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"
Sep 11 20:14:08.316318 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.316213    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8713f7e-9dd1-424e-baeb-ac7d1ad33503-sys\") pod \"perf-node-gather-daemonset-nvmdw\" (UID: \"f8713f7e-9dd1-424e-baeb-ac7d1ad33503\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"
Sep 11 20:14:08.316318 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.316238    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f8713f7e-9dd1-424e-baeb-ac7d1ad33503-proc\") pod \"perf-node-gather-daemonset-nvmdw\" (UID: \"f8713f7e-9dd1-424e-baeb-ac7d1ad33503\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"
Sep 11 20:14:08.316453 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.316348    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f8713f7e-9dd1-424e-baeb-ac7d1ad33503-proc\") pod \"perf-node-gather-daemonset-nvmdw\" (UID: \"f8713f7e-9dd1-424e-baeb-ac7d1ad33503\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"
Sep 11 20:14:08.316453 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.316353    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f8713f7e-9dd1-424e-baeb-ac7d1ad33503-lib-modules\") pod \"perf-node-gather-daemonset-nvmdw\" (UID: \"f8713f7e-9dd1-424e-baeb-ac7d1ad33503\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"
Sep 11 20:14:08.316453 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.316395    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f8713f7e-9dd1-424e-baeb-ac7d1ad33503-podres\") pod \"perf-node-gather-daemonset-nvmdw\" (UID: \"f8713f7e-9dd1-424e-baeb-ac7d1ad33503\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"
Sep 11 20:14:08.324543 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.324523    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kmm4\" (UniqueName: \"kubernetes.io/projected/f8713f7e-9dd1-424e-baeb-ac7d1ad33503-kube-api-access-7kmm4\") pod \"perf-node-gather-daemonset-nvmdw\" (UID: \"f8713f7e-9dd1-424e-baeb-ac7d1ad33503\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"
Sep 11 20:14:08.435545 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.435515    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"
Sep 11 20:14:08.707661 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.707628    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"]
Sep 11 20:14:08.709730 ip-10-0-138-43 kubenswrapper[2558]: W0911 20:14:08.709692    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf8713f7e_9dd1_424e_baeb_ac7d1ad33503.slice/crio-25f35fe6a7fb35147fab9c06e003ea3e6b857ea45d59c21b86a57b5e6a2b9d40 WatchSource:0}: Error finding container 25f35fe6a7fb35147fab9c06e003ea3e6b857ea45d59c21b86a57b5e6a2b9d40: Status 404 returned error can't find the container with id 25f35fe6a7fb35147fab9c06e003ea3e6b857ea45d59c21b86a57b5e6a2b9d40
Sep 11 20:14:08.967359 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:08.967272    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-55czf_88643ea7-21db-4b13-a376-ac84810bcf61/volume-data-source-validator/0.log"
Sep 11 20:14:09.209513 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:09.209478    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw" event={"ID":"f8713f7e-9dd1-424e-baeb-ac7d1ad33503","Type":"ContainerStarted","Data":"80a6a5045f7a11b0016cd4aa56d230509a5506a77f6f1b6d5a56afc16a938029"}
Sep 11 20:14:09.209954 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:09.209532    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw" event={"ID":"f8713f7e-9dd1-424e-baeb-ac7d1ad33503","Type":"ContainerStarted","Data":"25f35fe6a7fb35147fab9c06e003ea3e6b857ea45d59c21b86a57b5e6a2b9d40"}
Sep 11 20:14:09.209954 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:09.209560    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"
Sep 11 20:14:09.225939 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:09.225844    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw" podStartSLOduration=1.225826152 podStartE2EDuration="1.225826152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:14:09.225029652 +0000 UTC m=+2865.842199142" watchObservedRunningTime="2026-09-11 20:14:09.225826152 +0000 UTC m=+2865.842995643"
Sep 11 20:14:10.037974 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:10.037946    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qq9hm_22079f6a-8b4b-46a7-a9c9-5656255a8144/dns/0.log"
Sep 11 20:14:10.057993 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:10.057944    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qq9hm_22079f6a-8b4b-46a7-a9c9-5656255a8144/kube-rbac-proxy/0.log"
Sep 11 20:14:10.204008 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:10.203974    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z8xdl_f7027ce0-9c8b-41f9-98c8-47905e740a83/dns-node-resolver/0.log"
Sep 11 20:14:10.732040 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:10.732005    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-8dc4fc6b6-2l26s_ac79548d-9052-4c47-b8b0-5a9917c7e8e0/registry/0.log"
Sep 11 20:14:10.759726 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:10.759678    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d5jh8_0b137eb7-4e92-44fe-882d-0ca19704ee2c/node-ca/0.log"
Sep 11 20:14:12.208092 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:12.208047    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-pqmwd_c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba/istio-proxy/0.log"
Sep 11 20:14:12.212482 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:12.212456    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-pqmwd_c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba/istio-proxy/1.log"
Sep 11 20:14:12.255171 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:12.255109    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wkf6k_33f7cec5-65d3-4c19-a677-50f6e7969695/payload-pre-processing/0.log"
Sep 11 20:14:12.296357 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:12.296321    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xptvp_cb31f235-193f-4244-9c95-06e347d981ee/payload-processing/0.log"
Sep 11 20:14:13.236788 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:13.236756    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nflsh_b32f7b7b-b959-4213-b66f-e1840988b22b/serve-healthcheck-canary/0.log"
Sep 11 20:14:13.889460 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:13.889422    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xmrpx_55d96fb4-b74a-4fa1-9e45-36ac15f57e67/kube-rbac-proxy/0.log"
Sep 11 20:14:13.910314 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:13.910284    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xmrpx_55d96fb4-b74a-4fa1-9e45-36ac15f57e67/exporter/0.log"
Sep 11 20:14:13.932761 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:13.932732    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xmrpx_55d96fb4-b74a-4fa1-9e45-36ac15f57e67/extractor/0.log"
Sep 11 20:14:15.223907 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:15.223877    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kx729/perf-node-gather-daemonset-nvmdw"
Sep 11 20:14:16.227464 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:16.227431    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-bvhv7_d537e8db-07f2-4b1d-88cf-be3abd543e71/manager/0.log"
Sep 11 20:14:16.667511 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:16.667475    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56b74cbc4d-q84gm_d9e9a871-2036-4541-8279-d6ebf7c554e1/manager/0.log"
Sep 11 20:14:16.899790 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:16.899740    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c47b89ccd-ndv2n_4d22b8c9-3e1c-445b-9c2c-e873033b0761/manager/0.log"
Sep 11 20:14:16.917828 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:16.917752    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c47b89ccd-ndv2n_4d22b8c9-3e1c-445b-9c2c-e873033b0761/copy-manifests/0.log"
Sep 11 20:14:18.467204 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:18.467170    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6d6bf85947-78cbt_69a666ce-ef0f-4db5-8753-7e3907d59888/manager/0.log"
Sep 11 20:14:18.512261 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:18.512178    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-xmsv9_7ab88c1d-216b-4c86-8abf-80468a16c5b5/openshift-lws-operator/0.log"
Sep 11 20:14:19.206007 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:19.205975    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-pqmwd_c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba/istio-proxy/0.log"
Sep 11 20:14:19.210932 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:19.210906    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-pqmwd_c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba/istio-proxy/1.log"
Sep 11 20:14:19.240465 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:19.240432    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wkf6k_33f7cec5-65d3-4c19-a677-50f6e7969695/payload-pre-processing/0.log"
Sep 11 20:14:19.271792 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:19.271754    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xptvp_cb31f235-193f-4244-9c95-06e347d981ee/payload-processing/0.log"
Sep 11 20:14:21.298970 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:21.298929    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-pqmwd_c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba/istio-proxy/0.log"
Sep 11 20:14:21.304133 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:21.304106    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-pqmwd_c0eb5c20-3a6f-4c5c-b48b-14dab8bd23ba/istio-proxy/1.log"
Sep 11 20:14:21.336550 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:21.336511    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-wkf6k_33f7cec5-65d3-4c19-a677-50f6e7969695/payload-pre-processing/0.log"
Sep 11 20:14:21.370001 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:21.369965    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xptvp_cb31f235-193f-4244-9c95-06e347d981ee/payload-processing/0.log"
Sep 11 20:14:25.198993 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:25.198963    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whp4x_cb1870fa-237b-4a9e-91b1-9741250e32d0/kube-multus-additional-cni-plugins/0.log"
Sep 11 20:14:25.218759 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:25.218728    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whp4x_cb1870fa-237b-4a9e-91b1-9741250e32d0/egress-router-binary-copy/0.log"
Sep 11 20:14:25.239069 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:25.239041    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whp4x_cb1870fa-237b-4a9e-91b1-9741250e32d0/cni-plugins/0.log"
Sep 11 20:14:25.259848 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:25.259813    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whp4x_cb1870fa-237b-4a9e-91b1-9741250e32d0/bond-cni-plugin/0.log"
Sep 11 20:14:25.280339 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:25.280306    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whp4x_cb1870fa-237b-4a9e-91b1-9741250e32d0/routeoverride-cni/0.log"
Sep 11 20:14:25.300214 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:25.300182    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whp4x_cb1870fa-237b-4a9e-91b1-9741250e32d0/whereabouts-cni-bincopy/0.log"
Sep 11 20:14:25.321326 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:25.321297    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-whp4x_cb1870fa-237b-4a9e-91b1-9741250e32d0/whereabouts-cni/0.log"
Sep 11 20:14:25.378842 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:25.378808    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-b4f5h_952e8fe9-61fe-447f-8b98-62fa962b1d7a/kube-multus/0.log"
Sep 11 20:14:25.587888 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:25.587801    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zzscf_e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef/network-metrics-daemon/0.log"
Sep 11 20:14:25.606581 ip-10-0-138-43 kubenswrapper[2558]: I0911 20:14:25.606548    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zzscf_e92e2eb8-6205-4e9a-9f99-9afdb7cbcdef/kube-rbac-proxy/0.log"