Sep 16 16:34:35.566492 ip-10-0-140-120 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 16:34:35.566503 ip-10-0-140-120 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 16:34:35.566511 ip-10-0-140-120 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 16:34:35.566656 ip-10-0-140-120 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 16:34:45.630958 ip-10-0-140-120 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 16:34:45.630974 ip-10-0-140-120 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot d85cf469f86b470182075fd6f2837a6c --
Sep 16 16:36:40.128211 ip-10-0-140-120 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 16:36:40.513157 ip-10-0-140-120 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 16:36:40.513157 ip-10-0-140-120 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 16:36:40.513157 ip-10-0-140-120 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 16:36:40.513157 ip-10-0-140-120 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 16:36:40.513157 ip-10-0-140-120 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 16:36:40.513844 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.513235    2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 16:36:40.516704 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516688    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 16:36:40.516704 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516703    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 16:36:40.516776 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516706    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 16:36:40.516776 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516710    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 16:36:40.516776 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516713    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 16:36:40.516776 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516716    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 16:36:40.516776 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516718    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 16:36:40.516776 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516721    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 16:36:40.516776 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516724    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 16:36:40.516776 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516727    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 16:36:40.516776 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516730    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 16:36:40.516776 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516733    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 16:36:40.516776 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516735    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 16:36:40.516776 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516738    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 16:36:40.516776 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516741    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 16:36:40.516776 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516744    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 16:36:40.516776 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516747    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 16:36:40.516776 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516749    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 16:36:40.516776 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516752    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 16:36:40.516776 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516754    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 16:36:40.516776 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516758    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 16:36:40.516776 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516761    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 16:36:40.517275 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516763    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 16:36:40.517275 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516766    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 16:36:40.517275 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516768    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 16:36:40.517275 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516771    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 16:36:40.517275 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516773    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 16:36:40.517275 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516776    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 16:36:40.517275 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516778    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 16:36:40.517275 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516782    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 16:36:40.517275 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516784    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 16:36:40.517275 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516787    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 16:36:40.517275 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516790    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 16:36:40.517275 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516793    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 16:36:40.517275 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516796    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 16:36:40.517275 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516799    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 16:36:40.517275 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516802    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 16:36:40.517275 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516804    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 16:36:40.517275 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516807    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 16:36:40.517275 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516809    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 16:36:40.517275 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516812    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 16:36:40.517275 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516814    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 16:36:40.517808 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516818    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 16:36:40.517808 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516822    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 16:36:40.517808 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516825    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 16:36:40.517808 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516827    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 16:36:40.517808 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516830    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 16:36:40.517808 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516832    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 16:36:40.517808 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516835    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 16:36:40.517808 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516838    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 16:36:40.517808 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516840    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 16:36:40.517808 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516843    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 16:36:40.517808 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516845    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 16:36:40.517808 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516848    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 16:36:40.517808 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516852    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 16:36:40.517808 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516855    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 16:36:40.517808 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516859    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 16:36:40.517808 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516862    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 16:36:40.517808 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516865    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 16:36:40.517808 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516868    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 16:36:40.517808 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516871    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 16:36:40.518571 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516876    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 16:36:40.518571 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516880    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 16:36:40.518571 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516883    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 16:36:40.518571 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516886    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 16:36:40.518571 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516890    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 16:36:40.518571 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516892    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 16:36:40.518571 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516895    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 16:36:40.518571 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516898    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 16:36:40.518571 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516900    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 16:36:40.518571 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516903    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 16:36:40.518571 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516906    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 16:36:40.518571 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516909    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 16:36:40.518571 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516912    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 16:36:40.518571 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516914    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 16:36:40.518571 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516917    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 16:36:40.518571 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516920    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 16:36:40.518571 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516922    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 16:36:40.518571 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516925    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 16:36:40.518571 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516928    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 16:36:40.518571 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516930    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 16:36:40.519243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516933    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 16:36:40.519243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516948    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 16:36:40.519243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516951    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 16:36:40.519243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516954    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 16:36:40.519243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516957    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 16:36:40.519243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516960    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 16:36:40.519243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516962    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 16:36:40.519243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516965    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 16:36:40.519243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516967    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 16:36:40.519243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516970    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 16:36:40.519243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516973    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 16:36:40.519243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516975    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 16:36:40.519243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516978    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 16:36:40.519243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516981    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 16:36:40.519243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516983    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 16:36:40.519243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516986    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 16:36:40.519243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516988    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 16:36:40.519243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.516991    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 16:36:40.519769 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519754    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 16:36:40.519769 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519769    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519774    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519777    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519781    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519784    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519787    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519790    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519792    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519795    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519798    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519800    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519803    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519806    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519809    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519811    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519814    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519817    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519823    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519826    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519829    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519831    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 16:36:40.519830 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519834    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 16:36:40.520436 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519837    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 16:36:40.520436 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519840    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 16:36:40.520436 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519842    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 16:36:40.520436 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519845    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 16:36:40.520436 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519848    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 16:36:40.520436 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519850    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 16:36:40.520436 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519853    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 16:36:40.520436 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519855    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 16:36:40.520436 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519858    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 16:36:40.520436 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519860    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 16:36:40.520436 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519863    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 16:36:40.520436 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519866    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 16:36:40.520436 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519868    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 16:36:40.520436 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519871    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 16:36:40.520436 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519873    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 16:36:40.520436 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519875    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 16:36:40.520436 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519878    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 16:36:40.520436 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519880    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 16:36:40.520436 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519883    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 16:36:40.520436 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519885    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 16:36:40.521012 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519888    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 16:36:40.521012 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519890    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 16:36:40.521012 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519893    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 16:36:40.521012 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519895    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 16:36:40.521012 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519897    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 16:36:40.521012 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519900    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 16:36:40.521012 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519902    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 16:36:40.521012 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519912    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 16:36:40.521012 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519915    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 16:36:40.521012 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519920    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 16:36:40.521012 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519924    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 16:36:40.521012 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519926    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 16:36:40.521012 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519929    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 16:36:40.521012 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519932    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 16:36:40.521012 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519935    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 16:36:40.521012 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519951    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 16:36:40.521012 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519956    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 16:36:40.521012 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519959    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 16:36:40.521012 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519962    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 16:36:40.521548 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519965    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 16:36:40.521548 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519967    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 16:36:40.521548 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519970    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 16:36:40.521548 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519973    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 16:36:40.521548 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519976    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 16:36:40.521548 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519978    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 16:36:40.521548 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519981    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 16:36:40.521548 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519983    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 16:36:40.521548 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519986    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 16:36:40.521548 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519989    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 16:36:40.521548 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519991    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 16:36:40.521548 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519994    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 16:36:40.521548 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519997    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 16:36:40.521548 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.519999    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 16:36:40.521548 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520002    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 16:36:40.521548 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520004    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 16:36:40.521548 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520007    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 16:36:40.521548 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520009    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 16:36:40.521548 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520012    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 16:36:40.521548 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520015    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 16:36:40.522105 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520017    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 16:36:40.522105 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520026    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 16:36:40.522105 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520032    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 16:36:40.522105 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520035    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 16:36:40.522105 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520037    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 16:36:40.522105 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520040    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 16:36:40.522105 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520043    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 16:36:40.522105 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520045    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 16:36:40.522105 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520051    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 16:36:40.522105 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520054    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 16:36:40.522105 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520057    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 16:36:40.522105 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520060    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 16:36:40.522105 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520063    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 16:36:40.522105 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520066    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 16:36:40.522105 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520068    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 16:36:40.522105 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520071    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 16:36:40.522105 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520073    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 16:36:40.522105 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520454    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 16:36:40.522105 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520461    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 16:36:40.522608 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520465    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 16:36:40.522608 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520468    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 16:36:40.522608 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520471    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 16:36:40.522608 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520474    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 16:36:40.522608 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520476    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 16:36:40.522608 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520479    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 16:36:40.522608 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520482    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 16:36:40.522608 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520485    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 16:36:40.522608 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520487    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 16:36:40.522608 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520490    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 16:36:40.522608 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520492    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 16:36:40.522608 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520495    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 16:36:40.522608 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520498    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 16:36:40.522608 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520501    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 16:36:40.522608 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520504    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 16:36:40.522608 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520507    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 16:36:40.522608 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520510    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 16:36:40.522608 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520512    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 16:36:40.522608 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520515    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 16:36:40.522608 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520517    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 16:36:40.523135 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520520    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 16:36:40.523135 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520523    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 16:36:40.523135 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520525    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 16:36:40.523135 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520528    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 16:36:40.523135 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520530    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 16:36:40.523135 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520533    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 16:36:40.523135 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520535    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 16:36:40.523135 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520538    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 16:36:40.523135 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520541    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 16:36:40.523135 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520543    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 16:36:40.523135 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520545    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 16:36:40.523135 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520548    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 16:36:40.523135 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520550    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 16:36:40.523135 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520553    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 16:36:40.523135 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520555    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 16:36:40.523135 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520558    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 16:36:40.523135 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520560    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 16:36:40.523135 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520563    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 16:36:40.523135 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520565    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 16:36:40.523135 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520567    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 16:36:40.523634 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520570    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 16:36:40.523634 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520572    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 16:36:40.523634 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520575    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 16:36:40.523634 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520578    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 16:36:40.523634 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520581    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 16:36:40.523634 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520585    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 16:36:40.523634 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520588    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 16:36:40.523634 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520592    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 16:36:40.523634 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520596    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 16:36:40.523634 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520599    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 16:36:40.523634 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520601    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 16:36:40.523634 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520606    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 16:36:40.523634 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520609    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 16:36:40.523634 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520612    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 16:36:40.523634 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520614    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 16:36:40.523634 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520617    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 16:36:40.523634 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520619    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 16:36:40.523634 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520622    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 16:36:40.523634 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520625    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 16:36:40.524154 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520628    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 16:36:40.524154 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520630    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 16:36:40.524154 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520632    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 16:36:40.524154 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520635    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 16:36:40.524154 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520638    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 16:36:40.524154 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520641    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 16:36:40.524154 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520644    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 16:36:40.524154 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520646    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 16:36:40.524154 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520648    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 16:36:40.524154 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520651    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 16:36:40.524154 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520654    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 16:36:40.524154 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520656    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 16:36:40.524154 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520660    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 16:36:40.524154 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520663    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 16:36:40.524154 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520666    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 16:36:40.524154 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520669    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 16:36:40.524154 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520671    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 16:36:40.524154 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520673    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 16:36:40.524154 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520676    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 16:36:40.524154 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520678    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 16:36:40.524660 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520681    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 16:36:40.524660 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520684    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 16:36:40.524660 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520687    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 16:36:40.524660 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520689    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 16:36:40.524660 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520693    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 16:36:40.524660 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520696    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 16:36:40.524660 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520698    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 16:36:40.524660 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520701    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 16:36:40.524660 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520703    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 16:36:40.524660 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520706    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 16:36:40.524660 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520708    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 16:36:40.524660 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520711    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 16:36:40.524660 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520713    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 16:36:40.524660 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520716    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 16:36:40.524660 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520718    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 16:36:40.524660 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520720    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 16:36:40.524660 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520723    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 16:36:40.524660 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.520725    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 16:36:40.524660 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520788    2556 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 16:36:40.524660 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520821    2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520828    2556 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520835    2556 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520839    2556 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520842    2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520846    2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520849    2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520853    2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520855    2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520859    2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520862    2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520865    2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520868    2556 flags.go:64] FLAG: --cgroup-root=""
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520870    2556 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520873    2556 flags.go:64] FLAG: --client-ca-file=""
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520878    2556 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520881    2556 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520886    2556 flags.go:64] FLAG: --cluster-domain=""
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520889    2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520892    2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520895    2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520898    2556 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520906    2556 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 16:36:40.525160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520909    2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520912    2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520916    2556 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520919    2556 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520922    2556 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520925    2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520928    2556 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520931    2556 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520935    2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520952    2556 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520956    2556 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520960    2556 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520963    2556 flags.go:64] FLAG: --enable-server="true"
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520965    2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520970    2556 flags.go:64] FLAG: --event-burst="100"
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520973    2556 flags.go:64] FLAG: --event-qps="50"
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520976    2556 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520979    2556 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520982    2556 flags.go:64] FLAG: --eviction-hard=""
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520986    2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520989    2556 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520992    2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520995    2556 flags.go:64] FLAG: --eviction-soft=""
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.520998    2556 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521000    2556 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 16:36:40.525738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521004    2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521008    2556 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521011    2556 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521014    2556 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521017    2556 flags.go:64] FLAG: --feature-gates=""
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521021    2556 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521024    2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521027    2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521030    2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521033    2556 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521036    2556 flags.go:64] FLAG: --help="false"
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521039    2556 flags.go:64] FLAG: --hostname-override="ip-10-0-140-120.ec2.internal"
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521042    2556 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521045    2556 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521048    2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521051    2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521054    2556 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521057    2556 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521061    2556 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521064    2556 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521067    2556 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521070    2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521073    2556 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521076    2556 flags.go:64] FLAG: --kube-reserved=""
Sep 16 16:36:40.526361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521079    2556 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521082    2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521085    2556 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521087    2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521090    2556 flags.go:64] FLAG: --lock-file=""
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521093    2556 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521096    2556 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521099    2556 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521104    2556 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521108    2556 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521111    2556 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521114    2556 flags.go:64] FLAG: --logging-format="text"
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521117    2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521120    2556 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521123    2556 flags.go:64] FLAG: --manifest-url=""
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521126    2556 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521130    2556 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521133    2556 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521137    2556 flags.go:64] FLAG: --max-pods="110"
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521140    2556 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521143    2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521146    2556 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521149    2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521152    2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521155    2556 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 16:36:40.526956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521157    2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521165    2556 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521168    2556 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521171    2556 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521174    2556 flags.go:64] FLAG: --pod-cidr=""
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521177    2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521183    2556 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521185    2556 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521188    2556 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521192    2556 flags.go:64] FLAG: --port="10250"
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521195    2556 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521198    2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-011c7531548c5964c"
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521201    2556 flags.go:64] FLAG: --qos-reserved=""
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521204    2556 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521207    2556 flags.go:64] FLAG: --register-node="true"
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521210    2556 flags.go:64] FLAG: --register-with-taints=""
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521214    2556 flags.go:64] FLAG: --registry-burst="10"
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521218    2556 flags.go:64] FLAG: --registry-qps="5"
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521221    2556 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521224    2556 flags.go:64] FLAG: --reserved-memory=""
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521227    2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521230    2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521233    2556 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521236    2556 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521239    2556 flags.go:64] FLAG: --runonce="false"
Sep 16 16:36:40.527565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521242    2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521245    2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521248    2556 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521251    2556 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521253    2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521256    2556 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521259    2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521262    2556 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521265    2556 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521270    2556 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521273    2556 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521276    2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521279    2556 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521282    2556 flags.go:64] FLAG: --system-cgroups=""
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521285    2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521291    2556 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521293    2556 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521296    2556 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521300    2556 flags.go:64] FLAG: --tls-min-version=""
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521303    2556 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521306    2556 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521309    2556 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521311    2556 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521314    2556 flags.go:64] FLAG: --v="2"
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521318    2556 flags.go:64] FLAG: --version="false"
Sep 16 16:36:40.528240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521323    2556 flags.go:64] FLAG: --vmodule=""
Sep 16 16:36:40.528887 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521327    2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 16:36:40.528887 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.521330    2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 16:36:40.528887 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521436    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 16:36:40.528887 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521440    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 16:36:40.528887 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521444    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 16:36:40.528887 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521446    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 16:36:40.528887 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521449    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 16:36:40.528887 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521452    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 16:36:40.528887 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521454    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 16:36:40.528887 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521457    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 16:36:40.528887 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521460    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 16:36:40.528887 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521462    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 16:36:40.528887 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521465    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 16:36:40.528887 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521468    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 16:36:40.528887 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521470    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 16:36:40.528887 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521474    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 16:36:40.528887 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521476    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 16:36:40.528887 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521479    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 16:36:40.528887 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521482    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 16:36:40.528887 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521485    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 16:36:40.529394 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521487    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 16:36:40.529394 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521490    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 16:36:40.529394 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521492    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 16:36:40.529394 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521496    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 16:36:40.529394 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521500    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 16:36:40.529394 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521504    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 16:36:40.529394 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521506    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 16:36:40.529394 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521509    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 16:36:40.529394 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521512    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 16:36:40.529394 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521514    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 16:36:40.529394 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521517    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 16:36:40.529394 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521519    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 16:36:40.529394 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521523    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 16:36:40.529394 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521525    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 16:36:40.529394 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521528    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 16:36:40.529394 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521530    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 16:36:40.529394 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521533    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 16:36:40.529394 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521536    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 16:36:40.529394 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521538    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 16:36:40.529394 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521541    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 16:36:40.529873 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521543    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 16:36:40.529873 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521546    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 16:36:40.529873 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521549    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 16:36:40.529873 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521551    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 16:36:40.529873 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521554    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 16:36:40.529873 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521559    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 16:36:40.529873 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521561    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 16:36:40.529873 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521566    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 16:36:40.529873 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521577    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 16:36:40.529873 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521579    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 16:36:40.529873 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521582    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 16:36:40.529873 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521585    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 16:36:40.529873 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521588    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 16:36:40.529873 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521590    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 16:36:40.529873 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521593    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 16:36:40.529873 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521595    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 16:36:40.529873 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521598    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 16:36:40.529873 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521600    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 16:36:40.529873 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521603    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 16:36:40.529873 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521606    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 16:36:40.530379 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521608    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 16:36:40.530379 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521612    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 16:36:40.530379 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521616    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 16:36:40.530379 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521618    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 16:36:40.530379 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521621    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 16:36:40.530379 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521624    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 16:36:40.530379 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521627    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 16:36:40.530379 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521629    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 16:36:40.530379 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521632    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 16:36:40.530379 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521634    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 16:36:40.530379 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521637    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 16:36:40.530379 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521639    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 16:36:40.530379 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521642    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 16:36:40.530379 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521644    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 16:36:40.530379 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521647    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 16:36:40.530379 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521649    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 16:36:40.530379 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521652    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 16:36:40.530379 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521656    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 16:36:40.530379 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521658    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 16:36:40.530846 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521662    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 16:36:40.530846 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521664    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 16:36:40.530846 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521667    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 16:36:40.530846 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521670    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 16:36:40.530846 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521672    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 16:36:40.530846 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521675    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 16:36:40.530846 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521678    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 16:36:40.530846 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521681    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 16:36:40.530846 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521683    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 16:36:40.530846 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521687    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 16:36:40.530846 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521691    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 16:36:40.530846 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521694    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 16:36:40.530846 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521696    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 16:36:40.530846 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521699    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 16:36:40.530846 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521702    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 16:36:40.530846 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521704    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 16:36:40.530846 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521707    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 16:36:40.530846 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521710    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 16:36:40.530846 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521712    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 16:36:40.530846 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521715    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 16:36:40.531347 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521718    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 16:36:40.531347 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.521721    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 16:36:40.531347 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.522205    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 16:36:40.531347 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.529052    2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 16:36:40.531347 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.529068    2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 16:36:40.531347 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.529081    2556 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 16:36:40.531347 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.529088    2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 16:36:40.531347 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529142    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 16:36:40.531347 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529147    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 16:36:40.531347 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529150    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 16:36:40.531347 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529153    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 16:36:40.531347 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529156    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 16:36:40.531347 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529159    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 16:36:40.531347 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529162    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 16:36:40.531347 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529165    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 16:36:40.531727 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529167    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 16:36:40.531727 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529172    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 16:36:40.531727 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529176    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 16:36:40.531727 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529179    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 16:36:40.531727 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529182    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 16:36:40.531727 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529185    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 16:36:40.531727 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529188    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 16:36:40.531727 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529191    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 16:36:40.531727 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529195    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 16:36:40.531727 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529199    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 16:36:40.531727 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529203    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 16:36:40.531727 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529206    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 16:36:40.531727 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529209    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 16:36:40.531727 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529212    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 16:36:40.531727 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529214    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 16:36:40.531727 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529217    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 16:36:40.531727 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529219    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 16:36:40.531727 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529222    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 16:36:40.531727 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529224    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 16:36:40.532234 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529227    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 16:36:40.532234 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529230    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 16:36:40.532234 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529234    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 16:36:40.532234 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529237    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 16:36:40.532234 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529240    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 16:36:40.532234 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529243    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 16:36:40.532234 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529246    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 16:36:40.532234 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529248    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 16:36:40.532234 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529251    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 16:36:40.532234 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529254    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 16:36:40.532234 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529256    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 16:36:40.532234 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529259    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 16:36:40.532234 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529261    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 16:36:40.532234 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529264    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 16:36:40.532234 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529267    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 16:36:40.532234 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529269    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 16:36:40.532234 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529272    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 16:36:40.532234 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529274    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 16:36:40.532234 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529277    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 16:36:40.532234 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529279    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 16:36:40.532736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529282    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 16:36:40.532736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529284    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 16:36:40.532736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529287    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 16:36:40.532736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529290    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 16:36:40.532736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529292    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 16:36:40.532736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529295    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 16:36:40.532736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529297    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 16:36:40.532736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529300    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 16:36:40.532736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529303    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 16:36:40.532736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529305    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 16:36:40.532736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529308    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 16:36:40.532736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529311    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 16:36:40.532736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529313    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 16:36:40.532736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529316    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 16:36:40.532736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529319    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 16:36:40.532736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529323    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 16:36:40.532736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529326    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 16:36:40.532736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529328    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 16:36:40.532736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529331    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 16:36:40.532736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529333    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 16:36:40.533243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529336    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 16:36:40.533243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529339    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 16:36:40.533243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529341    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 16:36:40.533243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529344    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 16:36:40.533243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529346    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 16:36:40.533243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529349    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 16:36:40.533243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529352    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 16:36:40.533243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529354    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 16:36:40.533243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529357    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 16:36:40.533243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529359    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 16:36:40.533243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529362    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 16:36:40.533243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529364    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 16:36:40.533243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529367    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 16:36:40.533243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529370    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 16:36:40.533243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529372    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 16:36:40.533243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529375    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 16:36:40.533243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529377    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 16:36:40.533243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529380    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 16:36:40.533243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529382    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 16:36:40.533243 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529385    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 16:36:40.533723 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529387    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 16:36:40.533723 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529390    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 16:36:40.533723 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529394    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 16:36:40.533723 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529398    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 16:36:40.533723 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529401    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 16:36:40.533723 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529404    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 16:36:40.533723 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529408    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 16:36:40.533723 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529411    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 16:36:40.533723 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529414    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 16:36:40.533723 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529417    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 16:36:40.533723 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529419    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 16:36:40.533723 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529422    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 16:36:40.533723 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.529427    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 16:36:40.533723 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529537    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 16:36:40.534095 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529542    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 16:36:40.534095 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529546    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 16:36:40.534095 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529549    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 16:36:40.534095 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529552    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 16:36:40.534095 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529555    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 16:36:40.534095 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529558    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 16:36:40.534095 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529561    2556 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 16:36:40.534095 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529565    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 16:36:40.534095 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529568    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 16:36:40.534095 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529570    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 16:36:40.534095 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529573    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 16:36:40.534095 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529576    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 16:36:40.534095 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529579    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 16:36:40.534095 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529582    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 16:36:40.534095 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529584    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 16:36:40.534095 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529587    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 16:36:40.534095 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529590    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 16:36:40.534095 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529593    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 16:36:40.534095 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529595    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 16:36:40.534095 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529598    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 16:36:40.534590 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529601    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 16:36:40.534590 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529603    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 16:36:40.534590 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529606    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 16:36:40.534590 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529609    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 16:36:40.534590 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529612    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 16:36:40.534590 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529615    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 16:36:40.534590 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529618    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 16:36:40.534590 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529621    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 16:36:40.534590 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529623    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 16:36:40.534590 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529626    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 16:36:40.534590 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529629    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 16:36:40.534590 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529631    2556 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 16:36:40.534590 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529634    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 16:36:40.534590 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529636    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 16:36:40.534590 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529639    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 16:36:40.534590 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529642    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 16:36:40.534590 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529645    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 16:36:40.534590 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529647    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 16:36:40.534590 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529649    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 16:36:40.534590 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529652    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 16:36:40.535099 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529654    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 16:36:40.535099 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529657    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 16:36:40.535099 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529659    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 16:36:40.535099 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529662    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 16:36:40.535099 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529665    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 16:36:40.535099 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529667    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 16:36:40.535099 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529670    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 16:36:40.535099 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529673    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 16:36:40.535099 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529675    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 16:36:40.535099 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529678    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 16:36:40.535099 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529680    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 16:36:40.535099 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529683    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 16:36:40.535099 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529685    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 16:36:40.535099 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529688    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 16:36:40.535099 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529690    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 16:36:40.535099 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529693    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 16:36:40.535099 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529696    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 16:36:40.535099 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529699    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 16:36:40.535099 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529701    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 16:36:40.535099 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529704    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 16:36:40.535606 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529710    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 16:36:40.535606 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529712    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 16:36:40.535606 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529715    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 16:36:40.535606 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529718    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 16:36:40.535606 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529721    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 16:36:40.535606 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529725    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 16:36:40.535606 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529728    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 16:36:40.535606 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529730    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 16:36:40.535606 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529733    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 16:36:40.535606 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529736    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 16:36:40.535606 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529738    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 16:36:40.535606 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529742    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 16:36:40.535606 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529745    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 16:36:40.535606 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529748    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 16:36:40.535606 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529751    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 16:36:40.535606 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529754    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 16:36:40.535606 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529756    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 16:36:40.535606 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529759    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 16:36:40.535606 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529762    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 16:36:40.536166 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529764    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 16:36:40.536166 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529767    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 16:36:40.536166 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529769    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 16:36:40.536166 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529772    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 16:36:40.536166 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529775    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 16:36:40.536166 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529777    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 16:36:40.536166 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529780    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 16:36:40.536166 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529782    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 16:36:40.536166 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529785    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 16:36:40.536166 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529788    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 16:36:40.536166 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529790    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 16:36:40.536166 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529793    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 16:36:40.536166 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529795    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 16:36:40.536166 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529798    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 16:36:40.536166 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529801    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 16:36:40.536166 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529804    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 16:36:40.536166 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529807    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 16:36:40.536166 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529809    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 16:36:40.536166 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:40.529811    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 16:36:40.536633 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.529816    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 16:36:40.536633 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.529907    2556 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 16:36:40.536633 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.531953    2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 16:36:40.536633 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.532842    2556 server.go:1019] "Starting client certificate rotation"
Sep 16 16:36:40.536633 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.532950    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 16:36:40.536633 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.532986    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 16:36:40.536633 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.535303    2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 16:36:40.538707 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.538693    2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 16:36:40.549490 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.549474    2556 log.go:25] "Validated CRI v1 runtime API"
Sep 16 16:36:40.553221 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.553207    2556 log.go:25] "Validated CRI v1 image API"
Sep 16 16:36:40.554854 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.554832    2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 16:36:40.558191 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.558169    2556 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9325d317-8d72-468d-b788-386d4f28f11f:/dev/nvme0n1p4 de5fecde-6825-4467-bdc0-cf5e375ae2bb:/dev/nvme0n1p3]
Sep 16 16:36:40.558273 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.558190    2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 16:36:40.562978 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.562837    2556 manager.go:216] Machine: {Timestamp:2026-09-16 16:36:40.56173782 +0000 UTC m=+0.323699258 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100161 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec238da2c5c538b8151308cde49e91b5 SystemUUID:ec238da2-c5c5-38b8-1513-08cde49e91b5 BootID:d85cf469-f86b-4701-8207-5fd6f2837a6c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cd:3c:ad:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cd:3c:ad:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:79:c1:3f:c6:c6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 16:36:40.562978 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.562967    2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 16:36:40.563135 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.563070    2556 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 16:36:40.563999 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.563972    2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 16:36:40.564148 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.564001    2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-120.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 16:36:40.564767 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.564756    2556 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 16:36:40.564820 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.564772    2556 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 16:36:40.564820 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.564782    2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 16:36:40.564962 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.564932    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 16:36:40.565462 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.565450    2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 16:36:40.566175 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.566163    2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 16:36:40.569925 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.569911    2556 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 16:36:40.570058 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.570047    2556 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 16:36:40.570113 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.570087    2556 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 16:36:40.570113 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.570101    2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 16:36:40.570204 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.570119    2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 16:36:40.570204 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.570132    2556 kubelet.go:398] "Adding apiserver pod source"
Sep 16 16:36:40.570204 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.570144    2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 16:36:40.570485 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.570471    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 16:36:40.570551 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.570494    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 16:36:40.572573 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.572549    2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 16:36:40.574642 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.574625    2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 16:36:40.574731 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.574664    2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 16:36:40.576349 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.576336    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 16:36:40.576427 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.576355    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 16:36:40.576427 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.576364    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 16:36:40.576427 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.576383    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 16:36:40.576427 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.576397    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 16:36:40.576427 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.576406    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 16:36:40.576427 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.576425    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 16:36:40.576659 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.576434    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 16:36:40.576659 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.576444    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 16:36:40.576659 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.576452    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 16:36:40.576659 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.576472    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 16:36:40.576659 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.576480    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 16:36:40.577333 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.577322    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 16:36:40.577398 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.577336    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 16:36:40.580734 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.580715    2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-120.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 16:36:40.580832 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.580812    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-120.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 16:36:40.580890 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.580814    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 16:36:40.581120 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.581107    2556 server.go:1268] "Started kubelet"
Sep 16 16:36:40.581235 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.581184    2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 16:36:40.581294 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.581252    2556 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 16:36:40.581341 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.581306    2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 16:36:40.581881 ip-10-0-140-120 systemd[1]: Started Kubernetes Kubelet.
Sep 16 16:36:40.586526 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.586494    2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 16:36:40.587697 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.587660    2556 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 16:36:40.588596 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.588577    2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 16:36:40.591784 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.591770    2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 16:36:40.591863 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.591790    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 16:36:40.592008 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.591986    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-120.ec2.internal\" not found"
Sep 16 16:36:40.592331 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.592313    2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 16:36:40.592331 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.592311    2556 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 16:36:40.592481 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.592342    2556 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 16:36:40.592481 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.591560    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-120.ec2.internal.18d5da6fc7fc8da2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-120.ec2.internal,UID:ip-10-0-140-120.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-120.ec2.internal,},FirstTimestamp:2026-09-16 16:36:40.58108253 +0000 UTC m=+0.343043968,LastTimestamp:2026-09-16 16:36:40.58108253 +0000 UTC m=+0.343043968,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-120.ec2.internal,}"
Sep 16 16:36:40.592596 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.592512    2556 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 16:36:40.592596 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.592520    2556 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 16:36:40.592765 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.592604    2556 factory.go:55] Registering systemd factory
Sep 16 16:36:40.592811 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.592769    2556 factory.go:223] Registration of the systemd container factory successfully
Sep 16 16:36:40.592993 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.592981    2556 factory.go:153] Registering CRI-O factory
Sep 16 16:36:40.593046 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.592997    2556 factory.go:223] Registration of the crio container factory successfully
Sep 16 16:36:40.593046 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.593042    2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 16:36:40.593109 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.593059    2556 factory.go:103] Registering Raw factory
Sep 16 16:36:40.593109 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.593072    2556 manager.go:1196] Started watching for new ooms in manager
Sep 16 16:36:40.593340 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.593330    2556 manager.go:349] Starting recovery of all containers
Sep 16 16:36:40.604453 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.604434    2556 manager.go:354] Recovery completed
Sep 16 16:36:40.607207 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.607094    2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-120.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 16:36:40.607207 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.607115    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 16:36:40.611089 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.611077    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 16:36:40.611744 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.611728    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lwbr5"
Sep 16 16:36:40.613438 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.613423    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 16:36:40.613522 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.613451    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 16:36:40.613522 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.613464    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 16:36:40.614154 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.614123    2556 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 16:36:40.614154 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.614148    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 16:36:40.614307 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.614153    2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 16:36:40.614307 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.614212    2556 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 16:36:40.614897 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.614881    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 16:36:40.614970 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.614912    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 16:36:40.614970 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.614925    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 16:36:40.616396 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.616383    2556 policy_none.go:49] "None policy: Start"
Sep 16 16:36:40.616449 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.616400    2556 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 16:36:40.616449 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.616410    2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 16:36:40.618285 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.618273    2556 policy_none.go:47] "Start"
Sep 16 16:36:40.621171 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.621152    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lwbr5"
Sep 16 16:36:40.626827 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.626740    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-120.ec2.internal.18d5da6fc9ea3e12  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-120.ec2.internal,UID:ip-10-0-140-120.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-120.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-120.ec2.internal,},FirstTimestamp:2026-09-16 16:36:40.613436946 +0000 UTC m=+0.375398385,LastTimestamp:2026-09-16 16:36:40.613436946 +0000 UTC m=+0.375398385,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-120.ec2.internal,}"
Sep 16 16:36:40.641576 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.641514    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-120.ec2.internal.18d5da6fc9ea86e8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-120.ec2.internal,UID:ip-10-0-140-120.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-120.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-120.ec2.internal,},FirstTimestamp:2026-09-16 16:36:40.613455592 +0000 UTC m=+0.375417031,LastTimestamp:2026-09-16 16:36:40.613455592 +0000 UTC m=+0.375417031,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-120.ec2.internal,}"
Sep 16 16:36:40.760852 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.652736    2556 manager.go:337] "Starting Device Plugin manager"
Sep 16 16:36:40.760852 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.652766    2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 16:36:40.760852 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.652774    2556 server.go:87] "Starting device plugin registration server"
Sep 16 16:36:40.760852 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.653077    2556 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 16:36:40.760852 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.653087    2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 16:36:40.760852 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.653679    2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 16:36:40.760852 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.653760    2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 16:36:40.760852 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.653769    2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 16:36:40.760852 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.653827    2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 16:36:40.760852 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.653880    2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-120.ec2.internal\" not found"
Sep 16 16:36:40.760852 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.664449    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-120.ec2.internal.18d5da6fc9eac126  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-120.ec2.internal,UID:ip-10-0-140-120.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-120.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-120.ec2.internal,},FirstTimestamp:2026-09-16 16:36:40.613470502 +0000 UTC m=+0.375431944,LastTimestamp:2026-09-16 16:36:40.613470502 +0000 UTC m=+0.375431944,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-120.ec2.internal,}"
Sep 16 16:36:40.760852 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.675265    2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-120.ec2.internal.18d5da6fc9ea3e12\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-120.ec2.internal.18d5da6fc9ea3e12  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-120.ec2.internal,UID:ip-10-0-140-120.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-120.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-120.ec2.internal,},FirstTimestamp:2026-09-16 16:36:40.613436946 +0000 UTC m=+0.375398385,LastTimestamp:2026-09-16 16:36:40.614896914 +0000 UTC m=+0.376858356,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-120.ec2.internal,}"
Sep 16 16:36:40.760852 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.679869    2556 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2590 into the system container \"/system.slice\": <nil>"
Sep 16 16:36:40.760852 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.685648    2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-120.ec2.internal.18d5da6fc9ea86e8\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-120.ec2.internal.18d5da6fc9ea86e8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-120.ec2.internal,UID:ip-10-0-140-120.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-120.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-120.ec2.internal,},FirstTimestamp:2026-09-16 16:36:40.613455592 +0000 UTC m=+0.375417031,LastTimestamp:2026-09-16 16:36:40.614918508 +0000 UTC m=+0.376879949,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-120.ec2.internal,}"
Sep 16 16:36:40.760852 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.697004    2556 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-140-120.ec2.internal.18d5da6fc9eac126\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-120.ec2.internal.18d5da6fc9eac126  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-120.ec2.internal,UID:ip-10-0-140-120.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-120.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-120.ec2.internal,},FirstTimestamp:2026-09-16 16:36:40.613470502 +0000 UTC m=+0.375431944,LastTimestamp:2026-09-16 16:36:40.614930126 +0000 UTC m=+0.376891569,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-120.ec2.internal,}"
Sep 16 16:36:40.761931 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.711446    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-120.ec2.internal.18d5da6fccf28510  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-120.ec2.internal,UID:ip-10-0-140-120.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-140-120.ec2.internal,},FirstTimestamp:2026-09-16 16:36:40.664311056 +0000 UTC m=+0.426272481,LastTimestamp:2026-09-16 16:36:40.664311056 +0000 UTC m=+0.426272481,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-120.ec2.internal,}"
Sep 16 16:36:40.761931 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.753543    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 16:36:40.761931 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.754405    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 16:36:40.761931 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.754432    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 16:36:40.761931 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.754443    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 16:36:40.761931 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.754466    2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-120.ec2.internal"
Sep 16 16:36:40.776182 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.776117    2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-120.ec2.internal"
Sep 16 16:36:40.776182 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.776137    2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-120.ec2.internal\": node \"ip-10-0-140-120.ec2.internal\" not found"
Sep 16 16:36:40.816157 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.816135    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-120.ec2.internal\" not found"
Sep 16 16:36:40.840554 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.840517    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 16:36:40.841788 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.841775    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 16:36:40.841839 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.841820    2556 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 16:36:40.841875 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.841843    2556 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 16:36:40.841907 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.841888    2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 16:36:40.844695 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.844678    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 16:36:40.916580 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.916549    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-120.ec2.internal\" not found"
Sep 16 16:36:40.942705 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.942666    2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal"]
Sep 16 16:36:40.942765 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.942753    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 16:36:40.944171 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.944156    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 16:36:40.944231 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.944186    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 16:36:40.944231 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.944199    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 16:36:40.946520 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.946508    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 16:36:40.946650 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.946637    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal"
Sep 16 16:36:40.946685 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.946665    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 16:36:40.948175 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.948160    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 16:36:40.948264 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.948176    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 16:36:40.948264 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.948190    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 16:36:40.948264 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.948202    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 16:36:40.948264 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.948203    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 16:36:40.948264 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.948218    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 16:36:40.950439 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.950422    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal"
Sep 16 16:36:40.950526 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.950452    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 16:36:40.951219 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.951200    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 16:36:40.951304 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.951232    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 16:36:40.951304 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:40.951248    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 16:36:40.972598 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.972579    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-120.ec2.internal\" not found" node="ip-10-0-140-120.ec2.internal"
Sep 16 16:36:40.976960 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:40.976928    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-120.ec2.internal\" not found" node="ip-10-0-140-120.ec2.internal"
Sep 16 16:36:41.016649 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:41.016629    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-120.ec2.internal\" not found"
Sep 16 16:36:41.094872 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.094809    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/242eb3e6d5cc43fa8cbc80becad32de3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal\" (UID: \"242eb3e6d5cc43fa8cbc80becad32de3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal"
Sep 16 16:36:41.094872 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.094846    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4181901193613f3e77dd46929737ec7c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-120.ec2.internal\" (UID: \"4181901193613f3e77dd46929737ec7c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal"
Sep 16 16:36:41.095007 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.094885    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/242eb3e6d5cc43fa8cbc80becad32de3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal\" (UID: \"242eb3e6d5cc43fa8cbc80becad32de3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal"
Sep 16 16:36:41.116744 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:41.116728    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-120.ec2.internal\" not found"
Sep 16 16:36:41.195582 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.195560    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/242eb3e6d5cc43fa8cbc80becad32de3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal\" (UID: \"242eb3e6d5cc43fa8cbc80becad32de3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal"
Sep 16 16:36:41.195661 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.195584    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/242eb3e6d5cc43fa8cbc80becad32de3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal\" (UID: \"242eb3e6d5cc43fa8cbc80becad32de3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal"
Sep 16 16:36:41.195661 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.195601    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4181901193613f3e77dd46929737ec7c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-120.ec2.internal\" (UID: \"4181901193613f3e77dd46929737ec7c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal"
Sep 16 16:36:41.195661 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.195643    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4181901193613f3e77dd46929737ec7c-config\") pod \"kube-apiserver-proxy-ip-10-0-140-120.ec2.internal\" (UID: \"4181901193613f3e77dd46929737ec7c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal"
Sep 16 16:36:41.195758 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.195659    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/242eb3e6d5cc43fa8cbc80becad32de3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal\" (UID: \"242eb3e6d5cc43fa8cbc80becad32de3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal"
Sep 16 16:36:41.195758 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.195659    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/242eb3e6d5cc43fa8cbc80becad32de3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal\" (UID: \"242eb3e6d5cc43fa8cbc80becad32de3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal"
Sep 16 16:36:41.217690 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:41.217674    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-120.ec2.internal\" not found"
Sep 16 16:36:41.275835 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.275820    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal"
Sep 16 16:36:41.279247 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.279233    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal"
Sep 16 16:36:41.318571 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:41.318542    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-120.ec2.internal\" not found"
Sep 16 16:36:41.419601 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:41.419527    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-120.ec2.internal\" not found"
Sep 16 16:36:41.456998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.456979    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 16:36:41.474333 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.474311    2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 16:36:41.519951 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:41.519916    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-120.ec2.internal\" not found"
Sep 16 16:36:41.533044 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.533024    2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 16:36:41.533158 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.533138    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 16:36:41.533214 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.533178    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 16:36:41.533214 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.533186    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 16:36:41.620266 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:41.620245    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-120.ec2.internal\" not found"
Sep 16 16:36:41.672487 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.672437    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 16:36:41.673431 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.673415    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 16:36:41.673507 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.673448    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 16:36:41.673507 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.673459    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeHasSufficientPID"
Sep 16 16:36:41.720602 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:41.720576    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-120.ec2.internal\" not found"
Sep 16 16:36:41.736800 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.736782    2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 16:36:41.764600 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.764575    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 16:31:41 +0000 UTC" deadline="2028-05-17 21:00:19.329661011 +0000 UTC"
Sep 16 16:36:41.764600 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.764597    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14620h23m37.565066594s"
Sep 16 16:36:41.792325 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.792297    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal"
Sep 16 16:36:41.807559 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.807537    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 16:36:41.808316 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.808303    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal"
Sep 16 16:36:41.821183 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:41.821163    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 16:36:42.051054 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:42.051016    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4181901193613f3e77dd46929737ec7c.slice/crio-703f02351675f831ece827bf5db0f82fedff3f63e86adebe6d05a3d8ca59dcde WatchSource:0}: Error finding container 703f02351675f831ece827bf5db0f82fedff3f63e86adebe6d05a3d8ca59dcde: Status 404 returned error can't find the container with id 703f02351675f831ece827bf5db0f82fedff3f63e86adebe6d05a3d8ca59dcde
Sep 16 16:36:42.051266 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:42.051246    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242eb3e6d5cc43fa8cbc80becad32de3.slice/crio-58f148b5c36c951e642d9cff0a40ddb2b24ef3d16484a72da81fbbc0078420d6 WatchSource:0}: Error finding container 58f148b5c36c951e642d9cff0a40ddb2b24ef3d16484a72da81fbbc0078420d6: Status 404 returned error can't find the container with id 58f148b5c36c951e642d9cff0a40ddb2b24ef3d16484a72da81fbbc0078420d6
Sep 16 16:36:42.056743 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.056725    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 16:36:42.572263 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.572224    2556 apiserver.go:52] "Watching apiserver"
Sep 16 16:36:42.573760 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.573725    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 16:36:42.578990 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.578954    2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 16:36:42.579581 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.579521    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-lktnv","openshift-network-diagnostics/network-check-target-79v7b","openshift-network-operator/iptables-alerter-m4kpr","openshift-ovn-kubernetes/ovnkube-node-n42qf","kube-system/global-pull-secret-syncer-kmw95","kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r","openshift-cluster-node-tuning-operator/tuned-mlt5x","openshift-dns/node-resolver-knvfz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal","openshift-multus/multus-additional-cni-plugins-g7ksk","openshift-multus/network-metrics-daemon-hk8q9","kube-system/konnectivity-agent-tdf5s","openshift-image-registry/node-ca-qf2sz"]
Sep 16 16:36:42.582447 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.582427    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.584876 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.584853    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tdf5s"
Sep 16 16:36:42.586037 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.585680    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2xcn8\""
Sep 16 16:36:42.586037 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.585809    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 16:36:42.586037 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.585867    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 16:36:42.586380 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.586355    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 16:36:42.587458 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.587024    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 16:36:42.587458 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.587409    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hj6r7\""
Sep 16 16:36:42.587990 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.587697    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m4kpr"
Sep 16 16:36:42.589639 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.589620    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 16:36:42.590135 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.589920    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 16:36:42.590294 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.590233    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 16:36:42.590400 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.590331    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.590785 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.590591    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 16:36:42.590785 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.590655    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-996wx\""
Sep 16 16:36:42.592293 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.592272    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 16:36:42.593427 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.593126    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:42.593427 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:42.593215    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmw95" podUID="cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792"
Sep 16 16:36:42.594810 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.594618    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 16:36:42.594810 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.594662    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 16:36:42.595424 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.595163    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 16:36:42.595424 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.595209    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 16:36:42.595424 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.595353    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-srf27\""
Sep 16 16:36:42.595424 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.595383    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 16:36:42.595700 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.595452    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 16:36:42.595700 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.595620    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qf2sz"
Sep 16 16:36:42.598295 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.598162    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-knvfz"
Sep 16 16:36:42.599630 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.599605    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 16:36:42.600681 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.600658    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 16:36:42.600777 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.600745    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-c6bp9\""
Sep 16 16:36:42.600867 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.600809    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 16:36:42.602425 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.602386    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.602534 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.602490    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 16:36:42.602602 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.602393    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m9d2z\""
Sep 16 16:36:42.602657 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.602619    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-socket-dir\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.602710 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.602661    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-log-socket\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.602752 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.602715    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-sys-fs\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.602788 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.602771    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b7cc8be-e4bc-477e-8166-f539c64992a4-agent-certs\") pod \"konnectivity-agent-tdf5s\" (UID: \"0b7cc8be-e4bc-477e-8166-f539c64992a4\") " pod="kube-system/konnectivity-agent-tdf5s"
Sep 16 16:36:42.602835 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.602812    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-env-overrides\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.602906 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.602855    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqhf4\" (UniqueName: \"kubernetes.io/projected/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-kube-api-access-vqhf4\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.602994 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.602896    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.603077 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.602999    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-registration-dir\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.603077 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.603028    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b7cc8be-e4bc-477e-8166-f539c64992a4-konnectivity-ca\") pod \"konnectivity-agent-tdf5s\" (UID: \"0b7cc8be-e4bc-477e-8166-f539c64992a4\") " pod="kube-system/konnectivity-agent-tdf5s"
Sep 16 16:36:42.603077 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.603059    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-dbus\") pod \"global-pull-secret-syncer-kmw95\" (UID: \"cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792\") " pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:42.603269 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.603165    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c7eef487-63a5-4d81-bf5a-40e3f673333e-iptables-alerter-script\") pod \"iptables-alerter-m4kpr\" (UID: \"c7eef487-63a5-4d81-bf5a-40e3f673333e\") " pod="openshift-network-operator/iptables-alerter-m4kpr"
Sep 16 16:36:42.603269 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.603177    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 16:36:42.603269 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.603205    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-kubelet\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.603269 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.603237    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-systemd-units\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.603472 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.603261    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-run-ovn-kubernetes\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.603472 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.603343    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-ovnkube-script-lib\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.603472 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.603396    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-cni-netd\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.603788 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.603502    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-device-dir\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.603788 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.603541    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-etc-selinux\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.603788 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.603575    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-ovn-node-metrics-cert\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.603788 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.603615    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-etc-openvswitch\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.603788 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.603718    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-run-openvswitch\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.603788 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.603752    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-slash\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.603788 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.603784    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-run-netns\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.604231 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.603858    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-run-systemd\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.605378 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.603897    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-run-ovn\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.605378 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.604588    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7eef487-63a5-4d81-bf5a-40e3f673333e-host-slash\") pod \"iptables-alerter-m4kpr\" (UID: \"c7eef487-63a5-4d81-bf5a-40e3f673333e\") " pod="openshift-network-operator/iptables-alerter-m4kpr"
Sep 16 16:36:42.605378 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.604617    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhfbq\" (UniqueName: \"kubernetes.io/projected/c7eef487-63a5-4d81-bf5a-40e3f673333e-kube-api-access-qhfbq\") pod \"iptables-alerter-m4kpr\" (UID: \"c7eef487-63a5-4d81-bf5a-40e3f673333e\") " pod="openshift-network-operator/iptables-alerter-m4kpr"
Sep 16 16:36:42.605378 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.604643    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-var-lib-openvswitch\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.605378 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.604668    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-kubelet-config\") pod \"global-pull-secret-syncer-kmw95\" (UID: \"cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792\") " pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:42.605378 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.604695    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg758\" (UniqueName: \"kubernetes.io/projected/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-kube-api-access-xg758\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.605378 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.604720    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-node-log\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.605378 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.604740    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-cni-bin\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.605378 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.604760    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-ovnkube-config\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.605378 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.604785    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret\") pod \"global-pull-secret-syncer-kmw95\" (UID: \"cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792\") " pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:42.605378 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.604806    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.606286 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.606215    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:36:42.606381 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:42.606286    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79v7b" podUID="568fe78a-6a27-4de3-a23b-030e47b4d930"
Sep 16 16:36:42.609287 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.608105    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 16:36:42.609287 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.608472    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 16:36:42.609287 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.608683    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 16:36:42.609287 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.608917    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 16:36:42.611686 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.610147    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.611686 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.611685    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-m255f\""
Sep 16 16:36:42.612807 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.612757    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.615043 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.613625    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-h4mxg\""
Sep 16 16:36:42.615043 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.614447    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 16:36:42.615043 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.614529    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 16:36:42.615043 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.614675    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 16:36:42.616208 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.615191    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:36:42.616208 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:42.615249    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk8q9" podUID="c04662fe-4176-4a0b-ac5e-89ead5941975"
Sep 16 16:36:42.616208 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.615448    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 16:36:42.616387 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.616228    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 16:36:42.616990 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.616883    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pbcsj\""
Sep 16 16:36:42.643651 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.643625    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jsv7v"
Sep 16 16:36:42.652903 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.652874    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jsv7v"
Sep 16 16:36:42.694223 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.694151    2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 16:36:42.705185 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705153    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-sys\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.705308 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705199    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02df9d68-24e0-43d9-8774-273f8b55c906-hosts-file\") pod \"node-resolver-knvfz\" (UID: \"02df9d68-24e0-43d9-8774-273f8b55c906\") " pod="openshift-dns/node-resolver-knvfz"
Sep 16 16:36:42.705308 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705232    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76rb7\" (UniqueName: \"kubernetes.io/projected/2009b752-ad4d-4e71-8a91-3b3b41c44725-kube-api-access-76rb7\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.705308 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705268    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-ovn-node-metrics-cert\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.705308 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705295    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-run-ovn\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.705527 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705320    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhfbq\" (UniqueName: \"kubernetes.io/projected/c7eef487-63a5-4d81-bf5a-40e3f673333e-kube-api-access-qhfbq\") pod \"iptables-alerter-m4kpr\" (UID: \"c7eef487-63a5-4d81-bf5a-40e3f673333e\") " pod="openshift-network-operator/iptables-alerter-m4kpr"
Sep 16 16:36:42.705527 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705355    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-var-lib-openvswitch\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.705527 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705381    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg758\" (UniqueName: \"kubernetes.io/projected/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-kube-api-access-xg758\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.705527 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705404    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-host-run-netns\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.705527 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705429    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-host-var-lib-cni-bin\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.705527 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705454    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-etc-kubernetes\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.705527 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705479    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx6bg\" (UniqueName: \"kubernetes.io/projected/02df9d68-24e0-43d9-8774-273f8b55c906-kube-api-access-kx6bg\") pod \"node-resolver-knvfz\" (UID: \"02df9d68-24e0-43d9-8774-273f8b55c906\") " pod="openshift-dns/node-resolver-knvfz"
Sep 16 16:36:42.705527 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705507    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-node-log\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.705998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705533    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret\") pod \"global-pull-secret-syncer-kmw95\" (UID: \"cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792\") " pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:42.705998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705558    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.705998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705582    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/321e79fa-919d-436e-a11d-a4525450a85c-host\") pod \"node-ca-qf2sz\" (UID: \"321e79fa-919d-436e-a11d-a4525450a85c\") " pod="openshift-image-registry/node-ca-qf2sz"
Sep 16 16:36:42.705998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705608    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-multus-socket-dir-parent\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.705998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705635    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-sys-fs\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.705998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705660    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5adc7d5-f860-4c8d-992d-963756104953-cni-binary-copy\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.705998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705702    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-host-var-lib-cni-multus\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.705998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705729    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcnhp\" (UniqueName: \"kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp\") pod \"network-check-target-79v7b\" (UID: \"568fe78a-6a27-4de3-a23b-030e47b4d930\") " pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:36:42.705998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705753    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-etc-sysconfig\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.705998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705777    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-etc-sysctl-conf\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.705998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705801    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs\") pod \"network-metrics-daemon-hk8q9\" (UID: \"c04662fe-4176-4a0b-ac5e-89ead5941975\") " pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:36:42.705998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705834    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5adc7d5-f860-4c8d-992d-963756104953-multus-daemon-config\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.705998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705861    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqhf4\" (UniqueName: \"kubernetes.io/projected/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-kube-api-access-vqhf4\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.705998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705898    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.705998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705925    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-multus-cni-dir\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.705998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705953    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-etc-systemd\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.706857 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.705975    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-lib-modules\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.706857 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706002    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2009b752-ad4d-4e71-8a91-3b3b41c44725-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.706857 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706046    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c7eef487-63a5-4d81-bf5a-40e3f673333e-iptables-alerter-script\") pod \"iptables-alerter-m4kpr\" (UID: \"c7eef487-63a5-4d81-bf5a-40e3f673333e\") " pod="openshift-network-operator/iptables-alerter-m4kpr"
Sep 16 16:36:42.706857 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706072    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-systemd-units\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.706857 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706098    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-run-ovn-kubernetes\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.706857 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706126    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-system-cni-dir\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.706857 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706160    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-cnibin\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.706857 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706184    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-host-var-lib-kubelet\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.706857 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706208    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-etc-kubernetes\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.706857 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706233    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-etc-sysctl-d\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.706857 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706265    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-cni-netd\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.706857 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706292    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-device-dir\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.706857 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706318    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-etc-selinux\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.706857 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706347    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jf55\" (UniqueName: \"kubernetes.io/projected/e5adc7d5-f860-4c8d-992d-963756104953-kube-api-access-2jf55\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.706857 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706373    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2009b752-ad4d-4e71-8a91-3b3b41c44725-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.706857 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706409    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2009b752-ad4d-4e71-8a91-3b3b41c44725-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.707668 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706436    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dhp2\" (UniqueName: \"kubernetes.io/projected/c04662fe-4176-4a0b-ac5e-89ead5941975-kube-api-access-7dhp2\") pod \"network-metrics-daemon-hk8q9\" (UID: \"c04662fe-4176-4a0b-ac5e-89ead5941975\") " pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:36:42.707668 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706462    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-etc-openvswitch\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.707668 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706486    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-run-openvswitch\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.707668 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706513    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-slash\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.707668 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706539    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-run-netns\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.707668 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706564    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-run-systemd\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.707668 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706590    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02df9d68-24e0-43d9-8774-273f8b55c906-tmp-dir\") pod \"node-resolver-knvfz\" (UID: \"02df9d68-24e0-43d9-8774-273f8b55c906\") " pod="openshift-dns/node-resolver-knvfz"
Sep 16 16:36:42.707668 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706617    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2009b752-ad4d-4e71-8a91-3b3b41c44725-cnibin\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.707668 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706644    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7eef487-63a5-4d81-bf5a-40e3f673333e-host-slash\") pod \"iptables-alerter-m4kpr\" (UID: \"c7eef487-63a5-4d81-bf5a-40e3f673333e\") " pod="openshift-network-operator/iptables-alerter-m4kpr"
Sep 16 16:36:42.707668 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706672    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-kubelet-config\") pod \"global-pull-secret-syncer-kmw95\" (UID: \"cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792\") " pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:42.707668 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706697    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-var-lib-kubelet\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.707668 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706723    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-host\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.707668 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706747    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/701e8afc-27ea-41e2-8adb-4321da907d00-tmp\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.707668 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706776    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-cni-bin\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.707668 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706804    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-ovnkube-config\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.707668 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706830    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-socket-dir\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.707668 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706861    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-host-run-k8s-cni-cncf-io\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.708528 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706895    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2009b752-ad4d-4e71-8a91-3b3b41c44725-cni-binary-copy\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.708528 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706925    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-log-socket\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.708528 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706951    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b7cc8be-e4bc-477e-8166-f539c64992a4-agent-certs\") pod \"konnectivity-agent-tdf5s\" (UID: \"0b7cc8be-e4bc-477e-8166-f539c64992a4\") " pod="kube-system/konnectivity-agent-tdf5s"
Sep 16 16:36:42.708528 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.706979    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-env-overrides\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.708528 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.707005    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-registration-dir\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.708528 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.707047    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b7cc8be-e4bc-477e-8166-f539c64992a4-konnectivity-ca\") pod \"konnectivity-agent-tdf5s\" (UID: \"0b7cc8be-e4bc-477e-8166-f539c64992a4\") " pod="kube-system/konnectivity-agent-tdf5s"
Sep 16 16:36:42.708528 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.707075    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-multus-conf-dir\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.708528 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.707107    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-etc-modprobe-d\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.708528 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.707132    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2009b752-ad4d-4e71-8a91-3b3b41c44725-system-cni-dir\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.708528 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.707161    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-dbus\") pod \"global-pull-secret-syncer-kmw95\" (UID: \"cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792\") " pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:42.708528 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.707186    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/321e79fa-919d-436e-a11d-a4525450a85c-serviceca\") pod \"node-ca-qf2sz\" (UID: \"321e79fa-919d-436e-a11d-a4525450a85c\") " pod="openshift-image-registry/node-ca-qf2sz"
Sep 16 16:36:42.708528 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.707213    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfkt6\" (UniqueName: \"kubernetes.io/projected/321e79fa-919d-436e-a11d-a4525450a85c-kube-api-access-gfkt6\") pod \"node-ca-qf2sz\" (UID: \"321e79fa-919d-436e-a11d-a4525450a85c\") " pod="openshift-image-registry/node-ca-qf2sz"
Sep 16 16:36:42.708528 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.707246    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-hostroot\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.708528 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.707271    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-run\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.708528 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.707298    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfsjg\" (UniqueName: \"kubernetes.io/projected/701e8afc-27ea-41e2-8adb-4321da907d00-kube-api-access-xfsjg\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.708528 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.707327    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-kubelet\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.708528 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.707354    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-ovnkube-script-lib\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.709435 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.707381    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-host-run-multus-certs\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.709435 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.707407    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/701e8afc-27ea-41e2-8adb-4321da907d00-etc-tuned\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.709435 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.707755    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-etc-selinux\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.709435 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.707825    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-run-ovn\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.709435 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708180    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-var-lib-openvswitch\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.709435 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708188    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-socket-dir\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.709435 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708248    2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 16:36:42.709435 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708329    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.709435 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708350    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-sys-fs\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.709435 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708351    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-kubelet-config\") pod \"global-pull-secret-syncer-kmw95\" (UID: \"cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792\") " pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:42.709435 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708370    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-node-log\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.709435 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708405    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7eef487-63a5-4d81-bf5a-40e3f673333e-host-slash\") pod \"iptables-alerter-m4kpr\" (UID: \"c7eef487-63a5-4d81-bf5a-40e3f673333e\") " pod="openshift-network-operator/iptables-alerter-m4kpr"
Sep 16 16:36:42.709435 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:42.708424    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 16:36:42.709435 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708448    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.709435 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:42.708494    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret podName:cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792 nodeName:}" failed. No retries permitted until 2026-09-16 16:36:43.208470547 +0000 UTC m=+2.970432000 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret") pod "global-pull-secret-syncer-kmw95" (UID: "cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792") : object "kube-system"/"original-pull-secret" not registered
Sep 16 16:36:42.709435 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708500    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-cni-bin\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.709435 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708506    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-systemd-units\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.710291 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708512    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-dbus\") pod \"global-pull-secret-syncer-kmw95\" (UID: \"cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792\") " pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:42.710291 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708556    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-kubelet\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.710291 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708580    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-run-ovn-kubernetes\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.710291 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708591    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-slash\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.710291 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708623    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-cni-netd\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.710291 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708629    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-run-openvswitch\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.710291 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708662    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-etc-openvswitch\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.710291 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708715    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-host-run-netns\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.710291 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708746    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-run-systemd\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.710291 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708768    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-device-dir\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.710291 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.708802    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-registration-dir\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.710291 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.709316    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b7cc8be-e4bc-477e-8166-f539c64992a4-konnectivity-ca\") pod \"konnectivity-agent-tdf5s\" (UID: \"0b7cc8be-e4bc-477e-8166-f539c64992a4\") " pod="kube-system/konnectivity-agent-tdf5s"
Sep 16 16:36:42.710291 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.709526    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-log-socket\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.710291 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.709611    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c7eef487-63a5-4d81-bf5a-40e3f673333e-iptables-alerter-script\") pod \"iptables-alerter-m4kpr\" (UID: \"c7eef487-63a5-4d81-bf5a-40e3f673333e\") " pod="openshift-network-operator/iptables-alerter-m4kpr"
Sep 16 16:36:42.710291 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.709878    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-env-overrides\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.710291 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.709916    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-ovnkube-config\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.710291 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.710203    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-ovnkube-script-lib\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.713516 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.713397    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-ovn-node-metrics-cert\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.713624 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.713547    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b7cc8be-e4bc-477e-8166-f539c64992a4-agent-certs\") pod \"konnectivity-agent-tdf5s\" (UID: \"0b7cc8be-e4bc-477e-8166-f539c64992a4\") " pod="kube-system/konnectivity-agent-tdf5s"
Sep 16 16:36:42.724505 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.724482    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqhf4\" (UniqueName: \"kubernetes.io/projected/9280e2b8-f49e-4e07-b1af-a26cc9108ff9-kube-api-access-vqhf4\") pod \"ovnkube-node-n42qf\" (UID: \"9280e2b8-f49e-4e07-b1af-a26cc9108ff9\") " pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.725117 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.725096    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhfbq\" (UniqueName: \"kubernetes.io/projected/c7eef487-63a5-4d81-bf5a-40e3f673333e-kube-api-access-qhfbq\") pod \"iptables-alerter-m4kpr\" (UID: \"c7eef487-63a5-4d81-bf5a-40e3f673333e\") " pod="openshift-network-operator/iptables-alerter-m4kpr"
Sep 16 16:36:42.726386 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.726363    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg758\" (UniqueName: \"kubernetes.io/projected/3ce6095e-bc1c-489f-92b9-7bf8281c2e61-kube-api-access-xg758\") pod \"aws-ebs-csi-driver-node-fx69r\" (UID: \"3ce6095e-bc1c-489f-92b9-7bf8281c2e61\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.808767 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.808721    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2009b752-ad4d-4e71-8a91-3b3b41c44725-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.809922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.808773    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2009b752-ad4d-4e71-8a91-3b3b41c44725-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.809922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.808805    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dhp2\" (UniqueName: \"kubernetes.io/projected/c04662fe-4176-4a0b-ac5e-89ead5941975-kube-api-access-7dhp2\") pod \"network-metrics-daemon-hk8q9\" (UID: \"c04662fe-4176-4a0b-ac5e-89ead5941975\") " pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:36:42.809922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.808840    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02df9d68-24e0-43d9-8774-273f8b55c906-tmp-dir\") pod \"node-resolver-knvfz\" (UID: \"02df9d68-24e0-43d9-8774-273f8b55c906\") " pod="openshift-dns/node-resolver-knvfz"
Sep 16 16:36:42.809922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.808867    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2009b752-ad4d-4e71-8a91-3b3b41c44725-cnibin\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.809922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.808893    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-var-lib-kubelet\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.809922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.808994    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-var-lib-kubelet\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.809922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809051    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2009b752-ad4d-4e71-8a91-3b3b41c44725-cnibin\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.809922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809160    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2009b752-ad4d-4e71-8a91-3b3b41c44725-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.809922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809214    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-host\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.809922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809244    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/701e8afc-27ea-41e2-8adb-4321da907d00-tmp\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.809922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809274    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-host-run-k8s-cni-cncf-io\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.809922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809291    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02df9d68-24e0-43d9-8774-273f8b55c906-tmp-dir\") pod \"node-resolver-knvfz\" (UID: \"02df9d68-24e0-43d9-8774-273f8b55c906\") " pod="openshift-dns/node-resolver-knvfz"
Sep 16 16:36:42.809922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809321    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-host\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.809922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809300    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2009b752-ad4d-4e71-8a91-3b3b41c44725-cni-binary-copy\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.809922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809369    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-host-run-k8s-cni-cncf-io\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.809922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809376    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-multus-conf-dir\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.809922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809406    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-etc-modprobe-d\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.810957 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809432    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2009b752-ad4d-4e71-8a91-3b3b41c44725-system-cni-dir\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.810957 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809459    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/321e79fa-919d-436e-a11d-a4525450a85c-serviceca\") pod \"node-ca-qf2sz\" (UID: \"321e79fa-919d-436e-a11d-a4525450a85c\") " pod="openshift-image-registry/node-ca-qf2sz"
Sep 16 16:36:42.810957 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809483    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfkt6\" (UniqueName: \"kubernetes.io/projected/321e79fa-919d-436e-a11d-a4525450a85c-kube-api-access-gfkt6\") pod \"node-ca-qf2sz\" (UID: \"321e79fa-919d-436e-a11d-a4525450a85c\") " pod="openshift-image-registry/node-ca-qf2sz"
Sep 16 16:36:42.810957 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809508    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-hostroot\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.810957 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809536    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-run\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.810957 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809564    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfsjg\" (UniqueName: \"kubernetes.io/projected/701e8afc-27ea-41e2-8adb-4321da907d00-kube-api-access-xfsjg\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.810957 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809573    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2009b752-ad4d-4e71-8a91-3b3b41c44725-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.810957 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809591    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-host-run-multus-certs\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.810957 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809630    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/701e8afc-27ea-41e2-8adb-4321da907d00-etc-tuned\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.810957 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809641    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-host-run-multus-certs\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.810957 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809656    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-sys\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.810957 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809680    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02df9d68-24e0-43d9-8774-273f8b55c906-hosts-file\") pod \"node-resolver-knvfz\" (UID: \"02df9d68-24e0-43d9-8774-273f8b55c906\") " pod="openshift-dns/node-resolver-knvfz"
Sep 16 16:36:42.810957 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809687    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-multus-conf-dir\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.810957 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809760    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-sys\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.810957 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809756    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2009b752-ad4d-4e71-8a91-3b3b41c44725-system-cni-dir\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.810957 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809826    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-hostroot\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.810957 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809832    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-run\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.810957 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809860    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-etc-modprobe-d\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.811974 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809873    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2009b752-ad4d-4e71-8a91-3b3b41c44725-cni-binary-copy\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.811974 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809949    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02df9d68-24e0-43d9-8774-273f8b55c906-hosts-file\") pod \"node-resolver-knvfz\" (UID: \"02df9d68-24e0-43d9-8774-273f8b55c906\") " pod="openshift-dns/node-resolver-knvfz"
Sep 16 16:36:42.811974 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.809982    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76rb7\" (UniqueName: \"kubernetes.io/projected/2009b752-ad4d-4e71-8a91-3b3b41c44725-kube-api-access-76rb7\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.811974 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810012    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-host-run-netns\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.811974 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810050    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-host-var-lib-cni-bin\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.811974 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810091    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-host-var-lib-cni-bin\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.811974 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810127    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-host-run-netns\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.811974 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810157    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/321e79fa-919d-436e-a11d-a4525450a85c-serviceca\") pod \"node-ca-qf2sz\" (UID: \"321e79fa-919d-436e-a11d-a4525450a85c\") " pod="openshift-image-registry/node-ca-qf2sz"
Sep 16 16:36:42.811974 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810204    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-etc-kubernetes\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.811974 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810236    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx6bg\" (UniqueName: \"kubernetes.io/projected/02df9d68-24e0-43d9-8774-273f8b55c906-kube-api-access-kx6bg\") pod \"node-resolver-knvfz\" (UID: \"02df9d68-24e0-43d9-8774-273f8b55c906\") " pod="openshift-dns/node-resolver-knvfz"
Sep 16 16:36:42.811974 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810243    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-etc-kubernetes\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.811974 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810285    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/321e79fa-919d-436e-a11d-a4525450a85c-host\") pod \"node-ca-qf2sz\" (UID: \"321e79fa-919d-436e-a11d-a4525450a85c\") " pod="openshift-image-registry/node-ca-qf2sz"
Sep 16 16:36:42.811974 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810311    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-multus-socket-dir-parent\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.811974 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810337    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5adc7d5-f860-4c8d-992d-963756104953-cni-binary-copy\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.811974 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810356    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/321e79fa-919d-436e-a11d-a4525450a85c-host\") pod \"node-ca-qf2sz\" (UID: \"321e79fa-919d-436e-a11d-a4525450a85c\") " pod="openshift-image-registry/node-ca-qf2sz"
Sep 16 16:36:42.811974 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810363    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-host-var-lib-cni-multus\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.811974 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810400    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-host-var-lib-cni-multus\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.811974 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810406    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcnhp\" (UniqueName: \"kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp\") pod \"network-check-target-79v7b\" (UID: \"568fe78a-6a27-4de3-a23b-030e47b4d930\") " pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:36:42.812872 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810437    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-etc-sysconfig\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.812872 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810449    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-multus-socket-dir-parent\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.812872 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810464    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-etc-sysctl-conf\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.812872 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810523    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-etc-sysconfig\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.812872 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810553    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs\") pod \"network-metrics-daemon-hk8q9\" (UID: \"c04662fe-4176-4a0b-ac5e-89ead5941975\") " pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:36:42.812872 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810581    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5adc7d5-f860-4c8d-992d-963756104953-multus-daemon-config\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.812872 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810606    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-etc-sysctl-conf\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.812872 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810611    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-multus-cni-dir\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.812872 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810636    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-etc-systemd\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.812872 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810663    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-lib-modules\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.812872 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:42.810678    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 16:36:42.812872 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810689    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2009b752-ad4d-4e71-8a91-3b3b41c44725-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.812872 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810699    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-multus-cni-dir\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.812872 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810720    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-system-cni-dir\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.812872 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:42.810752    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs podName:c04662fe-4176-4a0b-ac5e-89ead5941975 nodeName:}" failed. No retries permitted until 2026-09-16 16:36:43.310733725 +0000 UTC m=+3.072695168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs") pod "network-metrics-daemon-hk8q9" (UID: "c04662fe-4176-4a0b-ac5e-89ead5941975") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 16:36:42.812872 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810776    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-cnibin\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.812872 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810802    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-lib-modules\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.813748 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810807    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-host-var-lib-kubelet\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.813748 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810818    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-system-cni-dir\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.813748 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810833    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-etc-kubernetes\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.813748 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810852    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-etc-systemd\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.813748 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810860    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-etc-sysctl-d\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.813748 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810892    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jf55\" (UniqueName: \"kubernetes.io/projected/e5adc7d5-f860-4c8d-992d-963756104953-kube-api-access-2jf55\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.813748 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810903    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-cnibin\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.813748 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810943    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e5adc7d5-f860-4c8d-992d-963756104953-cni-binary-copy\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.813748 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.810966    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5adc7d5-f860-4c8d-992d-963756104953-host-var-lib-kubelet\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.813748 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.811091    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-etc-sysctl-d\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.813748 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.811116    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e5adc7d5-f860-4c8d-992d-963756104953-multus-daemon-config\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.813748 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.811138    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/701e8afc-27ea-41e2-8adb-4321da907d00-etc-kubernetes\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.813748 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.811320    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2009b752-ad4d-4e71-8a91-3b3b41c44725-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.814418 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.813874    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/701e8afc-27ea-41e2-8adb-4321da907d00-tmp\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.815210 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.815164    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/701e8afc-27ea-41e2-8adb-4321da907d00-etc-tuned\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.832387 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.832292    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dhp2\" (UniqueName: \"kubernetes.io/projected/c04662fe-4176-4a0b-ac5e-89ead5941975-kube-api-access-7dhp2\") pod \"network-metrics-daemon-hk8q9\" (UID: \"c04662fe-4176-4a0b-ac5e-89ead5941975\") " pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:36:42.832644 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.832620    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfkt6\" (UniqueName: \"kubernetes.io/projected/321e79fa-919d-436e-a11d-a4525450a85c-kube-api-access-gfkt6\") pod \"node-ca-qf2sz\" (UID: \"321e79fa-919d-436e-a11d-a4525450a85c\") " pod="openshift-image-registry/node-ca-qf2sz"
Sep 16 16:36:42.834887 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:42.834575    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 16:36:42.834887 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:42.834605    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 16:36:42.834887 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:42.834620    2556 projected.go:196] Error preparing data for projected volume kube-api-access-gcnhp for pod openshift-network-diagnostics/network-check-target-79v7b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 16:36:42.834887 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:42.834705    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp podName:568fe78a-6a27-4de3-a23b-030e47b4d930 nodeName:}" failed. No retries permitted until 2026-09-16 16:36:43.334685126 +0000 UTC m=+3.096646577 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gcnhp" (UniqueName: "kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp") pod "network-check-target-79v7b" (UID: "568fe78a-6a27-4de3-a23b-030e47b4d930") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 16:36:42.838343 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.838316    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jf55\" (UniqueName: \"kubernetes.io/projected/e5adc7d5-f860-4c8d-992d-963756104953-kube-api-access-2jf55\") pod \"multus-lktnv\" (UID: \"e5adc7d5-f860-4c8d-992d-963756104953\") " pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.838791 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.838756    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx6bg\" (UniqueName: \"kubernetes.io/projected/02df9d68-24e0-43d9-8774-273f8b55c906-kube-api-access-kx6bg\") pod \"node-resolver-knvfz\" (UID: \"02df9d68-24e0-43d9-8774-273f8b55c906\") " pod="openshift-dns/node-resolver-knvfz"
Sep 16 16:36:42.839269 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.839210    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76rb7\" (UniqueName: \"kubernetes.io/projected/2009b752-ad4d-4e71-8a91-3b3b41c44725-kube-api-access-76rb7\") pod \"multus-additional-cni-plugins-g7ksk\" (UID: \"2009b752-ad4d-4e71-8a91-3b3b41c44725\") " pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.840591 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.840564    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfsjg\" (UniqueName: \"kubernetes.io/projected/701e8afc-27ea-41e2-8adb-4321da907d00-kube-api-access-xfsjg\") pod \"tuned-mlt5x\" (UID: \"701e8afc-27ea-41e2-8adb-4321da907d00\") " pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.849222 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.849174    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal" event={"ID":"4181901193613f3e77dd46929737ec7c","Type":"ContainerStarted","Data":"703f02351675f831ece827bf5db0f82fedff3f63e86adebe6d05a3d8ca59dcde"}
Sep 16 16:36:42.853602 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.853566    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal" event={"ID":"242eb3e6d5cc43fa8cbc80becad32de3","Type":"ContainerStarted","Data":"58f148b5c36c951e642d9cff0a40ddb2b24ef3d16484a72da81fbbc0078420d6"}
Sep 16 16:36:42.898603 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.898578    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r"
Sep 16 16:36:42.908637 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.908454    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tdf5s"
Sep 16 16:36:42.910215 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:42.910187    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce6095e_bc1c_489f_92b9_7bf8281c2e61.slice/crio-e93f49d24303cd41aa663fa3b9784dd468c7b6f8f7b8a7e0a889bc7011fc3d86 WatchSource:0}: Error finding container e93f49d24303cd41aa663fa3b9784dd468c7b6f8f7b8a7e0a889bc7011fc3d86: Status 404 returned error can't find the container with id e93f49d24303cd41aa663fa3b9784dd468c7b6f8f7b8a7e0a889bc7011fc3d86
Sep 16 16:36:42.919920 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:42.919814    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7cc8be_e4bc_477e_8166_f539c64992a4.slice/crio-b9c0256ca72d8d2f4bda4e377f09f2acebbf6c6ec298248db666944791143aaa WatchSource:0}: Error finding container b9c0256ca72d8d2f4bda4e377f09f2acebbf6c6ec298248db666944791143aaa: Status 404 returned error can't find the container with id b9c0256ca72d8d2f4bda4e377f09f2acebbf6c6ec298248db666944791143aaa
Sep 16 16:36:42.920730 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.920710    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m4kpr"
Sep 16 16:36:42.930225 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.930200    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:36:42.931108 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:42.931073    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7eef487_63a5_4d81_bf5a_40e3f673333e.slice/crio-910554d263e5830aff8652ef202a0d807510e2c54dd8df568ae89b5b2a8b4f0b WatchSource:0}: Error finding container 910554d263e5830aff8652ef202a0d807510e2c54dd8df568ae89b5b2a8b4f0b: Status 404 returned error can't find the container with id 910554d263e5830aff8652ef202a0d807510e2c54dd8df568ae89b5b2a8b4f0b
Sep 16 16:36:42.937701 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.937582    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qf2sz"
Sep 16 16:36:42.941777 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:42.941744    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9280e2b8_f49e_4e07_b1af_a26cc9108ff9.slice/crio-cdbdd908e2259d9e87e32f581ad18e5ea6ba29327af7fdf91749f8b0ba57c34a WatchSource:0}: Error finding container cdbdd908e2259d9e87e32f581ad18e5ea6ba29327af7fdf91749f8b0ba57c34a: Status 404 returned error can't find the container with id cdbdd908e2259d9e87e32f581ad18e5ea6ba29327af7fdf91749f8b0ba57c34a
Sep 16 16:36:42.946576 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.946554    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-knvfz"
Sep 16 16:36:42.948339 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:42.948311    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321e79fa_919d_436e_a11d_a4525450a85c.slice/crio-85b5f5568fef4275c58af5aad264ff6103bc5a7104fec285899810b77152e4a0 WatchSource:0}: Error finding container 85b5f5568fef4275c58af5aad264ff6103bc5a7104fec285899810b77152e4a0: Status 404 returned error can't find the container with id 85b5f5568fef4275c58af5aad264ff6103bc5a7104fec285899810b77152e4a0
Sep 16 16:36:42.957232 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.957200    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lktnv"
Sep 16 16:36:42.958334 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:42.958307    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02df9d68_24e0_43d9_8774_273f8b55c906.slice/crio-24bfbf03b6db328b2bb6714ac6f94b693b756d770c45a7953e0737126f0f8363 WatchSource:0}: Error finding container 24bfbf03b6db328b2bb6714ac6f94b693b756d770c45a7953e0737126f0f8363: Status 404 returned error can't find the container with id 24bfbf03b6db328b2bb6714ac6f94b693b756d770c45a7953e0737126f0f8363
Sep 16 16:36:42.967536 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.967513    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mlt5x"
Sep 16 16:36:42.968217 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:42.968192    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5adc7d5_f860_4c8d_992d_963756104953.slice/crio-a55d3908ce4374512e0a3ddcd4af918d54ef538cb6f06edfed827e87f5a1bd6b WatchSource:0}: Error finding container a55d3908ce4374512e0a3ddcd4af918d54ef538cb6f06edfed827e87f5a1bd6b: Status 404 returned error can't find the container with id a55d3908ce4374512e0a3ddcd4af918d54ef538cb6f06edfed827e87f5a1bd6b
Sep 16 16:36:42.975112 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:42.974938    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g7ksk"
Sep 16 16:36:42.977906 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:42.977878    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701e8afc_27ea_41e2_8adb_4321da907d00.slice/crio-11def52733578060adee2c70689ad7f31ca7c4fdf6c998375bf2650f4b245391 WatchSource:0}: Error finding container 11def52733578060adee2c70689ad7f31ca7c4fdf6c998375bf2650f4b245391: Status 404 returned error can't find the container with id 11def52733578060adee2c70689ad7f31ca7c4fdf6c998375bf2650f4b245391
Sep 16 16:36:42.986923 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:36:42.986651    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2009b752_ad4d_4e71_8a91_3b3b41c44725.slice/crio-98c767c70ed0c16a1a0db967ecaccc38f0ded268748f5511d0bf17afbf09484b WatchSource:0}: Error finding container 98c767c70ed0c16a1a0db967ecaccc38f0ded268748f5511d0bf17afbf09484b: Status 404 returned error can't find the container with id 98c767c70ed0c16a1a0db967ecaccc38f0ded268748f5511d0bf17afbf09484b
Sep 16 16:36:43.214260 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:43.214160    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret\") pod \"global-pull-secret-syncer-kmw95\" (UID: \"cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792\") " pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:43.214423 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:43.214382    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 16:36:43.214488 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:43.214479    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret podName:cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792 nodeName:}" failed. No retries permitted until 2026-09-16 16:36:44.214455209 +0000 UTC m=+3.976416657 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret") pod "global-pull-secret-syncer-kmw95" (UID: "cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792") : object "kube-system"/"original-pull-secret" not registered
Sep 16 16:36:43.315561 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:43.315522    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs\") pod \"network-metrics-daemon-hk8q9\" (UID: \"c04662fe-4176-4a0b-ac5e-89ead5941975\") " pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:36:43.315737 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:43.315678    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 16:36:43.315792 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:43.315752    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs podName:c04662fe-4176-4a0b-ac5e-89ead5941975 nodeName:}" failed. No retries permitted until 2026-09-16 16:36:44.315731744 +0000 UTC m=+4.077693178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs") pod "network-metrics-daemon-hk8q9" (UID: "c04662fe-4176-4a0b-ac5e-89ead5941975") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 16:36:43.318492 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:43.318468    2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 16:36:43.416555 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:43.416507    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcnhp\" (UniqueName: \"kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp\") pod \"network-check-target-79v7b\" (UID: \"568fe78a-6a27-4de3-a23b-030e47b4d930\") " pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:36:43.416734 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:43.416712    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 16:36:43.416801 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:43.416738    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 16:36:43.416801 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:43.416752    2556 projected.go:196] Error preparing data for projected volume kube-api-access-gcnhp for pod openshift-network-diagnostics/network-check-target-79v7b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 16:36:43.416891 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:43.416829    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp podName:568fe78a-6a27-4de3-a23b-030e47b4d930 nodeName:}" failed. No retries permitted until 2026-09-16 16:36:44.416806796 +0000 UTC m=+4.178768244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gcnhp" (UniqueName: "kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp") pod "network-check-target-79v7b" (UID: "568fe78a-6a27-4de3-a23b-030e47b4d930") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 16:36:43.656249 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:43.656113    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 16:31:43 +0000 UTC" deadline="2028-06-07 13:37:14.985565274 +0000 UTC"
Sep 16 16:36:43.656249 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:43.656192    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15117h0m31.329378038s"
Sep 16 16:36:43.843014 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:43.842975    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:36:43.843187 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:43.843146    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk8q9" podUID="c04662fe-4176-4a0b-ac5e-89ead5941975"
Sep 16 16:36:43.857449 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:43.857413    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7ksk" event={"ID":"2009b752-ad4d-4e71-8a91-3b3b41c44725","Type":"ContainerStarted","Data":"98c767c70ed0c16a1a0db967ecaccc38f0ded268748f5511d0bf17afbf09484b"}
Sep 16 16:36:43.859268 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:43.859231    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mlt5x" event={"ID":"701e8afc-27ea-41e2-8adb-4321da907d00","Type":"ContainerStarted","Data":"11def52733578060adee2c70689ad7f31ca7c4fdf6c998375bf2650f4b245391"}
Sep 16 16:36:43.860889 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:43.860862    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-knvfz" event={"ID":"02df9d68-24e0-43d9-8774-273f8b55c906","Type":"ContainerStarted","Data":"24bfbf03b6db328b2bb6714ac6f94b693b756d770c45a7953e0737126f0f8363"}
Sep 16 16:36:43.862566 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:43.862540    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n42qf" event={"ID":"9280e2b8-f49e-4e07-b1af-a26cc9108ff9","Type":"ContainerStarted","Data":"cdbdd908e2259d9e87e32f581ad18e5ea6ba29327af7fdf91749f8b0ba57c34a"}
Sep 16 16:36:43.864310 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:43.864271    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tdf5s" event={"ID":"0b7cc8be-e4bc-477e-8166-f539c64992a4","Type":"ContainerStarted","Data":"b9c0256ca72d8d2f4bda4e377f09f2acebbf6c6ec298248db666944791143aaa"}
Sep 16 16:36:43.865704 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:43.865648    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r" event={"ID":"3ce6095e-bc1c-489f-92b9-7bf8281c2e61","Type":"ContainerStarted","Data":"e93f49d24303cd41aa663fa3b9784dd468c7b6f8f7b8a7e0a889bc7011fc3d86"}
Sep 16 16:36:43.867165 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:43.867142    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lktnv" event={"ID":"e5adc7d5-f860-4c8d-992d-963756104953","Type":"ContainerStarted","Data":"a55d3908ce4374512e0a3ddcd4af918d54ef538cb6f06edfed827e87f5a1bd6b"}
Sep 16 16:36:43.869044 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:43.868989    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qf2sz" event={"ID":"321e79fa-919d-436e-a11d-a4525450a85c","Type":"ContainerStarted","Data":"85b5f5568fef4275c58af5aad264ff6103bc5a7104fec285899810b77152e4a0"}
Sep 16 16:36:43.870270 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:43.870245    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m4kpr" event={"ID":"c7eef487-63a5-4d81-bf5a-40e3f673333e","Type":"ContainerStarted","Data":"910554d263e5830aff8652ef202a0d807510e2c54dd8df568ae89b5b2a8b4f0b"}
Sep 16 16:36:43.977750 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:43.977505    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 16:36:44.222570 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:44.222520    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret\") pod \"global-pull-secret-syncer-kmw95\" (UID: \"cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792\") " pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:44.222739 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:44.222675    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 16:36:44.222814 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:44.222742    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret podName:cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792 nodeName:}" failed. No retries permitted until 2026-09-16 16:36:46.222722838 +0000 UTC m=+5.984684279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret") pod "global-pull-secret-syncer-kmw95" (UID: "cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792") : object "kube-system"/"original-pull-secret" not registered
Sep 16 16:36:44.323031 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:44.323003    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs\") pod \"network-metrics-daemon-hk8q9\" (UID: \"c04662fe-4176-4a0b-ac5e-89ead5941975\") " pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:36:44.323160 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:44.323136    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 16:36:44.323223 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:44.323181    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs podName:c04662fe-4176-4a0b-ac5e-89ead5941975 nodeName:}" failed. No retries permitted until 2026-09-16 16:36:46.323167839 +0000 UTC m=+6.085129266 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs") pod "network-metrics-daemon-hk8q9" (UID: "c04662fe-4176-4a0b-ac5e-89ead5941975") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 16:36:44.426225 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:44.424183    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcnhp\" (UniqueName: \"kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp\") pod \"network-check-target-79v7b\" (UID: \"568fe78a-6a27-4de3-a23b-030e47b4d930\") " pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:36:44.426225 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:44.424381    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 16:36:44.426225 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:44.424403    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 16:36:44.426225 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:44.424416    2556 projected.go:196] Error preparing data for projected volume kube-api-access-gcnhp for pod openshift-network-diagnostics/network-check-target-79v7b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 16:36:44.426225 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:44.424476    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp podName:568fe78a-6a27-4de3-a23b-030e47b4d930 nodeName:}" failed. No retries permitted until 2026-09-16 16:36:46.424457271 +0000 UTC m=+6.186418719 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gcnhp" (UniqueName: "kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp") pod "network-check-target-79v7b" (UID: "568fe78a-6a27-4de3-a23b-030e47b4d930") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 16:36:44.656909 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:44.656811    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 16:31:43 +0000 UTC" deadline="2028-04-22 11:55:27.63428912 +0000 UTC"
Sep 16 16:36:44.656909 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:44.656856    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14011h18m42.977438042s"
Sep 16 16:36:44.845436 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:44.844906    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:36:44.845436 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:44.845066    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79v7b" podUID="568fe78a-6a27-4de3-a23b-030e47b4d930"
Sep 16 16:36:44.846350 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:44.846039    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:44.846350 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:44.846155    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmw95" podUID="cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792"
Sep 16 16:36:44.896359 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:44.896154    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal" event={"ID":"4181901193613f3e77dd46929737ec7c","Type":"ContainerStarted","Data":"b8874b2d132b1af28e88bebb7256147e39230523072c852714908208bc18baee"}
Sep 16 16:36:44.916676 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:44.916246    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-120.ec2.internal" podStartSLOduration=2.916223221 podStartE2EDuration="2.916223221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 16:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:36:44.91515244 +0000 UTC m=+4.677113892" watchObservedRunningTime="2026-09-16 16:36:44.916223221 +0000 UTC m=+4.678184675"
Sep 16 16:36:45.842099 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:45.842059    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:36:45.842689 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:45.842224    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk8q9" podUID="c04662fe-4176-4a0b-ac5e-89ead5941975"
Sep 16 16:36:45.908213 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:45.908082    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal" event={"ID":"242eb3e6d5cc43fa8cbc80becad32de3","Type":"ContainerStarted","Data":"0625fcd2d35cc74d018c391e6d9a57d77fd3c7c54ff8e7e7f035b01071d13af9"}
Sep 16 16:36:46.243412 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:46.243316    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret\") pod \"global-pull-secret-syncer-kmw95\" (UID: \"cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792\") " pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:46.243633 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:46.243495    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 16:36:46.243633 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:46.243559    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret podName:cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792 nodeName:}" failed. No retries permitted until 2026-09-16 16:36:50.243539612 +0000 UTC m=+10.005501053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret") pod "global-pull-secret-syncer-kmw95" (UID: "cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792") : object "kube-system"/"original-pull-secret" not registered
Sep 16 16:36:46.344340 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:46.344282    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs\") pod \"network-metrics-daemon-hk8q9\" (UID: \"c04662fe-4176-4a0b-ac5e-89ead5941975\") " pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:36:46.344603 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:46.344484    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 16:36:46.344603 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:46.344551    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs podName:c04662fe-4176-4a0b-ac5e-89ead5941975 nodeName:}" failed. No retries permitted until 2026-09-16 16:36:50.344531181 +0000 UTC m=+10.106492626 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs") pod "network-metrics-daemon-hk8q9" (UID: "c04662fe-4176-4a0b-ac5e-89ead5941975") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 16:36:46.445470 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:46.445430    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcnhp\" (UniqueName: \"kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp\") pod \"network-check-target-79v7b\" (UID: \"568fe78a-6a27-4de3-a23b-030e47b4d930\") " pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:36:46.445683 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:46.445643    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 16:36:46.445683 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:46.445669    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 16:36:46.445683 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:46.445683    2556 projected.go:196] Error preparing data for projected volume kube-api-access-gcnhp for pod openshift-network-diagnostics/network-check-target-79v7b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 16:36:46.445853 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:46.445748    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp podName:568fe78a-6a27-4de3-a23b-030e47b4d930 nodeName:}" failed. No retries permitted until 2026-09-16 16:36:50.445725824 +0000 UTC m=+10.207687257 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gcnhp" (UniqueName: "kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp") pod "network-check-target-79v7b" (UID: "568fe78a-6a27-4de3-a23b-030e47b4d930") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 16:36:46.843134 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:46.843100    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:36:46.843622 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:46.843250    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79v7b" podUID="568fe78a-6a27-4de3-a23b-030e47b4d930"
Sep 16 16:36:46.843622 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:46.843341    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:46.843622 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:46.843468    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmw95" podUID="cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792"
Sep 16 16:36:47.843068 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:47.843032    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:36:47.843277 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:47.843183    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk8q9" podUID="c04662fe-4176-4a0b-ac5e-89ead5941975"
Sep 16 16:36:47.914754 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:47.914710    2556 generic.go:356] "Generic (PLEG): container finished" podID="242eb3e6d5cc43fa8cbc80becad32de3" containerID="0625fcd2d35cc74d018c391e6d9a57d77fd3c7c54ff8e7e7f035b01071d13af9" exitCode=0
Sep 16 16:36:47.914929 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:47.914765    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal" event={"ID":"242eb3e6d5cc43fa8cbc80becad32de3","Type":"ContainerDied","Data":"0625fcd2d35cc74d018c391e6d9a57d77fd3c7c54ff8e7e7f035b01071d13af9"}
Sep 16 16:36:48.843797 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:48.843101    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:48.843797 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:48.843243    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmw95" podUID="cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792"
Sep 16 16:36:48.843797 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:48.843653    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:36:48.843797 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:48.843750    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79v7b" podUID="568fe78a-6a27-4de3-a23b-030e47b4d930"
Sep 16 16:36:49.842712 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:49.842686    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:36:49.842843 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:49.842809    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk8q9" podUID="c04662fe-4176-4a0b-ac5e-89ead5941975"
Sep 16 16:36:50.277117 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:50.276464    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret\") pod \"global-pull-secret-syncer-kmw95\" (UID: \"cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792\") " pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:50.277117 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:50.276651    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 16:36:50.277117 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:50.276714    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret podName:cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792 nodeName:}" failed. No retries permitted until 2026-09-16 16:36:58.276695827 +0000 UTC m=+18.038657266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret") pod "global-pull-secret-syncer-kmw95" (UID: "cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792") : object "kube-system"/"original-pull-secret" not registered
Sep 16 16:36:50.377315 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:50.377280    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs\") pod \"network-metrics-daemon-hk8q9\" (UID: \"c04662fe-4176-4a0b-ac5e-89ead5941975\") " pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:36:50.377474 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:50.377415    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 16:36:50.377530 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:50.377500    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs podName:c04662fe-4176-4a0b-ac5e-89ead5941975 nodeName:}" failed. No retries permitted until 2026-09-16 16:36:58.377479884 +0000 UTC m=+18.139441326 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs") pod "network-metrics-daemon-hk8q9" (UID: "c04662fe-4176-4a0b-ac5e-89ead5941975") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 16:36:50.478314 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:50.478277    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcnhp\" (UniqueName: \"kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp\") pod \"network-check-target-79v7b\" (UID: \"568fe78a-6a27-4de3-a23b-030e47b4d930\") " pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:36:50.478484 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:50.478451    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 16:36:50.478484 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:50.478473    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 16:36:50.478649 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:50.478486    2556 projected.go:196] Error preparing data for projected volume kube-api-access-gcnhp for pod openshift-network-diagnostics/network-check-target-79v7b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 16:36:50.478649 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:50.478543    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp podName:568fe78a-6a27-4de3-a23b-030e47b4d930 nodeName:}" failed. No retries permitted until 2026-09-16 16:36:58.478525019 +0000 UTC m=+18.240486465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gcnhp" (UniqueName: "kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp") pod "network-check-target-79v7b" (UID: "568fe78a-6a27-4de3-a23b-030e47b4d930") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 16:36:50.844116 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:50.843651    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:36:50.844116 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:50.843756    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79v7b" podUID="568fe78a-6a27-4de3-a23b-030e47b4d930"
Sep 16 16:36:50.844116 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:50.843813    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:50.844116 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:50.843922    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmw95" podUID="cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792"
Sep 16 16:36:51.842973 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:51.842927    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:36:51.843423 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:51.843091    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk8q9" podUID="c04662fe-4176-4a0b-ac5e-89ead5941975"
Sep 16 16:36:52.843043 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:52.842600    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:52.843043 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:52.842653    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:36:52.843043 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:52.842726    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmw95" podUID="cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792"
Sep 16 16:36:52.843043 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:52.842839    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79v7b" podUID="568fe78a-6a27-4de3-a23b-030e47b4d930"
Sep 16 16:36:53.842594 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:53.842563    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:36:53.842743 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:53.842671    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk8q9" podUID="c04662fe-4176-4a0b-ac5e-89ead5941975"
Sep 16 16:36:54.842892 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:54.842694    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:36:54.843320 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:54.842765    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:54.843320 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:54.843021    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79v7b" podUID="568fe78a-6a27-4de3-a23b-030e47b4d930"
Sep 16 16:36:54.843320 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:54.843089    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmw95" podUID="cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792"
Sep 16 16:36:55.842416 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:55.842377    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:36:55.842586 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:55.842503    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk8q9" podUID="c04662fe-4176-4a0b-ac5e-89ead5941975"
Sep 16 16:36:56.842192 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:56.842159    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:56.842600 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:56.842210    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:36:56.842600 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:56.842292    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmw95" podUID="cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792"
Sep 16 16:36:56.842600 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:56.842407    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79v7b" podUID="568fe78a-6a27-4de3-a23b-030e47b4d930"
Sep 16 16:36:57.842486 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:57.842454    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:36:57.842849 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:57.842593    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk8q9" podUID="c04662fe-4176-4a0b-ac5e-89ead5941975"
Sep 16 16:36:58.341217 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:58.341174    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret\") pod \"global-pull-secret-syncer-kmw95\" (UID: \"cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792\") " pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:58.341430 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:58.341323    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 16:36:58.341430 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:58.341388    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret podName:cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792 nodeName:}" failed. No retries permitted until 2026-09-16 16:37:14.341368477 +0000 UTC m=+34.103329901 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret") pod "global-pull-secret-syncer-kmw95" (UID: "cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792") : object "kube-system"/"original-pull-secret" not registered
Sep 16 16:36:58.441761 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:58.441721    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs\") pod \"network-metrics-daemon-hk8q9\" (UID: \"c04662fe-4176-4a0b-ac5e-89ead5941975\") " pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:36:58.441960 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:58.441864    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 16:36:58.441960 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:58.441933    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs podName:c04662fe-4176-4a0b-ac5e-89ead5941975 nodeName:}" failed. No retries permitted until 2026-09-16 16:37:14.441913073 +0000 UTC m=+34.203874595 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs") pod "network-metrics-daemon-hk8q9" (UID: "c04662fe-4176-4a0b-ac5e-89ead5941975") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 16:36:58.542835 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:58.542802    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcnhp\" (UniqueName: \"kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp\") pod \"network-check-target-79v7b\" (UID: \"568fe78a-6a27-4de3-a23b-030e47b4d930\") " pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:36:58.543008 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:58.542972    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 16:36:58.543008 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:58.542988    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 16:36:58.543008 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:58.543000    2556 projected.go:196] Error preparing data for projected volume kube-api-access-gcnhp for pod openshift-network-diagnostics/network-check-target-79v7b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 16:36:58.543140 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:58.543061    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp podName:568fe78a-6a27-4de3-a23b-030e47b4d930 nodeName:}" failed. No retries permitted until 2026-09-16 16:37:14.543046168 +0000 UTC m=+34.305007810 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gcnhp" (UniqueName: "kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp") pod "network-check-target-79v7b" (UID: "568fe78a-6a27-4de3-a23b-030e47b4d930") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 16:36:58.842672 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:58.842647    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:36:58.843131 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:58.842681    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:36:58.843131 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:58.842763    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79v7b" podUID="568fe78a-6a27-4de3-a23b-030e47b4d930"
Sep 16 16:36:58.843131 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:58.842846    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmw95" podUID="cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792"
Sep 16 16:36:59.843163 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:36:59.843125    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:36:59.843632 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:36:59.843284    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk8q9" podUID="c04662fe-4176-4a0b-ac5e-89ead5941975"
Sep 16 16:37:00.843569 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:00.843512    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:37:00.844055 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:00.843627    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79v7b" podUID="568fe78a-6a27-4de3-a23b-030e47b4d930"
Sep 16 16:37:00.844055 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:00.843710    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:37:00.844055 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:00.843819    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmw95" podUID="cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792"
Sep 16 16:37:01.842581 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:01.842424    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:37:01.842682 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:01.842666    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk8q9" podUID="c04662fe-4176-4a0b-ac5e-89ead5941975"
Sep 16 16:37:01.942733 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:01.942704    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal" event={"ID":"242eb3e6d5cc43fa8cbc80becad32de3","Type":"ContainerStarted","Data":"462d4019379617313fdf7148a2ead274add53c2fff9a5f76c9f4fc63f10b8251"}
Sep 16 16:37:01.944193 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:01.944173    2556 generic.go:356] "Generic (PLEG): container finished" podID="2009b752-ad4d-4e71-8a91-3b3b41c44725" containerID="46bfafd95f849d36bbd05628db54f4c14fb035e33b120b2153c213aa83afe432" exitCode=0
Sep 16 16:37:01.944293 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:01.944245    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7ksk" event={"ID":"2009b752-ad4d-4e71-8a91-3b3b41c44725","Type":"ContainerDied","Data":"46bfafd95f849d36bbd05628db54f4c14fb035e33b120b2153c213aa83afe432"}
Sep 16 16:37:01.945807 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:01.945778    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mlt5x" event={"ID":"701e8afc-27ea-41e2-8adb-4321da907d00","Type":"ContainerStarted","Data":"9f116bf9889928b39099e19a8a50aadde4f8904a3d75af0a5ff9adf26174c86e"}
Sep 16 16:37:01.947413 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:01.947394    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-knvfz" event={"ID":"02df9d68-24e0-43d9-8774-273f8b55c906","Type":"ContainerStarted","Data":"7a50bdc28f132c13640ecfaa8048c5c6daae04b64a7c8d2a1b4caa17a51a51f0"}
Sep 16 16:37:01.949967 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:01.949933    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n42qf" event={"ID":"9280e2b8-f49e-4e07-b1af-a26cc9108ff9","Type":"ContainerStarted","Data":"f3426746fee3500388541c6b5772a7579c19eee0e914e118121abafa2da8a55e"}
Sep 16 16:37:01.950055 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:01.949998    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n42qf" event={"ID":"9280e2b8-f49e-4e07-b1af-a26cc9108ff9","Type":"ContainerStarted","Data":"b004ebc9ad3e15eb5871c8ee758376a7e4e768aabb0201f7e366356905208df7"}
Sep 16 16:37:01.950055 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:01.950033    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n42qf" event={"ID":"9280e2b8-f49e-4e07-b1af-a26cc9108ff9","Type":"ContainerStarted","Data":"1f53a1ac26422ce66511bddbb44d1ec8adcbc0f0bb9a26192d6bdbcf8266f5cd"}
Sep 16 16:37:01.950142 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:01.950077    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n42qf" event={"ID":"9280e2b8-f49e-4e07-b1af-a26cc9108ff9","Type":"ContainerStarted","Data":"57bba9a3029a0528da10e168d8a7675ad0c95650361b9a32ab5cf44a346f4805"}
Sep 16 16:37:01.951502 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:01.951481    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tdf5s" event={"ID":"0b7cc8be-e4bc-477e-8166-f539c64992a4","Type":"ContainerStarted","Data":"2cdf37864e42e91a89e571f692944ffe4c2ea128ed51ab3b14554a88d85a034b"}
Sep 16 16:37:01.953297 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:01.953273    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r" event={"ID":"3ce6095e-bc1c-489f-92b9-7bf8281c2e61","Type":"ContainerStarted","Data":"a69addcb2d5f2a17a602a21b3f045805f5ad406aca9f976b74c16522103614a8"}
Sep 16 16:37:01.954760 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:01.954734    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lktnv" event={"ID":"e5adc7d5-f860-4c8d-992d-963756104953","Type":"ContainerStarted","Data":"9b3b95b987d8a31712b90375ffc21b5d8d7659fc820afc95f3b27722e2fc3725"}
Sep 16 16:37:01.956160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:01.956140    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qf2sz" event={"ID":"321e79fa-919d-436e-a11d-a4525450a85c","Type":"ContainerStarted","Data":"e4a009843d9be56522e02dd355a33460c85ce63d0b9ada858a6ab1273feb0f37"}
Sep 16 16:37:01.957578 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:01.957541    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-120.ec2.internal" podStartSLOduration=19.957531126 podStartE2EDuration="19.957531126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 16:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:37:01.957344025 +0000 UTC m=+21.719305472" watchObservedRunningTime="2026-09-16 16:37:01.957531126 +0000 UTC m=+21.719492572"
Sep 16 16:37:01.973689 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:01.972424    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qf2sz" podStartSLOduration=2.5765471079999998 podStartE2EDuration="20.972406669s" totalImagesPullingTime="18.395859561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 16:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:37:01.971024972 +0000 UTC m=+21.732986419" watchObservedRunningTime="2026-09-16 16:37:01.972406669 +0000 UTC m=+21.734368118"
Sep 16 16:37:01.984830 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:01.984799    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-knvfz" podStartSLOduration=2.701460978 podStartE2EDuration="20.984788242s" totalImagesPullingTime="18.283327264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 16:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:37:01.984788068 +0000 UTC m=+21.746749518" watchObservedRunningTime="2026-09-16 16:37:01.984788242 +0000 UTC m=+21.746749689"
Sep 16 16:37:02.031186 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:02.031152    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tdf5s" podStartSLOduration=2.70911768 podStartE2EDuration="21.031139975s" totalImagesPullingTime="18.322022295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:37:02.031044229 +0000 UTC m=+21.793005675" watchObservedRunningTime="2026-09-16 16:37:02.031139975 +0000 UTC m=+21.793101421"
Sep 16 16:37:02.068692 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:02.068661    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lktnv" podStartSLOduration=2.641182458 podStartE2EDuration="21.068649112s" totalImagesPullingTime="18.427466654s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 16:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:37:02.05052814 +0000 UTC m=+21.812489586" watchObservedRunningTime="2026-09-16 16:37:02.068649112 +0000 UTC m=+21.830610558"
Sep 16 16:37:02.068997 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:02.068973    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mlt5x" podStartSLOduration=2.807360296 podStartE2EDuration="21.06896909s" totalImagesPullingTime="18.261608794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 16:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:37:02.068495635 +0000 UTC m=+21.830457095" watchObservedRunningTime="2026-09-16 16:37:02.06896909 +0000 UTC m=+21.830930549"
Sep 16 16:37:02.669995 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:02.669972    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tdf5s"
Sep 16 16:37:02.670635 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:02.670616    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tdf5s"
Sep 16 16:37:02.769392 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:02.769369    2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 16:37:02.842537 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:02.842499    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:37:02.842706 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:02.842634    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79v7b" podUID="568fe78a-6a27-4de3-a23b-030e47b4d930"
Sep 16 16:37:02.842706 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:02.842682    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:37:02.842824 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:02.842777    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmw95" podUID="cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792"
Sep 16 16:37:02.959998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:02.959864    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m4kpr" event={"ID":"c7eef487-63a5-4d81-bf5a-40e3f673333e","Type":"ContainerStarted","Data":"e6473ed85e9174217ee3e1ac7e1a91c8f7919a92de90ddde7dbebbfec32002de"}
Sep 16 16:37:02.963398 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:02.963372    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n42qf" event={"ID":"9280e2b8-f49e-4e07-b1af-a26cc9108ff9","Type":"ContainerStarted","Data":"6423f0116dd9592bfcb5fcda65bcd02cb0aec9048df845a6f08be571b803c224"}
Sep 16 16:37:02.963512 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:02.963447    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n42qf" event={"ID":"9280e2b8-f49e-4e07-b1af-a26cc9108ff9","Type":"ContainerStarted","Data":"c4b5c1e6e6b4bbd0f5d2f5b9dc75e1796eace60b1b0e1ea6b4b37a88208a0fc6"}
Sep 16 16:37:02.965343 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:02.965320    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r" event={"ID":"3ce6095e-bc1c-489f-92b9-7bf8281c2e61","Type":"ContainerStarted","Data":"a3845a2fb088574d1d716e361877b63052001230e8669f4e0f1e6d51baa69335"}
Sep 16 16:37:03.668491 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:03.668396    2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T16:37:02.769388408Z","UUID":"33a17638-c838-4285-bf09-18c7035252f1","Handler":null,"Name":"","Endpoint":""}
Sep 16 16:37:03.672452 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:03.672430    2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 16:37:03.672578 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:03.672474    2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 16:37:03.842733 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:03.842706    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:37:03.842869 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:03.842845    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk8q9" podUID="c04662fe-4176-4a0b-ac5e-89ead5941975"
Sep 16 16:37:03.970395 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:03.970347    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r" event={"ID":"3ce6095e-bc1c-489f-92b9-7bf8281c2e61","Type":"ContainerStarted","Data":"4e49b23612c3d33f57692dd3f57e1ae97d4548e398c5725795533110c1f86b78"}
Sep 16 16:37:03.970832 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:03.970819    2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 16:37:04.001606 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:04.001554    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-m4kpr" podStartSLOduration=4.691083663 podStartE2EDuration="23.001537997s" totalImagesPullingTime="18.310454334s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 16:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:37:02.977869399 +0000 UTC m=+22.739830859" watchObservedRunningTime="2026-09-16 16:37:04.001537997 +0000 UTC m=+23.763499443"
Sep 16 16:37:04.002011 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:04.001974    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fx69r" podStartSLOduration=6.258932906 podStartE2EDuration="23.00196267s" totalImagesPullingTime="16.743029764s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 16:36:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:37:04.001177497 +0000 UTC m=+23.763138946" watchObservedRunningTime="2026-09-16 16:37:04.00196267 +0000 UTC m=+23.763924110"
Sep 16 16:37:04.842683 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:04.842647    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:37:04.842848 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:04.842786    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmw95" podUID="cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792"
Sep 16 16:37:04.842848 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:04.842841    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:37:04.842987 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:04.842962    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79v7b" podUID="568fe78a-6a27-4de3-a23b-030e47b4d930"
Sep 16 16:37:04.976256 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:04.976224    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n42qf" event={"ID":"9280e2b8-f49e-4e07-b1af-a26cc9108ff9","Type":"ContainerStarted","Data":"2ac4c7b7a6cd2ea73ac14f40d068b5c329c1cdce423d0cfe35d60a87cdb0c335"}
Sep 16 16:37:05.842317 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:05.842286    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:37:05.842492 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:05.842412    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk8q9" podUID="c04662fe-4176-4a0b-ac5e-89ead5941975"
Sep 16 16:37:06.842706 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:06.842488    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:37:06.843319 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:06.842558    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:37:06.843319 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:06.843063    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79v7b" podUID="568fe78a-6a27-4de3-a23b-030e47b4d930"
Sep 16 16:37:06.843605 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:06.843213    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmw95" podUID="cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792"
Sep 16 16:37:06.982756 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:06.982718    2556 generic.go:356] "Generic (PLEG): container finished" podID="2009b752-ad4d-4e71-8a91-3b3b41c44725" containerID="c5e71a026fc08aa0a983b67ae0ec27599d8fa8be0a4d24104b298b8b719663f6" exitCode=0
Sep 16 16:37:06.982756 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:06.982757    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7ksk" event={"ID":"2009b752-ad4d-4e71-8a91-3b3b41c44725","Type":"ContainerDied","Data":"c5e71a026fc08aa0a983b67ae0ec27599d8fa8be0a4d24104b298b8b719663f6"}
Sep 16 16:37:06.986694 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:06.986674    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n42qf" event={"ID":"9280e2b8-f49e-4e07-b1af-a26cc9108ff9","Type":"ContainerStarted","Data":"10f297f7a3315716c567c4c70aee69b31008a5341e4fe447c6facc5097e22b80"}
Sep 16 16:37:06.986901 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:06.986886    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:37:06.986982 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:06.986967    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:37:07.001551 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:07.001532    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:37:07.075337 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:07.075295    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n42qf" podStartSLOduration=7.426729198 podStartE2EDuration="26.075281148s" totalImagesPullingTime="18.64855195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 16:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:37:07.071775593 +0000 UTC m=+26.833737034" watchObservedRunningTime="2026-09-16 16:37:07.075281148 +0000 UTC m=+26.837242595"
Sep 16 16:37:07.842870 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:07.842840    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:37:07.843276 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:07.842964    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk8q9" podUID="c04662fe-4176-4a0b-ac5e-89ead5941975"
Sep 16 16:37:07.991610 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:07.991579    2556 generic.go:356] "Generic (PLEG): container finished" podID="2009b752-ad4d-4e71-8a91-3b3b41c44725" containerID="00a9b5bd5860ca1a5a223271dd30fdb1c1dea045dbfda456902173c273b4476c" exitCode=0
Sep 16 16:37:07.991760 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:07.991659    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7ksk" event={"ID":"2009b752-ad4d-4e71-8a91-3b3b41c44725","Type":"ContainerDied","Data":"00a9b5bd5860ca1a5a223271dd30fdb1c1dea045dbfda456902173c273b4476c"}
Sep 16 16:37:07.992195 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:07.992171    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:37:08.006506 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:08.006487    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:37:08.523066 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:08.522810    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-79v7b"]
Sep 16 16:37:08.523194 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:08.523106    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:37:08.523229 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:08.523185    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79v7b" podUID="568fe78a-6a27-4de3-a23b-030e47b4d930"
Sep 16 16:37:08.533838 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:08.533815    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kmw95"]
Sep 16 16:37:08.533975 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:08.533929    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:37:08.534063 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:08.534044    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmw95" podUID="cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792"
Sep 16 16:37:08.534457 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:08.534436    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hk8q9"]
Sep 16 16:37:08.534531 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:08.534520    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:37:08.534608 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:08.534592    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk8q9" podUID="c04662fe-4176-4a0b-ac5e-89ead5941975"
Sep 16 16:37:09.998280 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:09.998248    2556 generic.go:356] "Generic (PLEG): container finished" podID="2009b752-ad4d-4e71-8a91-3b3b41c44725" containerID="724929b5ebe760caa0df9b1443ca227ff28c4fdf4518db4cbb573602f4c7ddea" exitCode=0
Sep 16 16:37:09.998777 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:09.998332    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7ksk" event={"ID":"2009b752-ad4d-4e71-8a91-3b3b41c44725","Type":"ContainerDied","Data":"724929b5ebe760caa0df9b1443ca227ff28c4fdf4518db4cbb573602f4c7ddea"}
Sep 16 16:37:10.843199 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:10.843166    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:37:10.843355 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:10.843246    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:37:10.843355 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:10.843256    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmw95" podUID="cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792"
Sep 16 16:37:10.843459 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:10.843365    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:37:10.843459 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:10.843370    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk8q9" podUID="c04662fe-4176-4a0b-ac5e-89ead5941975"
Sep 16 16:37:10.843459 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:10.843442    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79v7b" podUID="568fe78a-6a27-4de3-a23b-030e47b4d930"
Sep 16 16:37:11.992009 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:11.991976    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tdf5s"
Sep 16 16:37:11.992523 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:11.992114    2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 16:37:11.992574 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:11.992526    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tdf5s"
Sep 16 16:37:12.842300 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:12.842270    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:37:12.842300 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:12.842298    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:37:12.842492 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:12.842277    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:37:12.842492 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:12.842382    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kmw95" podUID="cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792"
Sep 16 16:37:12.842492 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:12.842456    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hk8q9" podUID="c04662fe-4176-4a0b-ac5e-89ead5941975"
Sep 16 16:37:12.842593 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:12.842518    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-79v7b" podUID="568fe78a-6a27-4de3-a23b-030e47b4d930"
Sep 16 16:37:13.964847 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:13.964824    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-120.ec2.internal" event="NodeReady"
Sep 16 16:37:13.965234 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:13.964989    2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 16:37:14.027091 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.027059    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z2tqs"]
Sep 16 16:37:14.048526 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.047917    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n4drw"]
Sep 16 16:37:14.048526 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.048116    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:37:14.050708 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.050667    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bqrg9\""
Sep 16 16:37:14.051196 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.051177    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 16:37:14.051344 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.051181    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 16:37:14.060008 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.059988    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n4drw"]
Sep 16 16:37:14.060094 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.060019    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z2tqs"]
Sep 16 16:37:14.060134 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.060113    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n4drw"
Sep 16 16:37:14.063719 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.063697    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 16:37:14.064339 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.064319    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 16:37:14.065056 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.065035    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 16:37:14.065256 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.065239    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6pkw2\""
Sep 16 16:37:14.149267 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.149231    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:37:14.149442 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.149277    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/611f8865-ebde-45ab-b546-9f0be6d42d81-config-volume\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:37:14.149442 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.149344    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9pr2\" (UniqueName: \"kubernetes.io/projected/611f8865-ebde-45ab-b546-9f0be6d42d81-kube-api-access-s9pr2\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:37:14.149442 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.149362    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/611f8865-ebde-45ab-b546-9f0be6d42d81-tmp-dir\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:37:14.149442 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.149376    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01475a36-0404-44f7-b7c9-7293e3729dd4-cert\") pod \"ingress-canary-n4drw\" (UID: \"01475a36-0404-44f7-b7c9-7293e3729dd4\") " pod="openshift-ingress-canary/ingress-canary-n4drw"
Sep 16 16:37:14.149627 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.149497    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnvtf\" (UniqueName: \"kubernetes.io/projected/01475a36-0404-44f7-b7c9-7293e3729dd4-kube-api-access-fnvtf\") pod \"ingress-canary-n4drw\" (UID: \"01475a36-0404-44f7-b7c9-7293e3729dd4\") " pod="openshift-ingress-canary/ingress-canary-n4drw"
Sep 16 16:37:14.250206 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.250130    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnvtf\" (UniqueName: \"kubernetes.io/projected/01475a36-0404-44f7-b7c9-7293e3729dd4-kube-api-access-fnvtf\") pod \"ingress-canary-n4drw\" (UID: \"01475a36-0404-44f7-b7c9-7293e3729dd4\") " pod="openshift-ingress-canary/ingress-canary-n4drw"
Sep 16 16:37:14.250206 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.250170    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:37:14.250206 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.250204    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/611f8865-ebde-45ab-b546-9f0be6d42d81-config-volume\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:37:14.250467 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.250251    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9pr2\" (UniqueName: \"kubernetes.io/projected/611f8865-ebde-45ab-b546-9f0be6d42d81-kube-api-access-s9pr2\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:37:14.250467 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.250272    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/611f8865-ebde-45ab-b546-9f0be6d42d81-tmp-dir\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:37:14.250467 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.250292    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01475a36-0404-44f7-b7c9-7293e3729dd4-cert\") pod \"ingress-canary-n4drw\" (UID: \"01475a36-0404-44f7-b7c9-7293e3729dd4\") " pod="openshift-ingress-canary/ingress-canary-n4drw"
Sep 16 16:37:14.250467 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:14.250302    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 16:37:14.250467 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:14.250371    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls podName:611f8865-ebde-45ab-b546-9f0be6d42d81 nodeName:}" failed. No retries permitted until 2026-09-16 16:37:14.75035226 +0000 UTC m=+34.512313705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls") pod "dns-default-z2tqs" (UID: "611f8865-ebde-45ab-b546-9f0be6d42d81") : secret "dns-default-metrics-tls" not found
Sep 16 16:37:14.250790 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.250766    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/611f8865-ebde-45ab-b546-9f0be6d42d81-tmp-dir\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:37:14.250904 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.250888    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/611f8865-ebde-45ab-b546-9f0be6d42d81-config-volume\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:37:14.255039 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.255014    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01475a36-0404-44f7-b7c9-7293e3729dd4-cert\") pod \"ingress-canary-n4drw\" (UID: \"01475a36-0404-44f7-b7c9-7293e3729dd4\") " pod="openshift-ingress-canary/ingress-canary-n4drw"
Sep 16 16:37:14.260498 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.260477    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnvtf\" (UniqueName: \"kubernetes.io/projected/01475a36-0404-44f7-b7c9-7293e3729dd4-kube-api-access-fnvtf\") pod \"ingress-canary-n4drw\" (UID: \"01475a36-0404-44f7-b7c9-7293e3729dd4\") " pod="openshift-ingress-canary/ingress-canary-n4drw"
Sep 16 16:37:14.261821 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.261798    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9pr2\" (UniqueName: \"kubernetes.io/projected/611f8865-ebde-45ab-b546-9f0be6d42d81-kube-api-access-s9pr2\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:37:14.351013 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.350980    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret\") pod \"global-pull-secret-syncer-kmw95\" (UID: \"cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792\") " pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:37:14.351182 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:14.351108    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 16:37:14.351182 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:14.351176    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret podName:cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792 nodeName:}" failed. No retries permitted until 2026-09-16 16:37:46.351158406 +0000 UTC m=+66.113119846 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret") pod "global-pull-secret-syncer-kmw95" (UID: "cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792") : object "kube-system"/"original-pull-secret" not registered
Sep 16 16:37:14.370270 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.370242    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n4drw"
Sep 16 16:37:14.452417 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.452177    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs\") pod \"network-metrics-daemon-hk8q9\" (UID: \"c04662fe-4176-4a0b-ac5e-89ead5941975\") " pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:37:14.452549 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:14.452333    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 16:37:14.452549 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:14.452488    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs podName:c04662fe-4176-4a0b-ac5e-89ead5941975 nodeName:}" failed. No retries permitted until 2026-09-16 16:37:46.452466901 +0000 UTC m=+66.214428329 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs") pod "network-metrics-daemon-hk8q9" (UID: "c04662fe-4176-4a0b-ac5e-89ead5941975") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 16:37:14.553642 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.553573    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcnhp\" (UniqueName: \"kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp\") pod \"network-check-target-79v7b\" (UID: \"568fe78a-6a27-4de3-a23b-030e47b4d930\") " pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:37:14.553760 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:14.553711    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 16:37:14.553760 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:14.553725    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 16:37:14.553760 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:14.553733    2556 projected.go:196] Error preparing data for projected volume kube-api-access-gcnhp for pod openshift-network-diagnostics/network-check-target-79v7b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 16:37:14.553881 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:14.553779    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp podName:568fe78a-6a27-4de3-a23b-030e47b4d930 nodeName:}" failed. No retries permitted until 2026-09-16 16:37:46.553765305 +0000 UTC m=+66.315726735 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gcnhp" (UniqueName: "kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp") pod "network-check-target-79v7b" (UID: "568fe78a-6a27-4de3-a23b-030e47b4d930") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 16:37:14.754956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.754904    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:37:14.755087 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:14.755055    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 16:37:14.755132 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:14.755122    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls podName:611f8865-ebde-45ab-b546-9f0be6d42d81 nodeName:}" failed. No retries permitted until 2026-09-16 16:37:15.755106361 +0000 UTC m=+35.517067786 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls") pod "dns-default-z2tqs" (UID: "611f8865-ebde-45ab-b546-9f0be6d42d81") : secret "dns-default-metrics-tls" not found
Sep 16 16:37:14.842074 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.842007    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:37:14.842074 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.842042    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:37:14.842249 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.842010    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:37:14.845239 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.845218    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jtfpq\""
Sep 16 16:37:14.845473 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.845453    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 16:37:14.845590 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.845486    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 16:37:14.845590 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.845500    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v6lts\""
Sep 16 16:37:14.845694 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.845631    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 16:37:14.845694 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.845690    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 16:37:14.845987 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:14.845971    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n4drw"]
Sep 16 16:37:14.849188 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:37:14.849166    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01475a36_0404_44f7_b7c9_7293e3729dd4.slice/crio-5617103c1a0f7b0f9053018c4ced5f0391f696e57f22dd744c88dea54c4f2d42 WatchSource:0}: Error finding container 5617103c1a0f7b0f9053018c4ced5f0391f696e57f22dd744c88dea54c4f2d42: Status 404 returned error can't find the container with id 5617103c1a0f7b0f9053018c4ced5f0391f696e57f22dd744c88dea54c4f2d42
Sep 16 16:37:15.012396 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:15.012367    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n4drw" event={"ID":"01475a36-0404-44f7-b7c9-7293e3729dd4","Type":"ContainerStarted","Data":"5617103c1a0f7b0f9053018c4ced5f0391f696e57f22dd744c88dea54c4f2d42"}
Sep 16 16:37:15.015452 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:15.015432    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7ksk" event={"ID":"2009b752-ad4d-4e71-8a91-3b3b41c44725","Type":"ContainerStarted","Data":"908338f5edc491bf52cb50a3cc304376435e5dc0cfb3b12182b6c9967693e6d5"}
Sep 16 16:37:15.762476 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:15.762419    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:37:15.762634 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:15.762593    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 16:37:15.762699 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:15.762666    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls podName:611f8865-ebde-45ab-b546-9f0be6d42d81 nodeName:}" failed. No retries permitted until 2026-09-16 16:37:17.762645403 +0000 UTC m=+37.524606829 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls") pod "dns-default-z2tqs" (UID: "611f8865-ebde-45ab-b546-9f0be6d42d81") : secret "dns-default-metrics-tls" not found
Sep 16 16:37:16.021031 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:16.020930    2556 generic.go:356] "Generic (PLEG): container finished" podID="2009b752-ad4d-4e71-8a91-3b3b41c44725" containerID="908338f5edc491bf52cb50a3cc304376435e5dc0cfb3b12182b6c9967693e6d5" exitCode=0
Sep 16 16:37:16.021031 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:16.020974    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7ksk" event={"ID":"2009b752-ad4d-4e71-8a91-3b3b41c44725","Type":"ContainerDied","Data":"908338f5edc491bf52cb50a3cc304376435e5dc0cfb3b12182b6c9967693e6d5"}
Sep 16 16:37:17.027111 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:17.027079    2556 generic.go:356] "Generic (PLEG): container finished" podID="2009b752-ad4d-4e71-8a91-3b3b41c44725" containerID="09f5d48a11bd3510fbfcf7dc359459accab1fbd7a048283302ac1e7a9cfe3647" exitCode=0
Sep 16 16:37:17.027693 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:17.027120    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7ksk" event={"ID":"2009b752-ad4d-4e71-8a91-3b3b41c44725","Type":"ContainerDied","Data":"09f5d48a11bd3510fbfcf7dc359459accab1fbd7a048283302ac1e7a9cfe3647"}
Sep 16 16:37:17.780523 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:17.780492    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:37:17.780705 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:17.780634    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 16:37:17.780705 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:17.780694    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls podName:611f8865-ebde-45ab-b546-9f0be6d42d81 nodeName:}" failed. No retries permitted until 2026-09-16 16:37:21.780677426 +0000 UTC m=+41.542638851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls") pod "dns-default-z2tqs" (UID: "611f8865-ebde-45ab-b546-9f0be6d42d81") : secret "dns-default-metrics-tls" not found
Sep 16 16:37:18.031220 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:18.031138    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n4drw" event={"ID":"01475a36-0404-44f7-b7c9-7293e3729dd4","Type":"ContainerStarted","Data":"fc3daab11e0a272df3ab773c9eefd997526c22c953ef79e3971179c89347ff58"}
Sep 16 16:37:18.034480 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:18.034452    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g7ksk" event={"ID":"2009b752-ad4d-4e71-8a91-3b3b41c44725","Type":"ContainerStarted","Data":"eb765a30f0204bcb5bb8c5476042fb8804e5a8ad656ea0e628ed11b17781ed6d"}
Sep 16 16:37:18.047388 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:18.047351    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n4drw" podStartSLOduration=1.457466604 podStartE2EDuration="4.047335669s" totalImagesPullingTime="2.589869065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:37:18.046670963 +0000 UTC m=+37.808632423" watchObservedRunningTime="2026-09-16 16:37:18.047335669 +0000 UTC m=+37.809297117"
Sep 16 16:37:18.070215 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:18.070168    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g7ksk" podStartSLOduration=7.64457986 podStartE2EDuration="37.070149725s" totalImagesPullingTime="29.425569865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 16:36:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:37:18.068792782 +0000 UTC m=+37.830754228" watchObservedRunningTime="2026-09-16 16:37:18.070149725 +0000 UTC m=+37.832111175"
Sep 16 16:37:21.805703 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:21.805663    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:37:21.806100 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:21.805786    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 16:37:21.806100 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:21.805837    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls podName:611f8865-ebde-45ab-b546-9f0be6d42d81 nodeName:}" failed. No retries permitted until 2026-09-16 16:37:29.805824582 +0000 UTC m=+49.567786007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls") pod "dns-default-z2tqs" (UID: "611f8865-ebde-45ab-b546-9f0be6d42d81") : secret "dns-default-metrics-tls" not found
Sep 16 16:37:29.855139 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:29.855097    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:37:29.855546 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:29.855219    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 16:37:29.855546 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:29.855272    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls podName:611f8865-ebde-45ab-b546-9f0be6d42d81 nodeName:}" failed. No retries permitted until 2026-09-16 16:37:45.855259271 +0000 UTC m=+65.617220696 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls") pod "dns-default-z2tqs" (UID: "611f8865-ebde-45ab-b546-9f0be6d42d81") : secret "dns-default-metrics-tls" not found
Sep 16 16:37:40.008470 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:40.008442    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n42qf"
Sep 16 16:37:45.860848 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:45.860813    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:37:45.861330 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:45.860962    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 16:37:45.861330 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:45.861015    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls podName:611f8865-ebde-45ab-b546-9f0be6d42d81 nodeName:}" failed. No retries permitted until 2026-09-16 16:38:17.861000008 +0000 UTC m=+97.622961433 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls") pod "dns-default-z2tqs" (UID: "611f8865-ebde-45ab-b546-9f0be6d42d81") : secret "dns-default-metrics-tls" not found
Sep 16 16:37:46.364055 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:46.364021    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret\") pod \"global-pull-secret-syncer-kmw95\" (UID: \"cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792\") " pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:37:46.366752 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:46.366735    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 16:37:46.377666 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:46.377644    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792-original-pull-secret\") pod \"global-pull-secret-syncer-kmw95\" (UID: \"cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792\") " pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:37:46.465027 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:46.465000    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs\") pod \"network-metrics-daemon-hk8q9\" (UID: \"c04662fe-4176-4a0b-ac5e-89ead5941975\") " pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:37:46.468469 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:46.468453    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 16:37:46.476051 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:46.476031    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 16:37:46.476109 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:37:46.476099    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs podName:c04662fe-4176-4a0b-ac5e-89ead5941975 nodeName:}" failed. No retries permitted until 2026-09-16 16:38:50.476082255 +0000 UTC m=+130.238043679 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs") pod "network-metrics-daemon-hk8q9" (UID: "c04662fe-4176-4a0b-ac5e-89ead5941975") : secret "metrics-daemon-secret" not found
Sep 16 16:37:46.566241 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:46.566214    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcnhp\" (UniqueName: \"kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp\") pod \"network-check-target-79v7b\" (UID: \"568fe78a-6a27-4de3-a23b-030e47b4d930\") " pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:37:46.569692 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:46.569675    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 16:37:46.578421 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:46.578403    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 16:37:46.589041 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:46.589023    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcnhp\" (UniqueName: \"kubernetes.io/projected/568fe78a-6a27-4de3-a23b-030e47b4d930-kube-api-access-gcnhp\") pod \"network-check-target-79v7b\" (UID: \"568fe78a-6a27-4de3-a23b-030e47b4d930\") " pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:37:46.654923 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:46.654876    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kmw95"
Sep 16 16:37:46.663734 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:46.663711    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jtfpq\""
Sep 16 16:37:46.671847 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:46.671820    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:37:46.940648 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:46.940626    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kmw95"]
Sep 16 16:37:46.942900 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:37:46.942874    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9b4fb9_8db8_432c_a6d6_d7a1d32f7792.slice/crio-7dfa983f0409af2812ef29ba9b334e7596c6f7239e3d815c8dbbc81aee638be9 WatchSource:0}: Error finding container 7dfa983f0409af2812ef29ba9b334e7596c6f7239e3d815c8dbbc81aee638be9: Status 404 returned error can't find the container with id 7dfa983f0409af2812ef29ba9b334e7596c6f7239e3d815c8dbbc81aee638be9
Sep 16 16:37:46.961149 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:46.961127    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-79v7b"]
Sep 16 16:37:46.964453 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:37:46.964429    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568fe78a_6a27_4de3_a23b_030e47b4d930.slice/crio-ba55de80190a96b6f23d8728bace7275205edede308557fd401abb316a17c426 WatchSource:0}: Error finding container ba55de80190a96b6f23d8728bace7275205edede308557fd401abb316a17c426: Status 404 returned error can't find the container with id ba55de80190a96b6f23d8728bace7275205edede308557fd401abb316a17c426
Sep 16 16:37:47.103200 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:47.103162    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-79v7b" event={"ID":"568fe78a-6a27-4de3-a23b-030e47b4d930","Type":"ContainerStarted","Data":"ba55de80190a96b6f23d8728bace7275205edede308557fd401abb316a17c426"}
Sep 16 16:37:47.104248 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:47.104225    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kmw95" event={"ID":"cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792","Type":"ContainerStarted","Data":"7dfa983f0409af2812ef29ba9b334e7596c6f7239e3d815c8dbbc81aee638be9"}
Sep 16 16:37:52.119060 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:52.118969    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kmw95" event={"ID":"cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792","Type":"ContainerStarted","Data":"5b1b8923c00ce161003912174ff97c917e414d5ebe955fc5a1aa4d89709e034a"}
Sep 16 16:37:52.120295 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:52.120274    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-79v7b" event={"ID":"568fe78a-6a27-4de3-a23b-030e47b4d930","Type":"ContainerStarted","Data":"ba48425ffe97325e505c7242d10b8cc77ae0b5f9726f37370d823aea840cd3a0"}
Sep 16 16:37:52.120406 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:52.120396    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:37:52.133264 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:52.133224    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kmw95" podStartSLOduration=66.325758519 podStartE2EDuration="1m11.133211232s" totalImagesPullingTime="4.807452713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 16:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:37:52.133009522 +0000 UTC m=+71.894970969" watchObservedRunningTime="2026-09-16 16:37:52.133211232 +0000 UTC m=+71.895172679"
Sep 16 16:37:52.148357 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:37:52.148321    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-79v7b" podStartSLOduration=66.365479763 podStartE2EDuration="1m11.148308947s" totalImagesPullingTime="4.782829184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:37:52.147790265 +0000 UTC m=+71.909751713" watchObservedRunningTime="2026-09-16 16:37:52.148308947 +0000 UTC m=+71.910270393"
Sep 16 16:38:14.662093 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.662052    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx"]
Sep 16 16:38:14.664451 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.664434    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx"
Sep 16 16:38:14.666672 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.666650    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 16 16:38:14.666672 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.666668    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-bk6gg\""
Sep 16 16:38:14.666828 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.666730    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 16 16:38:14.667357 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.667342    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 16 16:38:14.667497 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.667480    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 16 16:38:14.675979 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.675958    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx"]
Sep 16 16:38:14.680313 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.680295    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"]
Sep 16 16:38:14.682204 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.682190    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:38:14.685237 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.685220    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 16 16:38:14.685772 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.685754    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 16 16:38:14.686065 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.686049    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 16 16:38:14.687367 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.687351    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 16 16:38:14.705268 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.705236    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"]
Sep 16 16:38:14.749780 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.749756    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/62ecfba4-b079-4546-bfe9-79950aeedea6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56ff7c4fc4-sndw2\" (UID: \"62ecfba4-b079-4546-bfe9-79950aeedea6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:38:14.749881 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.749784    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/62ecfba4-b079-4546-bfe9-79950aeedea6-hub\") pod \"cluster-proxy-proxy-agent-56ff7c4fc4-sndw2\" (UID: \"62ecfba4-b079-4546-bfe9-79950aeedea6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:38:14.749881 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.749807    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ecfba4-b079-4546-bfe9-79950aeedea6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56ff7c4fc4-sndw2\" (UID: \"62ecfba4-b079-4546-bfe9-79950aeedea6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:38:14.749881 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.749834    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/62ecfba4-b079-4546-bfe9-79950aeedea6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56ff7c4fc4-sndw2\" (UID: \"62ecfba4-b079-4546-bfe9-79950aeedea6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:38:14.749999 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.749917    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/62ecfba4-b079-4546-bfe9-79950aeedea6-ca\") pod \"cluster-proxy-proxy-agent-56ff7c4fc4-sndw2\" (UID: \"62ecfba4-b079-4546-bfe9-79950aeedea6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:38:14.749999 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.749984    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp9l6\" (UniqueName: \"kubernetes.io/projected/62ecfba4-b079-4546-bfe9-79950aeedea6-kube-api-access-xp9l6\") pod \"cluster-proxy-proxy-agent-56ff7c4fc4-sndw2\" (UID: \"62ecfba4-b079-4546-bfe9-79950aeedea6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:38:14.750065 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.750004    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55870326-5ba3-40d3-913c-5349a78f44d4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx\" (UID: \"55870326-5ba3-40d3-913c-5349a78f44d4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx"
Sep 16 16:38:14.750065 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.750021    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dbj2\" (UniqueName: \"kubernetes.io/projected/55870326-5ba3-40d3-913c-5349a78f44d4-kube-api-access-6dbj2\") pod \"managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx\" (UID: \"55870326-5ba3-40d3-913c-5349a78f44d4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx"
Sep 16 16:38:14.851141 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.851116    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/62ecfba4-b079-4546-bfe9-79950aeedea6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56ff7c4fc4-sndw2\" (UID: \"62ecfba4-b079-4546-bfe9-79950aeedea6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:38:14.851270 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.851145    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/62ecfba4-b079-4546-bfe9-79950aeedea6-hub\") pod \"cluster-proxy-proxy-agent-56ff7c4fc4-sndw2\" (UID: \"62ecfba4-b079-4546-bfe9-79950aeedea6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:38:14.851270 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.851164    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ecfba4-b079-4546-bfe9-79950aeedea6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56ff7c4fc4-sndw2\" (UID: \"62ecfba4-b079-4546-bfe9-79950aeedea6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:38:14.851270 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.851184    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/62ecfba4-b079-4546-bfe9-79950aeedea6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56ff7c4fc4-sndw2\" (UID: \"62ecfba4-b079-4546-bfe9-79950aeedea6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:38:14.851413 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.851300    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/62ecfba4-b079-4546-bfe9-79950aeedea6-ca\") pod \"cluster-proxy-proxy-agent-56ff7c4fc4-sndw2\" (UID: \"62ecfba4-b079-4546-bfe9-79950aeedea6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:38:14.851413 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.851369    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp9l6\" (UniqueName: \"kubernetes.io/projected/62ecfba4-b079-4546-bfe9-79950aeedea6-kube-api-access-xp9l6\") pod \"cluster-proxy-proxy-agent-56ff7c4fc4-sndw2\" (UID: \"62ecfba4-b079-4546-bfe9-79950aeedea6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:38:14.851413 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.851394    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55870326-5ba3-40d3-913c-5349a78f44d4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx\" (UID: \"55870326-5ba3-40d3-913c-5349a78f44d4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx"
Sep 16 16:38:14.851565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.851416    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dbj2\" (UniqueName: \"kubernetes.io/projected/55870326-5ba3-40d3-913c-5349a78f44d4-kube-api-access-6dbj2\") pod \"managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx\" (UID: \"55870326-5ba3-40d3-913c-5349a78f44d4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx"
Sep 16 16:38:14.852036 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.852010    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ecfba4-b079-4546-bfe9-79950aeedea6-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56ff7c4fc4-sndw2\" (UID: \"62ecfba4-b079-4546-bfe9-79950aeedea6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:38:14.853729 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.853703    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/62ecfba4-b079-4546-bfe9-79950aeedea6-ca\") pod \"cluster-proxy-proxy-agent-56ff7c4fc4-sndw2\" (UID: \"62ecfba4-b079-4546-bfe9-79950aeedea6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:38:14.853925 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.853907    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/62ecfba4-b079-4546-bfe9-79950aeedea6-hub\") pod \"cluster-proxy-proxy-agent-56ff7c4fc4-sndw2\" (UID: \"62ecfba4-b079-4546-bfe9-79950aeedea6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:38:14.854001 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.853961    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/62ecfba4-b079-4546-bfe9-79950aeedea6-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56ff7c4fc4-sndw2\" (UID: \"62ecfba4-b079-4546-bfe9-79950aeedea6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:38:14.854095 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.854081    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/62ecfba4-b079-4546-bfe9-79950aeedea6-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56ff7c4fc4-sndw2\" (UID: \"62ecfba4-b079-4546-bfe9-79950aeedea6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:38:14.854177 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.854159    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/55870326-5ba3-40d3-913c-5349a78f44d4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx\" (UID: \"55870326-5ba3-40d3-913c-5349a78f44d4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx"
Sep 16 16:38:14.859233 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.859216    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dbj2\" (UniqueName: \"kubernetes.io/projected/55870326-5ba3-40d3-913c-5349a78f44d4-kube-api-access-6dbj2\") pod \"managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx\" (UID: \"55870326-5ba3-40d3-913c-5349a78f44d4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx"
Sep 16 16:38:14.859357 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.859320    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp9l6\" (UniqueName: \"kubernetes.io/projected/62ecfba4-b079-4546-bfe9-79950aeedea6-kube-api-access-xp9l6\") pod \"cluster-proxy-proxy-agent-56ff7c4fc4-sndw2\" (UID: \"62ecfba4-b079-4546-bfe9-79950aeedea6\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:38:14.984661 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.984638    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx"
Sep 16 16:38:14.991194 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:14.991082    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:38:15.252741 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:15.252552    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx"]
Sep 16 16:38:15.255519 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:38:15.255483    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55870326_5ba3_40d3_913c_5349a78f44d4.slice/crio-f05ccab1140619cf802c8fa79378737fb471405c17f8615fd608a82f9d5d803c WatchSource:0}: Error finding container f05ccab1140619cf802c8fa79378737fb471405c17f8615fd608a82f9d5d803c: Status 404 returned error can't find the container with id f05ccab1140619cf802c8fa79378737fb471405c17f8615fd608a82f9d5d803c
Sep 16 16:38:15.274451 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:15.274426    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"]
Sep 16 16:38:15.277156 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:38:15.277127    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62ecfba4_b079_4546_bfe9_79950aeedea6.slice/crio-cd6dd45148820b6a7bd68216360fa21133cb29a8f9e545cba9bccd65025d1eb7 WatchSource:0}: Error finding container cd6dd45148820b6a7bd68216360fa21133cb29a8f9e545cba9bccd65025d1eb7: Status 404 returned error can't find the container with id cd6dd45148820b6a7bd68216360fa21133cb29a8f9e545cba9bccd65025d1eb7
Sep 16 16:38:16.182874 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:16.182823    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx" event={"ID":"55870326-5ba3-40d3-913c-5349a78f44d4","Type":"ContainerStarted","Data":"f05ccab1140619cf802c8fa79378737fb471405c17f8615fd608a82f9d5d803c"}
Sep 16 16:38:16.184385 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:16.184342    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2" event={"ID":"62ecfba4-b079-4546-bfe9-79950aeedea6","Type":"ContainerStarted","Data":"cd6dd45148820b6a7bd68216360fa21133cb29a8f9e545cba9bccd65025d1eb7"}
Sep 16 16:38:17.873325 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:17.873293    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:38:17.873659 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:38:17.873438    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 16:38:17.873659 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:38:17.873507    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls podName:611f8865-ebde-45ab-b546-9f0be6d42d81 nodeName:}" failed. No retries permitted until 2026-09-16 16:39:21.873489893 +0000 UTC m=+161.635451332 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls") pod "dns-default-z2tqs" (UID: "611f8865-ebde-45ab-b546-9f0be6d42d81") : secret "dns-default-metrics-tls" not found
Sep 16 16:38:18.190404 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:18.190367    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx" event={"ID":"55870326-5ba3-40d3-913c-5349a78f44d4","Type":"ContainerStarted","Data":"74ee806093b9e98a49b6fa66af309b7b5b77dc686eff27c7d91c1a595d1a186f"}
Sep 16 16:38:18.208395 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:18.208340    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx" podStartSLOduration=1.7135440339999999 podStartE2EDuration="4.208322247s" totalImagesPullingTime="2.494778213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:38:18.207822298 +0000 UTC m=+97.969783746" watchObservedRunningTime="2026-09-16 16:38:18.208322247 +0000 UTC m=+97.970283724"
Sep 16 16:38:23.125154 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:23.125121    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-79v7b"
Sep 16 16:38:35.232586 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:35.232553    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2" event={"ID":"62ecfba4-b079-4546-bfe9-79950aeedea6","Type":"ContainerStarted","Data":"5332901a26855a04e012a023f728c187d435fcfefcde91d2220b0b23ff9c1232"}
Sep 16 16:38:35.233042 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:35.232598    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2" event={"ID":"62ecfba4-b079-4546-bfe9-79950aeedea6","Type":"ContainerStarted","Data":"1aa5fcfd4e031045325c37990d6bc2e4e9db1513d382432b8a52f69aa3e01a67"}
Sep 16 16:38:35.233042 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:35.232615    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2" event={"ID":"62ecfba4-b079-4546-bfe9-79950aeedea6","Type":"ContainerStarted","Data":"101964016ad87b5fd577d32b5d16ceffb0f9df3ba7f4b00b32d1386c0c0144a5"}
Sep 16 16:38:35.250763 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:35.250719    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2" podStartSLOduration=1.487908634 podStartE2EDuration="21.250705617s" totalImagesPullingTime="19.762796983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:38:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:38:35.249964072 +0000 UTC m=+115.011925512" watchObservedRunningTime="2026-09-16 16:38:35.250705617 +0000 UTC m=+115.012667064"
Sep 16 16:38:50.486510 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:38:50.486452    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs\") pod \"network-metrics-daemon-hk8q9\" (UID: \"c04662fe-4176-4a0b-ac5e-89ead5941975\") " pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:38:50.487018 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:38:50.486577    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 16:38:50.487018 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:38:50.486635    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs podName:c04662fe-4176-4a0b-ac5e-89ead5941975 nodeName:}" failed. No retries permitted until 2026-09-16 16:40:52.486619718 +0000 UTC m=+252.248581143 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs") pod "network-metrics-daemon-hk8q9" (UID: "c04662fe-4176-4a0b-ac5e-89ead5941975") : secret "metrics-daemon-secret" not found
Sep 16 16:39:17.059368 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:39:17.059322    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-z2tqs" podUID="611f8865-ebde-45ab-b546-9f0be6d42d81"
Sep 16 16:39:17.352832 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:39:17.352746    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:39:17.866460 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:39:17.866423    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-hk8q9" podUID="c04662fe-4176-4a0b-ac5e-89ead5941975"
Sep 16 16:39:21.893517 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:39:21.893475    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:39:21.893901 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:39:21.893600    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 16:39:21.893901 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:39:21.893666    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls podName:611f8865-ebde-45ab-b546-9f0be6d42d81 nodeName:}" failed. No retries permitted until 2026-09-16 16:41:23.893652448 +0000 UTC m=+283.655613874 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls") pod "dns-default-z2tqs" (UID: "611f8865-ebde-45ab-b546-9f0be6d42d81") : secret "dns-default-metrics-tls" not found
Sep 16 16:39:30.843833 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:39:30.843798    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:40:18.527437 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:18.527401    2556 generic.go:356] "Generic (PLEG): container finished" podID="55870326-5ba3-40d3-913c-5349a78f44d4" containerID="74ee806093b9e98a49b6fa66af309b7b5b77dc686eff27c7d91c1a595d1a186f" exitCode=255
Sep 16 16:40:18.527986 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:18.527479    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx" event={"ID":"55870326-5ba3-40d3-913c-5349a78f44d4","Type":"ContainerDied","Data":"74ee806093b9e98a49b6fa66af309b7b5b77dc686eff27c7d91c1a595d1a186f"}
Sep 16 16:40:18.527986 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:18.527841    2556 scope.go:117] "RemoveContainer" containerID="74ee806093b9e98a49b6fa66af309b7b5b77dc686eff27c7d91c1a595d1a186f"
Sep 16 16:40:19.531765 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:19.531733    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5846dd6f8d-4hnfx" event={"ID":"55870326-5ba3-40d3-913c-5349a78f44d4","Type":"ContainerStarted","Data":"1bab11e1b0248dca626aac67a2b4295758fa0883525fd04e4925b08364af9f62"}
Sep 16 16:40:52.487435 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:52.487400    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs\") pod \"network-metrics-daemon-hk8q9\" (UID: \"c04662fe-4176-4a0b-ac5e-89ead5941975\") " pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:40:52.487910 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:40:52.487543    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 16:40:52.487910 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:40:52.487637    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs podName:c04662fe-4176-4a0b-ac5e-89ead5941975 nodeName:}" failed. No retries permitted until 2026-09-16 16:42:54.487618942 +0000 UTC m=+374.249580370 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs") pod "network-metrics-daemon-hk8q9" (UID: "c04662fe-4176-4a0b-ac5e-89ead5941975") : secret "metrics-daemon-secret" not found
Sep 16 16:40:56.038720 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.038687    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p6srw"]
Sep 16 16:40:56.041971 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.041933    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:40:56.043991 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.043967    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 16:40:56.044112 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.044013    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 16:40:56.044811 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.044796    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8nnmt\""
Sep 16 16:40:56.044871 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.044821    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 16:40:56.044917 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.044800    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 16:40:56.054186 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.054165    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p6srw"]
Sep 16 16:40:56.111167 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.111144    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5dd963a7-182d-4dae-a474-aa7fc60faf2a-crio-socket\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:40:56.111167 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.111166    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5dd963a7-182d-4dae-a474-aa7fc60faf2a-data-volume\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:40:56.111324 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.111194    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcl9g\" (UniqueName: \"kubernetes.io/projected/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-api-access-pcl9g\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:40:56.111324 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.111250    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:40:56.111324 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.111284    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:40:56.212344 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.212308    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:40:56.212464 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.212370    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5dd963a7-182d-4dae-a474-aa7fc60faf2a-crio-socket\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:40:56.212464 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.212386    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5dd963a7-182d-4dae-a474-aa7fc60faf2a-data-volume\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:40:56.212464 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.212411    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcl9g\" (UniqueName: \"kubernetes.io/projected/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-api-access-pcl9g\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:40:56.212464 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.212432    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:40:56.212464 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:40:56.212451    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 16:40:56.212724 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:40:56.212509    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls podName:5dd963a7-182d-4dae-a474-aa7fc60faf2a nodeName:}" failed. No retries permitted until 2026-09-16 16:40:56.712492888 +0000 UTC m=+256.474454312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p6srw" (UID: "5dd963a7-182d-4dae-a474-aa7fc60faf2a") : secret "insights-runtime-extractor-tls" not found
Sep 16 16:40:56.212724 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.212448    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5dd963a7-182d-4dae-a474-aa7fc60faf2a-crio-socket\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:40:56.212724 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:40:56.212562    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 16:40:56.212724 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:40:56.212618    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-rbac-proxy-cm podName:5dd963a7-182d-4dae-a474-aa7fc60faf2a nodeName:}" failed. No retries permitted until 2026-09-16 16:40:56.712605583 +0000 UTC m=+256.474567026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p6srw" (UID: "5dd963a7-182d-4dae-a474-aa7fc60faf2a") : configmap "kube-rbac-proxy" not found
Sep 16 16:40:56.212882 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.212748    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5dd963a7-182d-4dae-a474-aa7fc60faf2a-data-volume\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:40:56.220988 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.220958    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcl9g\" (UniqueName: \"kubernetes.io/projected/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-api-access-pcl9g\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:40:56.715581 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.715540    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:40:56.715581 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:56.715580    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:40:56.715802 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:40:56.715656    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 16:40:56.715802 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:40:56.715671    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 16:40:56.715802 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:40:56.715714    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-rbac-proxy-cm podName:5dd963a7-182d-4dae-a474-aa7fc60faf2a nodeName:}" failed. No retries permitted until 2026-09-16 16:40:57.715698931 +0000 UTC m=+257.477660356 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p6srw" (UID: "5dd963a7-182d-4dae-a474-aa7fc60faf2a") : configmap "kube-rbac-proxy" not found
Sep 16 16:40:56.715802 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:40:56.715730    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls podName:5dd963a7-182d-4dae-a474-aa7fc60faf2a nodeName:}" failed. No retries permitted until 2026-09-16 16:40:57.715722158 +0000 UTC m=+257.477683582 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p6srw" (UID: "5dd963a7-182d-4dae-a474-aa7fc60faf2a") : secret "insights-runtime-extractor-tls" not found
Sep 16 16:40:57.723293 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:57.723261    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:40:57.723293 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:57.723291    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:40:57.723694 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:40:57.723371    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 16:40:57.723694 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:40:57.723408    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 16:40:57.723694 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:40:57.723432    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-rbac-proxy-cm podName:5dd963a7-182d-4dae-a474-aa7fc60faf2a nodeName:}" failed. No retries permitted until 2026-09-16 16:40:59.723416577 +0000 UTC m=+259.485378032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p6srw" (UID: "5dd963a7-182d-4dae-a474-aa7fc60faf2a") : configmap "kube-rbac-proxy" not found
Sep 16 16:40:57.723694 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:40:57.723445    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls podName:5dd963a7-182d-4dae-a474-aa7fc60faf2a nodeName:}" failed. No retries permitted until 2026-09-16 16:40:59.723439489 +0000 UTC m=+259.485400914 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p6srw" (UID: "5dd963a7-182d-4dae-a474-aa7fc60faf2a") : secret "insights-runtime-extractor-tls" not found
Sep 16 16:40:58.179159 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:58.179086    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-knvfz_02df9d68-24e0-43d9-8774-273f8b55c906/dns-node-resolver/0.log"
Sep 16 16:40:59.178102 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:59.178076    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qf2sz_321e79fa-919d-436e-a11d-a4525450a85c/node-ca/0.log"
Sep 16 16:40:59.579179 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:59.579152    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n4drw_01475a36-0404-44f7-b7c9-7293e3729dd4/serve-healthcheck-canary/0.log"
Sep 16 16:40:59.737712 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:59.737664    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:40:59.737712 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:40:59.737715    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:40:59.737980 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:40:59.737786    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 16:40:59.737980 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:40:59.737822    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 16:40:59.737980 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:40:59.737844    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-rbac-proxy-cm podName:5dd963a7-182d-4dae-a474-aa7fc60faf2a nodeName:}" failed. No retries permitted until 2026-09-16 16:41:03.737831129 +0000 UTC m=+263.499792553 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p6srw" (UID: "5dd963a7-182d-4dae-a474-aa7fc60faf2a") : configmap "kube-rbac-proxy" not found
Sep 16 16:40:59.737980 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:40:59.737871    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls podName:5dd963a7-182d-4dae-a474-aa7fc60faf2a nodeName:}" failed. No retries permitted until 2026-09-16 16:41:03.737856356 +0000 UTC m=+263.499817781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p6srw" (UID: "5dd963a7-182d-4dae-a474-aa7fc60faf2a") : secret "insights-runtime-extractor-tls" not found
Sep 16 16:41:03.768740 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:03.768700    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:41:03.768740 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:03.768740    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:41:03.769291 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:41:03.768839    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 16:41:03.769291 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:41:03.768924    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-rbac-proxy-cm podName:5dd963a7-182d-4dae-a474-aa7fc60faf2a nodeName:}" failed. No retries permitted until 2026-09-16 16:41:11.768903518 +0000 UTC m=+271.530864948 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p6srw" (UID: "5dd963a7-182d-4dae-a474-aa7fc60faf2a") : configmap "kube-rbac-proxy" not found
Sep 16 16:41:03.769291 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:41:03.768852    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 16:41:03.769291 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:41:03.769006    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls podName:5dd963a7-182d-4dae-a474-aa7fc60faf2a nodeName:}" failed. No retries permitted until 2026-09-16 16:41:11.768989552 +0000 UTC m=+271.530950977 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p6srw" (UID: "5dd963a7-182d-4dae-a474-aa7fc60faf2a") : secret "insights-runtime-extractor-tls" not found
Sep 16 16:41:11.822807 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:11.822755    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:41:11.822807 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:11.822808    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:41:11.823369 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:41:11.822909    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 16:41:11.823369 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:41:11.823004    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls podName:5dd963a7-182d-4dae-a474-aa7fc60faf2a nodeName:}" failed. No retries permitted until 2026-09-16 16:41:27.822982758 +0000 UTC m=+287.584944188 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p6srw" (UID: "5dd963a7-182d-4dae-a474-aa7fc60faf2a") : secret "insights-runtime-extractor-tls" not found
Sep 16 16:41:11.823369 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:11.823277    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5dd963a7-182d-4dae-a474-aa7fc60faf2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:41:20.354099 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:41:20.354058    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-z2tqs" podUID="611f8865-ebde-45ab-b546-9f0be6d42d81"
Sep 16 16:41:20.702866 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:20.702838    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:41:23.910391 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:23.910341    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:41:23.912717 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:23.912689    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/611f8865-ebde-45ab-b546-9f0be6d42d81-metrics-tls\") pod \"dns-default-z2tqs\" (UID: \"611f8865-ebde-45ab-b546-9f0be6d42d81\") " pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:41:24.005989 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:24.005964    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bqrg9\""
Sep 16 16:41:24.013973 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:24.013954    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:41:24.280231 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:24.280201    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z2tqs"]
Sep 16 16:41:24.283190 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:41:24.283164    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611f8865_ebde_45ab_b546_9f0be6d42d81.slice/crio-e4eb972df7b91055c8af1c91dcb8568637708fa565422ba33f477cf555fbd0a1 WatchSource:0}: Error finding container e4eb972df7b91055c8af1c91dcb8568637708fa565422ba33f477cf555fbd0a1: Status 404 returned error can't find the container with id e4eb972df7b91055c8af1c91dcb8568637708fa565422ba33f477cf555fbd0a1
Sep 16 16:41:24.716339 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:24.716304    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z2tqs" event={"ID":"611f8865-ebde-45ab-b546-9f0be6d42d81","Type":"ContainerStarted","Data":"e4eb972df7b91055c8af1c91dcb8568637708fa565422ba33f477cf555fbd0a1"}
Sep 16 16:41:24.992752 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:24.992660    2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2" podUID="62ecfba4-b079-4546-bfe9-79950aeedea6" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 16:41:26.724673 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:26.724638    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z2tqs" event={"ID":"611f8865-ebde-45ab-b546-9f0be6d42d81","Type":"ContainerStarted","Data":"cff9e6f044e76c3a4af945c14d443e4ef76f5f2f2c274704763ef34881c0840b"}
Sep 16 16:41:26.725199 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:26.724692    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z2tqs" event={"ID":"611f8865-ebde-45ab-b546-9f0be6d42d81","Type":"ContainerStarted","Data":"b428c815e1b0ee121e9dd57f5c486a31be46a68a2e717081a8f19c87deb546e6"}
Sep 16 16:41:26.725199 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:26.724773    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:41:26.740989 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:26.740910    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-z2tqs" podStartSLOduration=250.937488315 podStartE2EDuration="4m12.740892691s" totalImagesPullingTime="1.803404376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:41:26.740784747 +0000 UTC m=+286.502746217" watchObservedRunningTime="2026-09-16 16:41:26.740892691 +0000 UTC m=+286.502854138"
Sep 16 16:41:27.839934 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:27.839876    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:41:27.842146 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:27.842124    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5dd963a7-182d-4dae-a474-aa7fc60faf2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p6srw\" (UID: \"5dd963a7-182d-4dae-a474-aa7fc60faf2a\") " pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:41:27.850458 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:27.850437    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p6srw"
Sep 16 16:41:28.118918 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:28.118842    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p6srw"]
Sep 16 16:41:28.121663 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:41:28.121639    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd963a7_182d_4dae_a474_aa7fc60faf2a.slice/crio-8e4b637c4cd33c2d57e8b9c9b4c9c9f4318357291956560b8a722bcc703fe22e WatchSource:0}: Error finding container 8e4b637c4cd33c2d57e8b9c9b4c9c9f4318357291956560b8a722bcc703fe22e: Status 404 returned error can't find the container with id 8e4b637c4cd33c2d57e8b9c9b4c9c9f4318357291956560b8a722bcc703fe22e
Sep 16 16:41:28.732424 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:28.732390    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p6srw" event={"ID":"5dd963a7-182d-4dae-a474-aa7fc60faf2a","Type":"ContainerStarted","Data":"ffdbb02f4c75d51572904d5271e6db55c32dfcf4a87dab0877a86fa8fd001376"}
Sep 16 16:41:28.732584 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:28.732442    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p6srw" event={"ID":"5dd963a7-182d-4dae-a474-aa7fc60faf2a","Type":"ContainerStarted","Data":"8e4b637c4cd33c2d57e8b9c9b4c9c9f4318357291956560b8a722bcc703fe22e"}
Sep 16 16:41:29.736629 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:29.736594    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p6srw" event={"ID":"5dd963a7-182d-4dae-a474-aa7fc60faf2a","Type":"ContainerStarted","Data":"c4e712dab00a601164f3a63e437e792eca3b814a4ab1462c67cc5a15f2f68f73"}
Sep 16 16:41:31.743908 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:31.743872    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p6srw" event={"ID":"5dd963a7-182d-4dae-a474-aa7fc60faf2a","Type":"ContainerStarted","Data":"f8684c7aded25c1b444276f26d67961f3ba8849e673e1b0bb23818e00e589f33"}
Sep 16 16:41:31.761534 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:31.761487    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p6srw" podStartSLOduration=33.011418049 podStartE2EDuration="35.761474671s" totalImagesPullingTime="2.750056622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 16:40:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:41:31.761137646 +0000 UTC m=+291.523099092" watchObservedRunningTime="2026-09-16 16:41:31.761474671 +0000 UTC m=+291.523436118"
Sep 16 16:41:32.598326 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.598289    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6"]
Sep 16 16:41:32.601620 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.601601    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6"
Sep 16 16:41:32.604931 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.604842    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 16:41:32.604931 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.604854    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 16 16:41:32.604931 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.604876    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 16:41:32.604931 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.604886    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-wd8xt\""
Sep 16 16:41:32.605216 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.604841    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 16 16:41:32.605216 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.605051    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 16:41:32.611908 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.611890    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6"]
Sep 16 16:41:32.773237 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.773206    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c1388e8-16e7-4f78-b30e-033c4f9b5256-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-qkmb6\" (UID: \"6c1388e8-16e7-4f78-b30e-033c4f9b5256\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6"
Sep 16 16:41:32.773608 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.773244    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdcnj\" (UniqueName: \"kubernetes.io/projected/6c1388e8-16e7-4f78-b30e-033c4f9b5256-kube-api-access-jdcnj\") pod \"prometheus-operator-68bcd7f987-qkmb6\" (UID: \"6c1388e8-16e7-4f78-b30e-033c4f9b5256\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6"
Sep 16 16:41:32.773608 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.773289    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c1388e8-16e7-4f78-b30e-033c4f9b5256-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-qkmb6\" (UID: \"6c1388e8-16e7-4f78-b30e-033c4f9b5256\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6"
Sep 16 16:41:32.773608 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.773341    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c1388e8-16e7-4f78-b30e-033c4f9b5256-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-qkmb6\" (UID: \"6c1388e8-16e7-4f78-b30e-033c4f9b5256\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6"
Sep 16 16:41:32.873794 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.873726    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdcnj\" (UniqueName: \"kubernetes.io/projected/6c1388e8-16e7-4f78-b30e-033c4f9b5256-kube-api-access-jdcnj\") pod \"prometheus-operator-68bcd7f987-qkmb6\" (UID: \"6c1388e8-16e7-4f78-b30e-033c4f9b5256\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6"
Sep 16 16:41:32.873794 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.873766    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c1388e8-16e7-4f78-b30e-033c4f9b5256-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-qkmb6\" (UID: \"6c1388e8-16e7-4f78-b30e-033c4f9b5256\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6"
Sep 16 16:41:32.873976 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.873817    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c1388e8-16e7-4f78-b30e-033c4f9b5256-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-qkmb6\" (UID: \"6c1388e8-16e7-4f78-b30e-033c4f9b5256\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6"
Sep 16 16:41:32.873976 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.873868    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c1388e8-16e7-4f78-b30e-033c4f9b5256-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-qkmb6\" (UID: \"6c1388e8-16e7-4f78-b30e-033c4f9b5256\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6"
Sep 16 16:41:32.874513 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.874490    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c1388e8-16e7-4f78-b30e-033c4f9b5256-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-qkmb6\" (UID: \"6c1388e8-16e7-4f78-b30e-033c4f9b5256\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6"
Sep 16 16:41:32.876190 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.876169    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c1388e8-16e7-4f78-b30e-033c4f9b5256-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-qkmb6\" (UID: \"6c1388e8-16e7-4f78-b30e-033c4f9b5256\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6"
Sep 16 16:41:32.876279 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.876261    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/6c1388e8-16e7-4f78-b30e-033c4f9b5256-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-qkmb6\" (UID: \"6c1388e8-16e7-4f78-b30e-033c4f9b5256\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6"
Sep 16 16:41:32.886095 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.886077    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdcnj\" (UniqueName: \"kubernetes.io/projected/6c1388e8-16e7-4f78-b30e-033c4f9b5256-kube-api-access-jdcnj\") pod \"prometheus-operator-68bcd7f987-qkmb6\" (UID: \"6c1388e8-16e7-4f78-b30e-033c4f9b5256\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6"
Sep 16 16:41:32.910005 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:32.909975    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6"
Sep 16 16:41:33.176753 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:33.176576    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6"]
Sep 16 16:41:33.179820 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:41:33.179796    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1388e8_16e7_4f78_b30e_033c4f9b5256.slice/crio-e5856a01815ba9de45f9b127b2aed9f198ddc2b69b3855327607dde3f00017dd WatchSource:0}: Error finding container e5856a01815ba9de45f9b127b2aed9f198ddc2b69b3855327607dde3f00017dd: Status 404 returned error can't find the container with id e5856a01815ba9de45f9b127b2aed9f198ddc2b69b3855327607dde3f00017dd
Sep 16 16:41:33.750808 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:33.750767    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6" event={"ID":"6c1388e8-16e7-4f78-b30e-033c4f9b5256","Type":"ContainerStarted","Data":"e5856a01815ba9de45f9b127b2aed9f198ddc2b69b3855327607dde3f00017dd"}
Sep 16 16:41:33.844579 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:41:33.844539    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-hk8q9" podUID="c04662fe-4176-4a0b-ac5e-89ead5941975"
Sep 16 16:41:34.992542 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:34.992510    2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2" podUID="62ecfba4-b079-4546-bfe9-79950aeedea6" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 16:41:35.758813 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:35.758774    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6" event={"ID":"6c1388e8-16e7-4f78-b30e-033c4f9b5256","Type":"ContainerStarted","Data":"ab14aea7b120bc56997605372e64c3587a846d2ce49f266da3baa875afaf19a3"}
Sep 16 16:41:35.759033 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:35.758854    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6" event={"ID":"6c1388e8-16e7-4f78-b30e-033c4f9b5256","Type":"ContainerStarted","Data":"c80e000f0eaedf3d967bd406b15b68a6a6d8067fd027537b341d8f5da222c1c3"}
Sep 16 16:41:35.776021 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:35.775973    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-qkmb6" podStartSLOduration=1.9795198950000001 podStartE2EDuration="3.775959908s" totalImagesPullingTime="1.796440013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:41:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:41:35.774469305 +0000 UTC m=+295.536430752" watchObservedRunningTime="2026-09-16 16:41:35.775959908 +0000 UTC m=+295.537921378"
Sep 16 16:41:36.730069 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:36.730045    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z2tqs"
Sep 16 16:41:38.064737 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.064703    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5w9v5"]
Sep 16 16:41:38.067769 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.067752    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.073115 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.073090    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 16:41:38.073227 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.073111    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 16:41:38.073227 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.073188    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-284lx\""
Sep 16 16:41:38.073227 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.073206    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 16:41:38.209259 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.209231    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.209259 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.209263    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxmsp\" (UniqueName: \"kubernetes.io/projected/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-kube-api-access-qxmsp\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.209469 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.209295    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.209469 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.209321    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-node-exporter-textfile\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.209469 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.209336    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-sys\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.209469 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.209374    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-node-exporter-tls\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.209469 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.209387    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-root\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.209469 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.209403    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-metrics-client-ca\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.209469 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.209430    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-node-exporter-wtmp\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.309788 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.309758    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-node-exporter-tls\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.309788 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.309785    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-root\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.310028 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.309801    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-metrics-client-ca\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.310028 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.309821    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-node-exporter-wtmp\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.310028 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.309845    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.310028 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.309905    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-root\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.310028 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.309964    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxmsp\" (UniqueName: \"kubernetes.io/projected/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-kube-api-access-qxmsp\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.310028 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.309999    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-node-exporter-wtmp\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.310028 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.310018    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.310338 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.310046    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-node-exporter-textfile\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.310338 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.310073    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-sys\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.310338 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.310139    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-sys\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.310450 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.310390    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-node-exporter-textfile\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.310508 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.310461    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-metrics-client-ca\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.310508 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.310461    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-node-exporter-accelerators-collector-config\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.312171 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.312149    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-node-exporter-tls\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.312282 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.312224    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.318651 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.318597    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxmsp\" (UniqueName: \"kubernetes.io/projected/5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf-kube-api-access-qxmsp\") pod \"node-exporter-5w9v5\" (UID: \"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf\") " pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.376699 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.376678    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5w9v5"
Sep 16 16:41:38.387179 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:41:38.387153    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c1f9fe1_0c7f_4c63_b4c6_e155e5a079bf.slice/crio-f4a476316a6e9dc0900588bfd2e007e5e44e66ba5226c35578844a24efd9d6d3 WatchSource:0}: Error finding container f4a476316a6e9dc0900588bfd2e007e5e44e66ba5226c35578844a24efd9d6d3: Status 404 returned error can't find the container with id f4a476316a6e9dc0900588bfd2e007e5e44e66ba5226c35578844a24efd9d6d3
Sep 16 16:41:38.769604 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:38.769574    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5w9v5" event={"ID":"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf","Type":"ContainerStarted","Data":"f4a476316a6e9dc0900588bfd2e007e5e44e66ba5226c35578844a24efd9d6d3"}
Sep 16 16:41:39.774356 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:39.774323    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5w9v5" event={"ID":"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf","Type":"ContainerStarted","Data":"daa9ea3d1763391c293c89a280c653306fceaa229bb81d48135778e8e0e9ee44"}
Sep 16 16:41:40.615043 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:40.615013    2556 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 16:41:40.778505 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:40.778470    2556 generic.go:356] "Generic (PLEG): container finished" podID="5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf" containerID="daa9ea3d1763391c293c89a280c653306fceaa229bb81d48135778e8e0e9ee44" exitCode=0
Sep 16 16:41:40.780790 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:40.778522    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5w9v5" event={"ID":"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf","Type":"ContainerDied","Data":"daa9ea3d1763391c293c89a280c653306fceaa229bb81d48135778e8e0e9ee44"}
Sep 16 16:41:41.099738 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.099705    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-9c68594d-fmnln"]
Sep 16 16:41:41.103074 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.103056    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.105762 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.105690    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 16:41:41.105762 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.105696    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 16:41:41.105762 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.105743    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 16:41:41.105762 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.105755    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-mlrrb\""
Sep 16 16:41:41.106082 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.106057    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 16:41:41.106125 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.106102    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6oaatsdc64jiq\""
Sep 16 16:41:41.106174 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.106159    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 16:41:41.117044 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.117023    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9c68594d-fmnln"]
Sep 16 16:41:41.237859 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.237834    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dcfb\" (UniqueName: \"kubernetes.io/projected/707aeedc-ba66-41c1-a191-409f10661609-kube-api-access-7dcfb\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.238025 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.237864    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/707aeedc-ba66-41c1-a191-409f10661609-metrics-client-ca\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.238025 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.237883    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/707aeedc-ba66-41c1-a191-409f10661609-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.238025 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.237920    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/707aeedc-ba66-41c1-a191-409f10661609-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.238025 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.238014    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/707aeedc-ba66-41c1-a191-409f10661609-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.238231 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.238056    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/707aeedc-ba66-41c1-a191-409f10661609-secret-thanos-querier-tls\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.238231 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.238109    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/707aeedc-ba66-41c1-a191-409f10661609-secret-grpc-tls\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.238231 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.238139    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/707aeedc-ba66-41c1-a191-409f10661609-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.339134 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.339114    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/707aeedc-ba66-41c1-a191-409f10661609-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.339259 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.339143    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/707aeedc-ba66-41c1-a191-409f10661609-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.339259 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.339163    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/707aeedc-ba66-41c1-a191-409f10661609-secret-thanos-querier-tls\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.339259 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.339190    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/707aeedc-ba66-41c1-a191-409f10661609-secret-grpc-tls\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.339259 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.339209    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/707aeedc-ba66-41c1-a191-409f10661609-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.339447 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.339380    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dcfb\" (UniqueName: \"kubernetes.io/projected/707aeedc-ba66-41c1-a191-409f10661609-kube-api-access-7dcfb\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.339447 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.339422    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/707aeedc-ba66-41c1-a191-409f10661609-metrics-client-ca\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.339556 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.339451    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/707aeedc-ba66-41c1-a191-409f10661609-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.340336 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.340310    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/707aeedc-ba66-41c1-a191-409f10661609-metrics-client-ca\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.341833 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.341806    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/707aeedc-ba66-41c1-a191-409f10661609-secret-thanos-querier-tls\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.341927 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.341849    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/707aeedc-ba66-41c1-a191-409f10661609-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.342072 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.342054    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/707aeedc-ba66-41c1-a191-409f10661609-secret-grpc-tls\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.342196 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.342174    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/707aeedc-ba66-41c1-a191-409f10661609-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.342247 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.342217    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/707aeedc-ba66-41c1-a191-409f10661609-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.342247 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.342215    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/707aeedc-ba66-41c1-a191-409f10661609-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.349660 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.349636    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dcfb\" (UniqueName: \"kubernetes.io/projected/707aeedc-ba66-41c1-a191-409f10661609-kube-api-access-7dcfb\") pod \"thanos-querier-9c68594d-fmnln\" (UID: \"707aeedc-ba66-41c1-a191-409f10661609\") " pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.411891 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.411869    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:41.683489 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.683420    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-9c68594d-fmnln"]
Sep 16 16:41:41.686342 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:41:41.686317    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707aeedc_ba66_41c1_a191_409f10661609.slice/crio-216c21edca5027340f1919ea1a58f641faa7e63f7b570308687fa604b97c1ab0 WatchSource:0}: Error finding container 216c21edca5027340f1919ea1a58f641faa7e63f7b570308687fa604b97c1ab0: Status 404 returned error can't find the container with id 216c21edca5027340f1919ea1a58f641faa7e63f7b570308687fa604b97c1ab0
Sep 16 16:41:41.783172 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.783137    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9c68594d-fmnln" event={"ID":"707aeedc-ba66-41c1-a191-409f10661609","Type":"ContainerStarted","Data":"216c21edca5027340f1919ea1a58f641faa7e63f7b570308687fa604b97c1ab0"}
Sep 16 16:41:41.785132 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.785100    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5w9v5" event={"ID":"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf","Type":"ContainerStarted","Data":"7075f228767eb998d5c4c4c8cd2d83bb60399d7e8faf772f1623a988418bea43"}
Sep 16 16:41:41.785238 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.785149    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5w9v5" event={"ID":"5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf","Type":"ContainerStarted","Data":"b836f608567c36585eb3eb22b339bd64579ee9ca2cfee325c7e62fb1e53d5d63"}
Sep 16 16:41:41.806348 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:41.806303    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5w9v5" podStartSLOduration=2.479081615 podStartE2EDuration="3.80628856s" totalImagesPullingTime="1.266461306s" totalInitContainerRuntime="60.745639ms" isStatefulPod=true podCreationTimestamp="2026-09-16 16:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:41:41.80470501 +0000 UTC m=+301.566666457" watchObservedRunningTime="2026-09-16 16:41:41.80628856 +0000 UTC m=+301.568250006"
Sep 16 16:41:42.732891 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:42.732727    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-xg2dr"]
Sep 16 16:41:42.736976 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:42.736951    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-xg2dr"
Sep 16 16:41:42.739122 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:42.739100    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4c7wq\""
Sep 16 16:41:42.739238 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:42.739098    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 16:41:42.746107 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:42.743814    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-xg2dr"]
Sep 16 16:41:42.850127 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:42.850090    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b4d8056-88a6-4900-889a-ef4abb61518a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-xg2dr\" (UID: \"5b4d8056-88a6-4900-889a-ef4abb61518a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-xg2dr"
Sep 16 16:41:42.850514 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:42.850137    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56pch\" (UniqueName: \"kubernetes.io/projected/5b4d8056-88a6-4900-889a-ef4abb61518a-kube-api-access-56pch\") pod \"monitoring-plugin-5d85778b9d-xg2dr\" (UID: \"5b4d8056-88a6-4900-889a-ef4abb61518a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-xg2dr"
Sep 16 16:41:42.950926 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:42.950885    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b4d8056-88a6-4900-889a-ef4abb61518a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-xg2dr\" (UID: \"5b4d8056-88a6-4900-889a-ef4abb61518a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-xg2dr"
Sep 16 16:41:42.951121 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:42.950960    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56pch\" (UniqueName: \"kubernetes.io/projected/5b4d8056-88a6-4900-889a-ef4abb61518a-kube-api-access-56pch\") pod \"monitoring-plugin-5d85778b9d-xg2dr\" (UID: \"5b4d8056-88a6-4900-889a-ef4abb61518a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-xg2dr"
Sep 16 16:41:42.951121 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:41:42.951019    2556 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 16 16:41:42.951121 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:41:42.951091    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b4d8056-88a6-4900-889a-ef4abb61518a-monitoring-plugin-cert podName:5b4d8056-88a6-4900-889a-ef4abb61518a nodeName:}" failed. No retries permitted until 2026-09-16 16:41:43.451072114 +0000 UTC m=+303.213033540 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/5b4d8056-88a6-4900-889a-ef4abb61518a-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-xg2dr" (UID: "5b4d8056-88a6-4900-889a-ef4abb61518a") : secret "monitoring-plugin-cert" not found
Sep 16 16:41:42.961431 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:42.961396    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56pch\" (UniqueName: \"kubernetes.io/projected/5b4d8056-88a6-4900-889a-ef4abb61518a-kube-api-access-56pch\") pod \"monitoring-plugin-5d85778b9d-xg2dr\" (UID: \"5b4d8056-88a6-4900-889a-ef4abb61518a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-xg2dr"
Sep 16 16:41:43.452917 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:43.452876    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b4d8056-88a6-4900-889a-ef4abb61518a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-xg2dr\" (UID: \"5b4d8056-88a6-4900-889a-ef4abb61518a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-xg2dr"
Sep 16 16:41:43.455671 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:43.455646    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5b4d8056-88a6-4900-889a-ef4abb61518a-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-xg2dr\" (UID: \"5b4d8056-88a6-4900-889a-ef4abb61518a\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-xg2dr"
Sep 16 16:41:43.652646 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:43.652626    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-xg2dr"
Sep 16 16:41:43.751991 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:43.751971    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 16:41:43.793983 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:43.793953    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9c68594d-fmnln" event={"ID":"707aeedc-ba66-41c1-a191-409f10661609","Type":"ContainerStarted","Data":"cfbb724cf3c52fa8fb8ae3fa5bd4e5697bd562f1aefd309ce56ad5f441ab9c5d"}
Sep 16 16:41:43.794113 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:43.794013    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9c68594d-fmnln" event={"ID":"707aeedc-ba66-41c1-a191-409f10661609","Type":"ContainerStarted","Data":"25329519568a00bb508e47a4a09161953cc2697dad4dd2176136322ac9441937"}
Sep 16 16:41:43.794113 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:43.794039    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9c68594d-fmnln" event={"ID":"707aeedc-ba66-41c1-a191-409f10661609","Type":"ContainerStarted","Data":"022a2e3e19166e4bbb164e3d8e8f7fe789b9be3aff2835f7aba299df969e777c"}
Sep 16 16:41:43.926517 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:43.926494    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-xg2dr"]
Sep 16 16:41:43.928693 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:41:43.928667    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b4d8056_88a6_4900_889a_ef4abb61518a.slice/crio-7fba9bf72b174121a0d44efcbc1b9935fb3dcd002618e0d951fc6b88a931acad WatchSource:0}: Error finding container 7fba9bf72b174121a0d44efcbc1b9935fb3dcd002618e0d951fc6b88a931acad: Status 404 returned error can't find the container with id 7fba9bf72b174121a0d44efcbc1b9935fb3dcd002618e0d951fc6b88a931acad
Sep 16 16:41:44.799636 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:44.799602    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-xg2dr" event={"ID":"5b4d8056-88a6-4900-889a-ef4abb61518a","Type":"ContainerStarted","Data":"7fba9bf72b174121a0d44efcbc1b9935fb3dcd002618e0d951fc6b88a931acad"}
Sep 16 16:41:44.803266 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:44.803240    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9c68594d-fmnln" event={"ID":"707aeedc-ba66-41c1-a191-409f10661609","Type":"ContainerStarted","Data":"7efbfa0c7bb23ee4c899216b0ec5c2926812367847b653ecb570107297b0cb38"}
Sep 16 16:41:44.993021 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:44.992976    2556 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2" podUID="62ecfba4-b079-4546-bfe9-79950aeedea6" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 16 16:41:44.993396 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:44.993041    2556 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2"
Sep 16 16:41:44.993622 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:44.993584    2556 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"5332901a26855a04e012a023f728c187d435fcfefcde91d2220b0b23ff9c1232"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 16 16:41:44.993660 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:44.993643    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2" podUID="62ecfba4-b079-4546-bfe9-79950aeedea6" containerName="service-proxy" containerID="cri-o://5332901a26855a04e012a023f728c187d435fcfefcde91d2220b0b23ff9c1232" gracePeriod=30
Sep 16 16:41:45.808642 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:45.808605    2556 generic.go:356] "Generic (PLEG): container finished" podID="62ecfba4-b079-4546-bfe9-79950aeedea6" containerID="5332901a26855a04e012a023f728c187d435fcfefcde91d2220b0b23ff9c1232" exitCode=2
Sep 16 16:41:45.808804 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:45.808679    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2" event={"ID":"62ecfba4-b079-4546-bfe9-79950aeedea6","Type":"ContainerDied","Data":"5332901a26855a04e012a023f728c187d435fcfefcde91d2220b0b23ff9c1232"}
Sep 16 16:41:45.808804 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:45.808726    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56ff7c4fc4-sndw2" event={"ID":"62ecfba4-b079-4546-bfe9-79950aeedea6","Type":"ContainerStarted","Data":"217db5ebbce780f6011366aa663f62564bb2928f5fd004d5b8a8f2232db4234a"}
Sep 16 16:41:45.810008 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:45.809989    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-xg2dr" event={"ID":"5b4d8056-88a6-4900-889a-ef4abb61518a","Type":"ContainerStarted","Data":"967c4bd08ef4b7825ddc12a071ab8e1673772871e31a54f8edf731f2e643df50"}
Sep 16 16:41:45.810115 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:45.810025    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-xg2dr"
Sep 16 16:41:45.813453 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:45.813428    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9c68594d-fmnln" event={"ID":"707aeedc-ba66-41c1-a191-409f10661609","Type":"ContainerStarted","Data":"b321e28c7e285b3ba80d4deda0f92b8750f17e446124637c0ef159f98a1b3f8c"}
Sep 16 16:41:45.813561 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:45.813479    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-9c68594d-fmnln" event={"ID":"707aeedc-ba66-41c1-a191-409f10661609","Type":"ContainerStarted","Data":"0225e828474bb31bf3ae102eeda95f2bd8751e2e421ddf60849c7e0ef1847e73"}
Sep 16 16:41:45.813614 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:45.813574    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:41:45.815423 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:45.815403    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-xg2dr"
Sep 16 16:41:45.848789 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:45.848749    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-9c68594d-fmnln" podStartSLOduration=2.028597512 podStartE2EDuration="4.848736278s" totalImagesPullingTime="2.820138766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:41:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:41:45.847791549 +0000 UTC m=+305.609752999" watchObservedRunningTime="2026-09-16 16:41:45.848736278 +0000 UTC m=+305.610697735"
Sep 16 16:41:45.863198 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:45.863161    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-xg2dr" podStartSLOduration=2.3944676 podStartE2EDuration="3.863150321s" totalImagesPullingTime="1.468682721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:41:45.86229595 +0000 UTC m=+305.624257400" watchObservedRunningTime="2026-09-16 16:41:45.863150321 +0000 UTC m=+305.625111767"
Sep 16 16:41:48.626993 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:48.626958    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-qdq8l"]
Sep 16 16:41:48.630003 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:48.629988    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-qdq8l"
Sep 16 16:41:48.632283 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:48.632264    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 16:41:48.632364 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:48.632278    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 16:41:48.632364 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:48.632329    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-56jfs\""
Sep 16 16:41:48.640931 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:48.640909    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-qdq8l"]
Sep 16 16:41:48.696839 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:48.696815    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbwjf\" (UniqueName: \"kubernetes.io/projected/ccb79307-8c26-492a-8c6f-ce741345a6be-kube-api-access-fbwjf\") pod \"downloads-76f99584f4-qdq8l\" (UID: \"ccb79307-8c26-492a-8c6f-ce741345a6be\") " pod="openshift-console/downloads-76f99584f4-qdq8l"
Sep 16 16:41:48.797614 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:48.797586    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbwjf\" (UniqueName: \"kubernetes.io/projected/ccb79307-8c26-492a-8c6f-ce741345a6be-kube-api-access-fbwjf\") pod \"downloads-76f99584f4-qdq8l\" (UID: \"ccb79307-8c26-492a-8c6f-ce741345a6be\") " pod="openshift-console/downloads-76f99584f4-qdq8l"
Sep 16 16:41:48.805783 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:48.805757    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbwjf\" (UniqueName: \"kubernetes.io/projected/ccb79307-8c26-492a-8c6f-ce741345a6be-kube-api-access-fbwjf\") pod \"downloads-76f99584f4-qdq8l\" (UID: \"ccb79307-8c26-492a-8c6f-ce741345a6be\") " pod="openshift-console/downloads-76f99584f4-qdq8l"
Sep 16 16:41:48.842933 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:48.842899    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:41:48.939015 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:48.938995    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-qdq8l"
Sep 16 16:41:49.212243 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:49.212130    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-qdq8l"]
Sep 16 16:41:49.215409 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:41:49.215385    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb79307_8c26_492a_8c6f_ce741345a6be.slice/crio-9277f3ad4b667854e9a9635748c20f55891b75be60f87c4e402c5b862522e256 WatchSource:0}: Error finding container 9277f3ad4b667854e9a9635748c20f55891b75be60f87c4e402c5b862522e256: Status 404 returned error can't find the container with id 9277f3ad4b667854e9a9635748c20f55891b75be60f87c4e402c5b862522e256
Sep 16 16:41:49.828245 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:49.828209    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-qdq8l" event={"ID":"ccb79307-8c26-492a-8c6f-ce741345a6be","Type":"ContainerStarted","Data":"9277f3ad4b667854e9a9635748c20f55891b75be60f87c4e402c5b862522e256"}
Sep 16 16:41:51.826062 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:41:51.826030    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-9c68594d-fmnln"
Sep 16 16:42:06.895834 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:42:06.895783    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-qdq8l" event={"ID":"ccb79307-8c26-492a-8c6f-ce741345a6be","Type":"ContainerStarted","Data":"a8e07ed7b80560614cf493f9132a4f764b93c8ce130e7b17e3aac623eb8c5258"}
Sep 16 16:42:06.896327 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:42:06.895971    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-qdq8l"
Sep 16 16:42:06.915021 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:42:06.914974    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-qdq8l" podStartSLOduration=2.078631272 podStartE2EDuration="18.914955776s" totalImagesPullingTime="16.836324504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:41:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:42:06.913235733 +0000 UTC m=+326.675197185" watchObservedRunningTime="2026-09-16 16:42:06.914955776 +0000 UTC m=+326.676917334"
Sep 16 16:42:06.915696 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:42:06.915666    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-qdq8l"
Sep 16 16:42:54.494688 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:42:54.494604    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs\") pod \"network-metrics-daemon-hk8q9\" (UID: \"c04662fe-4176-4a0b-ac5e-89ead5941975\") " pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:42:54.496869 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:42:54.496850    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c04662fe-4176-4a0b-ac5e-89ead5941975-metrics-certs\") pod \"network-metrics-daemon-hk8q9\" (UID: \"c04662fe-4176-4a0b-ac5e-89ead5941975\") " pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:42:54.546498 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:42:54.546469    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v6lts\""
Sep 16 16:42:54.554281 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:42:54.554263    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hk8q9"
Sep 16 16:42:54.820817 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:42:54.820728    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hk8q9"]
Sep 16 16:42:54.823201 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:42:54.823167    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04662fe_4176_4a0b_ac5e_89ead5941975.slice/crio-593a422c272d7ec18e347885793a6a5fede3185f3b3a09720ba458b80b6388ad WatchSource:0}: Error finding container 593a422c272d7ec18e347885793a6a5fede3185f3b3a09720ba458b80b6388ad: Status 404 returned error can't find the container with id 593a422c272d7ec18e347885793a6a5fede3185f3b3a09720ba458b80b6388ad
Sep 16 16:42:55.082903 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:42:55.082827    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hk8q9" event={"ID":"c04662fe-4176-4a0b-ac5e-89ead5941975","Type":"ContainerStarted","Data":"593a422c272d7ec18e347885793a6a5fede3185f3b3a09720ba458b80b6388ad"}
Sep 16 16:42:56.088125 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:42:56.088042    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hk8q9" event={"ID":"c04662fe-4176-4a0b-ac5e-89ead5941975","Type":"ContainerStarted","Data":"25d81c44ad88fe71bbf246ea2b725013fdf361a9b740f25a80b1c3beb1a0e3b0"}
Sep 16 16:42:56.088125 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:42:56.088101    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hk8q9" event={"ID":"c04662fe-4176-4a0b-ac5e-89ead5941975","Type":"ContainerStarted","Data":"149196b7021e56a1df4524726af57055cde59027dd19a49e09f705bd7a6985c0"}
Sep 16 16:42:56.102658 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:42:56.102612    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hk8q9" podStartSLOduration=374.185952226 podStartE2EDuration="6m15.102598013s" totalImagesPullingTime="916.645787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:42:56.1020815 +0000 UTC m=+375.864042947" watchObservedRunningTime="2026-09-16 16:42:56.102598013 +0000 UTC m=+375.864559501"
Sep 16 16:44:36.779300 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:44:36.779267    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-shglh"]
Sep 16 16:44:36.781200 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:44:36.781184    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-shglh"
Sep 16 16:44:36.784701 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:44:36.784670    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 16:44:36.784901 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:44:36.784887    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 16:44:36.790412 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:44:36.790394    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-whmvp\""
Sep 16 16:44:36.809704 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:44:36.809683    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-shglh"]
Sep 16 16:44:36.883597 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:44:36.883573    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qhcn\" (UniqueName: \"kubernetes.io/projected/67974f43-8a10-48be-ac1c-d7eca4d1b31b-kube-api-access-5qhcn\") pod \"cert-manager-webhook-6486ffd997-shglh\" (UID: \"67974f43-8a10-48be-ac1c-d7eca4d1b31b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-shglh"
Sep 16 16:44:36.883741 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:44:36.883611    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67974f43-8a10-48be-ac1c-d7eca4d1b31b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-shglh\" (UID: \"67974f43-8a10-48be-ac1c-d7eca4d1b31b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-shglh"
Sep 16 16:44:36.984299 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:44:36.984264    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67974f43-8a10-48be-ac1c-d7eca4d1b31b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-shglh\" (UID: \"67974f43-8a10-48be-ac1c-d7eca4d1b31b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-shglh"
Sep 16 16:44:36.984462 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:44:36.984362    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qhcn\" (UniqueName: \"kubernetes.io/projected/67974f43-8a10-48be-ac1c-d7eca4d1b31b-kube-api-access-5qhcn\") pod \"cert-manager-webhook-6486ffd997-shglh\" (UID: \"67974f43-8a10-48be-ac1c-d7eca4d1b31b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-shglh"
Sep 16 16:44:37.002651 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:44:37.002612    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/67974f43-8a10-48be-ac1c-d7eca4d1b31b-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-shglh\" (UID: \"67974f43-8a10-48be-ac1c-d7eca4d1b31b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-shglh"
Sep 16 16:44:37.003146 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:44:37.003126    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qhcn\" (UniqueName: \"kubernetes.io/projected/67974f43-8a10-48be-ac1c-d7eca4d1b31b-kube-api-access-5qhcn\") pod \"cert-manager-webhook-6486ffd997-shglh\" (UID: \"67974f43-8a10-48be-ac1c-d7eca4d1b31b\") " pod="cert-manager/cert-manager-webhook-6486ffd997-shglh"
Sep 16 16:44:37.090134 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:44:37.090066    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-shglh"
Sep 16 16:44:37.364376 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:44:37.364293    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-shglh"]
Sep 16 16:44:37.367096 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:44:37.367068    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67974f43_8a10_48be_ac1c_d7eca4d1b31b.slice/crio-81ed4e67cbffa0b062f2e058f6ddba33a09b0b6c5a2749811cd2c80ada1f9f1b WatchSource:0}: Error finding container 81ed4e67cbffa0b062f2e058f6ddba33a09b0b6c5a2749811cd2c80ada1f9f1b: Status 404 returned error can't find the container with id 81ed4e67cbffa0b062f2e058f6ddba33a09b0b6c5a2749811cd2c80ada1f9f1b
Sep 16 16:44:37.443951 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:44:37.443909    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-shglh" event={"ID":"67974f43-8a10-48be-ac1c-d7eca4d1b31b","Type":"ContainerStarted","Data":"81ed4e67cbffa0b062f2e058f6ddba33a09b0b6c5a2749811cd2c80ada1f9f1b"}
Sep 16 16:44:43.468818 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:44:43.468781    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-shglh" event={"ID":"67974f43-8a10-48be-ac1c-d7eca4d1b31b","Type":"ContainerStarted","Data":"203b0b1cde1a0f153b2b029e378e15b54ebcb867a2a0f01610a028807d4c3ce5"}
Sep 16 16:44:43.469199 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:44:43.468918    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-shglh"
Sep 16 16:44:43.489274 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:44:43.489228    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-shglh" podStartSLOduration=1.5243552070000002 podStartE2EDuration="7.48921497s" totalImagesPullingTime="5.964859763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:44:43.48828108 +0000 UTC m=+483.250242527" watchObservedRunningTime="2026-09-16 16:44:43.48921497 +0000 UTC m=+483.251176417"
Sep 16 16:44:49.476000 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:44:49.475961    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-shglh"
Sep 16 16:45:27.561357 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.561322    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"]
Sep 16 16:45:27.565927 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.565907    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"
Sep 16 16:45:27.568157 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.568121    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 16 16:45:27.568290 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.568164    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mhjfp\""
Sep 16 16:45:27.568771 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.568750    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 16 16:45:27.569076 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.569046    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 16 16:45:27.570501 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.570483    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 16 16:45:27.594277 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.594254    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"]
Sep 16 16:45:27.745964 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.745898    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-54d7c7f658-2rwx4\" (UID: \"6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0\") " pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"
Sep 16 16:45:27.745964 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.745973    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0-manifests\") pod \"opendatahub-operator-controller-manager-54d7c7f658-2rwx4\" (UID: \"6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0\") " pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"
Sep 16 16:45:27.746159 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.745996    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0-tmp\") pod \"opendatahub-operator-controller-manager-54d7c7f658-2rwx4\" (UID: \"6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0\") " pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"
Sep 16 16:45:27.746159 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.746016    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjxvg\" (UniqueName: \"kubernetes.io/projected/6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0-kube-api-access-qjxvg\") pod \"opendatahub-operator-controller-manager-54d7c7f658-2rwx4\" (UID: \"6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0\") " pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"
Sep 16 16:45:27.746159 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.746080    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54d7c7f658-2rwx4\" (UID: \"6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0\") " pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"
Sep 16 16:45:27.847137 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.847069    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0-manifests\") pod \"opendatahub-operator-controller-manager-54d7c7f658-2rwx4\" (UID: \"6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0\") " pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"
Sep 16 16:45:27.847137 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.847100    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0-tmp\") pod \"opendatahub-operator-controller-manager-54d7c7f658-2rwx4\" (UID: \"6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0\") " pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"
Sep 16 16:45:27.847137 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.847121    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjxvg\" (UniqueName: \"kubernetes.io/projected/6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0-kube-api-access-qjxvg\") pod \"opendatahub-operator-controller-manager-54d7c7f658-2rwx4\" (UID: \"6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0\") " pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"
Sep 16 16:45:27.847367 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.847162    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54d7c7f658-2rwx4\" (UID: \"6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0\") " pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"
Sep 16 16:45:27.847367 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.847208    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-54d7c7f658-2rwx4\" (UID: \"6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0\") " pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"
Sep 16 16:45:27.847545 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.847523    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0-manifests\") pod \"opendatahub-operator-controller-manager-54d7c7f658-2rwx4\" (UID: \"6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0\") " pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"
Sep 16 16:45:27.849468 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.849440    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0-tmp\") pod \"opendatahub-operator-controller-manager-54d7c7f658-2rwx4\" (UID: \"6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0\") " pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"
Sep 16 16:45:27.849569 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.849502    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54d7c7f658-2rwx4\" (UID: \"6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0\") " pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"
Sep 16 16:45:27.849617 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.849589    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0-webhook-cert\") pod \"opendatahub-operator-controller-manager-54d7c7f658-2rwx4\" (UID: \"6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0\") " pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"
Sep 16 16:45:27.855469 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.855445    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjxvg\" (UniqueName: \"kubernetes.io/projected/6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0-kube-api-access-qjxvg\") pod \"opendatahub-operator-controller-manager-54d7c7f658-2rwx4\" (UID: \"6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0\") " pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"
Sep 16 16:45:27.876076 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:27.876056    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"
Sep 16 16:45:28.156192 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:28.155981    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"]
Sep 16 16:45:28.158884 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:45:28.158856    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7e8dc3_46fe_4fc2_98fa_7a2b25ec46e0.slice/crio-d3fe1ca423e287a36aa269e5b31481e28aabdd437456c5c9073ee8fce839889c WatchSource:0}: Error finding container d3fe1ca423e287a36aa269e5b31481e28aabdd437456c5c9073ee8fce839889c: Status 404 returned error can't find the container with id d3fe1ca423e287a36aa269e5b31481e28aabdd437456c5c9073ee8fce839889c
Sep 16 16:45:28.631574 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:28.631533    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4" event={"ID":"6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0","Type":"ContainerStarted","Data":"d3fe1ca423e287a36aa269e5b31481e28aabdd437456c5c9073ee8fce839889c"}
Sep 16 16:45:31.645821 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:31.645782    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4" event={"ID":"6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0","Type":"ContainerStarted","Data":"7b5df9664381f57b0d652cdbd60a1a7b0b686a044dd29ff27c436a5751c9b2ad"}
Sep 16 16:45:32.651087 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:32.651053    2556 generic.go:356] "Generic (PLEG): container finished" podID="6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0" containerID="7b5df9664381f57b0d652cdbd60a1a7b0b686a044dd29ff27c436a5751c9b2ad" exitCode=0
Sep 16 16:45:32.651488 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:32.651101    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4" event={"ID":"6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0","Type":"ContainerDied","Data":"7b5df9664381f57b0d652cdbd60a1a7b0b686a044dd29ff27c436a5751c9b2ad"}
Sep 16 16:45:33.657818 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:33.657778    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4" event={"ID":"6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0","Type":"ContainerStarted","Data":"ce198a3b7b19e8c51643d92147adddb03b0a8d90a4d69da35247fd545d82be21"}
Sep 16 16:45:33.658225 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:33.657885    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"
Sep 16 16:45:33.712461 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:33.712409    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4" podStartSLOduration=2.332152319 podStartE2EDuration="6.712392311s" totalImagesPullingTime="3.372735001s" totalInitContainerRuntime="1.007504991s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:45:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:45:33.704831559 +0000 UTC m=+533.466793005" watchObservedRunningTime="2026-09-16 16:45:33.712392311 +0000 UTC m=+533.474353758"
Sep 16 16:45:44.663428 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:44.663396    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-54d7c7f658-2rwx4"
Sep 16 16:45:57.824338 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:57.824308    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rsbqk"]
Sep 16 16:45:57.832740 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:57.832714    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rsbqk"
Sep 16 16:45:57.835418 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:57.835394    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-7trt5\""
Sep 16 16:45:57.835737 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:57.835722    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 16 16:45:57.842719 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:57.842696    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rsbqk"]
Sep 16 16:45:57.849543 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:57.849522    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8775n\" (UniqueName: \"kubernetes.io/projected/333d82ae-12bc-47a8-8a8a-f6d4621b53a2-kube-api-access-8775n\") pod \"odh-model-controller-7998c67dc4-rsbqk\" (UID: \"333d82ae-12bc-47a8-8a8a-f6d4621b53a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-rsbqk"
Sep 16 16:45:57.849637 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:57.849566    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/333d82ae-12bc-47a8-8a8a-f6d4621b53a2-cert\") pod \"odh-model-controller-7998c67dc4-rsbqk\" (UID: \"333d82ae-12bc-47a8-8a8a-f6d4621b53a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-rsbqk"
Sep 16 16:45:57.950694 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:57.950659    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/333d82ae-12bc-47a8-8a8a-f6d4621b53a2-cert\") pod \"odh-model-controller-7998c67dc4-rsbqk\" (UID: \"333d82ae-12bc-47a8-8a8a-f6d4621b53a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-rsbqk"
Sep 16 16:45:57.950856 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:57.950721    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8775n\" (UniqueName: \"kubernetes.io/projected/333d82ae-12bc-47a8-8a8a-f6d4621b53a2-kube-api-access-8775n\") pod \"odh-model-controller-7998c67dc4-rsbqk\" (UID: \"333d82ae-12bc-47a8-8a8a-f6d4621b53a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-rsbqk"
Sep 16 16:45:57.950856 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:45:57.950815    2556 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 16 16:45:57.951014 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:45:57.950891    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/333d82ae-12bc-47a8-8a8a-f6d4621b53a2-cert podName:333d82ae-12bc-47a8-8a8a-f6d4621b53a2 nodeName:}" failed. No retries permitted until 2026-09-16 16:45:58.450873213 +0000 UTC m=+558.212834642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/333d82ae-12bc-47a8-8a8a-f6d4621b53a2-cert") pod "odh-model-controller-7998c67dc4-rsbqk" (UID: "333d82ae-12bc-47a8-8a8a-f6d4621b53a2") : secret "odh-model-controller-webhook-cert" not found
Sep 16 16:45:57.962105 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:57.962075    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8775n\" (UniqueName: \"kubernetes.io/projected/333d82ae-12bc-47a8-8a8a-f6d4621b53a2-kube-api-access-8775n\") pod \"odh-model-controller-7998c67dc4-rsbqk\" (UID: \"333d82ae-12bc-47a8-8a8a-f6d4621b53a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-rsbqk"
Sep 16 16:45:58.136177 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.136097    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b966d4c98-vw7ts"]
Sep 16 16:45:58.138545 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.138529    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b966d4c98-vw7ts"
Sep 16 16:45:58.140979 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.140935    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 16 16:45:58.141156 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.141014    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 16:45:58.141156 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.141146    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 16:45:58.141304 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.141229    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 16 16:45:58.141356 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.141337    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-kqsnz\""
Sep 16 16:45:58.150883 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.150861    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b966d4c98-vw7ts"]
Sep 16 16:45:58.253353 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.253319    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f27d87e1-4421-46fb-812b-f159f43dc302-tls-certs\") pod \"kube-auth-proxy-b966d4c98-vw7ts\" (UID: \"f27d87e1-4421-46fb-812b-f159f43dc302\") " pod="openshift-ingress/kube-auth-proxy-b966d4c98-vw7ts"
Sep 16 16:45:58.253533 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.253372    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f27d87e1-4421-46fb-812b-f159f43dc302-tmp\") pod \"kube-auth-proxy-b966d4c98-vw7ts\" (UID: \"f27d87e1-4421-46fb-812b-f159f43dc302\") " pod="openshift-ingress/kube-auth-proxy-b966d4c98-vw7ts"
Sep 16 16:45:58.253533 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.253487    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8sxt\" (UniqueName: \"kubernetes.io/projected/f27d87e1-4421-46fb-812b-f159f43dc302-kube-api-access-z8sxt\") pod \"kube-auth-proxy-b966d4c98-vw7ts\" (UID: \"f27d87e1-4421-46fb-812b-f159f43dc302\") " pod="openshift-ingress/kube-auth-proxy-b966d4c98-vw7ts"
Sep 16 16:45:58.354759 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.354719    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8sxt\" (UniqueName: \"kubernetes.io/projected/f27d87e1-4421-46fb-812b-f159f43dc302-kube-api-access-z8sxt\") pod \"kube-auth-proxy-b966d4c98-vw7ts\" (UID: \"f27d87e1-4421-46fb-812b-f159f43dc302\") " pod="openshift-ingress/kube-auth-proxy-b966d4c98-vw7ts"
Sep 16 16:45:58.354909 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.354796    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f27d87e1-4421-46fb-812b-f159f43dc302-tls-certs\") pod \"kube-auth-proxy-b966d4c98-vw7ts\" (UID: \"f27d87e1-4421-46fb-812b-f159f43dc302\") " pod="openshift-ingress/kube-auth-proxy-b966d4c98-vw7ts"
Sep 16 16:45:58.354909 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.354857    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f27d87e1-4421-46fb-812b-f159f43dc302-tmp\") pod \"kube-auth-proxy-b966d4c98-vw7ts\" (UID: \"f27d87e1-4421-46fb-812b-f159f43dc302\") " pod="openshift-ingress/kube-auth-proxy-b966d4c98-vw7ts"
Sep 16 16:45:58.355061 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:45:58.354928    2556 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 16 16:45:58.355061 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:45:58.355017    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f27d87e1-4421-46fb-812b-f159f43dc302-tls-certs podName:f27d87e1-4421-46fb-812b-f159f43dc302 nodeName:}" failed. No retries permitted until 2026-09-16 16:45:58.855001748 +0000 UTC m=+558.616963173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f27d87e1-4421-46fb-812b-f159f43dc302-tls-certs") pod "kube-auth-proxy-b966d4c98-vw7ts" (UID: "f27d87e1-4421-46fb-812b-f159f43dc302") : secret "kube-auth-proxy-tls" not found
Sep 16 16:45:58.357183 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.357157    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f27d87e1-4421-46fb-812b-f159f43dc302-tmp\") pod \"kube-auth-proxy-b966d4c98-vw7ts\" (UID: \"f27d87e1-4421-46fb-812b-f159f43dc302\") " pod="openshift-ingress/kube-auth-proxy-b966d4c98-vw7ts"
Sep 16 16:45:58.363324 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.363306    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8sxt\" (UniqueName: \"kubernetes.io/projected/f27d87e1-4421-46fb-812b-f159f43dc302-kube-api-access-z8sxt\") pod \"kube-auth-proxy-b966d4c98-vw7ts\" (UID: \"f27d87e1-4421-46fb-812b-f159f43dc302\") " pod="openshift-ingress/kube-auth-proxy-b966d4c98-vw7ts"
Sep 16 16:45:58.456005 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.455979    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/333d82ae-12bc-47a8-8a8a-f6d4621b53a2-cert\") pod \"odh-model-controller-7998c67dc4-rsbqk\" (UID: \"333d82ae-12bc-47a8-8a8a-f6d4621b53a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-rsbqk"
Sep 16 16:45:58.458173 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.458149    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/333d82ae-12bc-47a8-8a8a-f6d4621b53a2-cert\") pod \"odh-model-controller-7998c67dc4-rsbqk\" (UID: \"333d82ae-12bc-47a8-8a8a-f6d4621b53a2\") " pod="opendatahub/odh-model-controller-7998c67dc4-rsbqk"
Sep 16 16:45:58.619785 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.619747    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vdj6m"]
Sep 16 16:45:58.622868 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.622853    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vdj6m"
Sep 16 16:45:58.625381 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.625360    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 16 16:45:58.625665 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.625646    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-zj9kp\""
Sep 16 16:45:58.634458 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.634435    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vdj6m"]
Sep 16 16:45:58.658218 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.658192    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h47l\" (UniqueName: \"kubernetes.io/projected/9a522445-b1b9-4528-b4bf-5144cd3aae87-kube-api-access-8h47l\") pod \"model-serving-api-54d89bd5fc-vdj6m\" (UID: \"9a522445-b1b9-4528-b4bf-5144cd3aae87\") " pod="opendatahub/model-serving-api-54d89bd5fc-vdj6m"
Sep 16 16:45:58.658352 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.658229    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a522445-b1b9-4528-b4bf-5144cd3aae87-tls-certs\") pod \"model-serving-api-54d89bd5fc-vdj6m\" (UID: \"9a522445-b1b9-4528-b4bf-5144cd3aae87\") " pod="opendatahub/model-serving-api-54d89bd5fc-vdj6m"
Sep 16 16:45:58.743382 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.743318    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rsbqk"
Sep 16 16:45:58.758706 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.758681    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h47l\" (UniqueName: \"kubernetes.io/projected/9a522445-b1b9-4528-b4bf-5144cd3aae87-kube-api-access-8h47l\") pod \"model-serving-api-54d89bd5fc-vdj6m\" (UID: \"9a522445-b1b9-4528-b4bf-5144cd3aae87\") " pod="opendatahub/model-serving-api-54d89bd5fc-vdj6m"
Sep 16 16:45:58.758804 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.758722    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a522445-b1b9-4528-b4bf-5144cd3aae87-tls-certs\") pod \"model-serving-api-54d89bd5fc-vdj6m\" (UID: \"9a522445-b1b9-4528-b4bf-5144cd3aae87\") " pod="opendatahub/model-serving-api-54d89bd5fc-vdj6m"
Sep 16 16:45:58.758849 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:45:58.758838    2556 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 16 16:45:58.758907 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:45:58.758896    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a522445-b1b9-4528-b4bf-5144cd3aae87-tls-certs podName:9a522445-b1b9-4528-b4bf-5144cd3aae87 nodeName:}" failed. No retries permitted until 2026-09-16 16:45:59.258876499 +0000 UTC m=+559.020837924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/9a522445-b1b9-4528-b4bf-5144cd3aae87-tls-certs") pod "model-serving-api-54d89bd5fc-vdj6m" (UID: "9a522445-b1b9-4528-b4bf-5144cd3aae87") : secret "model-serving-api-tls" not found
Sep 16 16:45:58.769266 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.769244    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h47l\" (UniqueName: \"kubernetes.io/projected/9a522445-b1b9-4528-b4bf-5144cd3aae87-kube-api-access-8h47l\") pod \"model-serving-api-54d89bd5fc-vdj6m\" (UID: \"9a522445-b1b9-4528-b4bf-5144cd3aae87\") " pod="opendatahub/model-serving-api-54d89bd5fc-vdj6m"
Sep 16 16:45:58.859555 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.859528    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f27d87e1-4421-46fb-812b-f159f43dc302-tls-certs\") pod \"kube-auth-proxy-b966d4c98-vw7ts\" (UID: \"f27d87e1-4421-46fb-812b-f159f43dc302\") " pod="openshift-ingress/kube-auth-proxy-b966d4c98-vw7ts"
Sep 16 16:45:58.862221 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:58.862197    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f27d87e1-4421-46fb-812b-f159f43dc302-tls-certs\") pod \"kube-auth-proxy-b966d4c98-vw7ts\" (UID: \"f27d87e1-4421-46fb-812b-f159f43dc302\") " pod="openshift-ingress/kube-auth-proxy-b966d4c98-vw7ts"
Sep 16 16:45:59.027454 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:59.027381    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rsbqk"]
Sep 16 16:45:59.037750 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:45:59.037719    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333d82ae_12bc_47a8_8a8a_f6d4621b53a2.slice/crio-7a79f38c5bee36bfb86c0a4cf1936f8dd0ac0fc2ab203165f0f9106b89158ca1 WatchSource:0}: Error finding container 7a79f38c5bee36bfb86c0a4cf1936f8dd0ac0fc2ab203165f0f9106b89158ca1: Status 404 returned error can't find the container with id 7a79f38c5bee36bfb86c0a4cf1936f8dd0ac0fc2ab203165f0f9106b89158ca1
Sep 16 16:45:59.049212 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:59.049194    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b966d4c98-vw7ts"
Sep 16 16:45:59.264090 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:59.264055    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a522445-b1b9-4528-b4bf-5144cd3aae87-tls-certs\") pod \"model-serving-api-54d89bd5fc-vdj6m\" (UID: \"9a522445-b1b9-4528-b4bf-5144cd3aae87\") " pod="opendatahub/model-serving-api-54d89bd5fc-vdj6m"
Sep 16 16:45:59.266505 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:59.266471    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a522445-b1b9-4528-b4bf-5144cd3aae87-tls-certs\") pod \"model-serving-api-54d89bd5fc-vdj6m\" (UID: \"9a522445-b1b9-4528-b4bf-5144cd3aae87\") " pod="opendatahub/model-serving-api-54d89bd5fc-vdj6m"
Sep 16 16:45:59.326749 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:59.326669    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b966d4c98-vw7ts"]
Sep 16 16:45:59.330271 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:45:59.330245    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27d87e1_4421_46fb_812b_f159f43dc302.slice/crio-24130155869e1073779014290b3c300ce305c21d23d89f35375f346b5aae7e41 WatchSource:0}: Error finding container 24130155869e1073779014290b3c300ce305c21d23d89f35375f346b5aae7e41: Status 404 returned error can't find the container with id 24130155869e1073779014290b3c300ce305c21d23d89f35375f346b5aae7e41
Sep 16 16:45:59.532500 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:59.532468    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vdj6m"
Sep 16 16:45:59.759985 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:59.759931    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b966d4c98-vw7ts" event={"ID":"f27d87e1-4421-46fb-812b-f159f43dc302","Type":"ContainerStarted","Data":"24130155869e1073779014290b3c300ce305c21d23d89f35375f346b5aae7e41"}
Sep 16 16:45:59.761469 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:59.761431    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rsbqk" event={"ID":"333d82ae-12bc-47a8-8a8a-f6d4621b53a2","Type":"ContainerStarted","Data":"7a79f38c5bee36bfb86c0a4cf1936f8dd0ac0fc2ab203165f0f9106b89158ca1"}
Sep 16 16:45:59.810572 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:45:59.810521    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vdj6m"]
Sep 16 16:45:59.814736 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:45:59.814705    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a522445_b1b9_4528_b4bf_5144cd3aae87.slice/crio-6eb86b0fe183f3de1c20783ca78b953bf015f316c7d691ecaed1ad71d6a25182 WatchSource:0}: Error finding container 6eb86b0fe183f3de1c20783ca78b953bf015f316c7d691ecaed1ad71d6a25182: Status 404 returned error can't find the container with id 6eb86b0fe183f3de1c20783ca78b953bf015f316c7d691ecaed1ad71d6a25182
Sep 16 16:46:00.774071 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:00.773908    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vdj6m" event={"ID":"9a522445-b1b9-4528-b4bf-5144cd3aae87","Type":"ContainerStarted","Data":"6eb86b0fe183f3de1c20783ca78b953bf015f316c7d691ecaed1ad71d6a25182"}
Sep 16 16:46:04.794862 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:04.794814    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rsbqk" event={"ID":"333d82ae-12bc-47a8-8a8a-f6d4621b53a2","Type":"ContainerStarted","Data":"ef891937e767c91dcad5ca3e430c07f612ef00676a3ec681c0e1ba727cf79e67"}
Sep 16 16:46:04.795472 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:04.794921    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-rsbqk"
Sep 16 16:46:04.796498 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:04.796462    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vdj6m" event={"ID":"9a522445-b1b9-4528-b4bf-5144cd3aae87","Type":"ContainerStarted","Data":"9ab140f2e06611a24847a24c9fccd3123cf0e6671f6fc593b71761ee03983ce7"}
Sep 16 16:46:04.796624 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:04.796519    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-vdj6m"
Sep 16 16:46:04.797876 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:04.797846    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b966d4c98-vw7ts" event={"ID":"f27d87e1-4421-46fb-812b-f159f43dc302","Type":"ContainerStarted","Data":"868dc586677ba7fd346cb453087f31c066aababb31b143fe93d3d54511ca53c9"}
Sep 16 16:46:04.815264 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:04.815222    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-rsbqk" podStartSLOduration=2.432740743 podStartE2EDuration="7.815207914s" totalImagesPullingTime="5.382467171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:45:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:46:04.814343801 +0000 UTC m=+564.576305249" watchObservedRunningTime="2026-09-16 16:46:04.815207914 +0000 UTC m=+564.577169362"
Sep 16 16:46:04.840182 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:04.840145    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-vdj6m" podStartSLOduration=2.235655347 podStartE2EDuration="6.840131901s" totalImagesPullingTime="4.604476554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:46:04.839138165 +0000 UTC m=+564.601099612" watchObservedRunningTime="2026-09-16 16:46:04.840131901 +0000 UTC m=+564.602093348"
Sep 16 16:46:04.860762 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:04.860710    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b966d4c98-vw7ts" podStartSLOduration=1.781323176 podStartE2EDuration="6.86069366s" totalImagesPullingTime="5.079370484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:46:04.860251548 +0000 UTC m=+564.622212997" watchObservedRunningTime="2026-09-16 16:46:04.86069366 +0000 UTC m=+564.622655124"
Sep 16 16:46:06.263693 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.263655    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"]
Sep 16 16:46:06.266648 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.266625    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.269528 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.269504    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 16:46:06.269772 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.269756    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 16:46:06.270530 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.270510    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 16:46:06.270622 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.270563    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 16:46:06.272393 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.272368    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ws4cl\""
Sep 16 16:46:06.295135 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.295108    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"]
Sep 16 16:46:06.330998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.330896    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81ff471a-3fb8-4c64-bfff-fd1be577fe05-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.330998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.330966    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvfzz\" (UniqueName: \"kubernetes.io/projected/81ff471a-3fb8-4c64-bfff-fd1be577fe05-kube-api-access-vvfzz\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.331193 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.331011    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81ff471a-3fb8-4c64-bfff-fd1be577fe05-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.331193 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.331033    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ff471a-3fb8-4c64-bfff-fd1be577fe05-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.331193 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.331055    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ff471a-3fb8-4c64-bfff-fd1be577fe05-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.331193 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.331085    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81ff471a-3fb8-4c64-bfff-fd1be577fe05-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.331193 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.331105    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81ff471a-3fb8-4c64-bfff-fd1be577fe05-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.431855 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.431822    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvfzz\" (UniqueName: \"kubernetes.io/projected/81ff471a-3fb8-4c64-bfff-fd1be577fe05-kube-api-access-vvfzz\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.432039 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.431861    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81ff471a-3fb8-4c64-bfff-fd1be577fe05-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.432039 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.431884    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ff471a-3fb8-4c64-bfff-fd1be577fe05-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.432039 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.431902    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ff471a-3fb8-4c64-bfff-fd1be577fe05-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.432039 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.431959    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81ff471a-3fb8-4c64-bfff-fd1be577fe05-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.432039 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.431991    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81ff471a-3fb8-4c64-bfff-fd1be577fe05-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.432039 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.432031    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81ff471a-3fb8-4c64-bfff-fd1be577fe05-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.432688 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.432663    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81ff471a-3fb8-4c64-bfff-fd1be577fe05-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.434389 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.434360    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81ff471a-3fb8-4c64-bfff-fd1be577fe05-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.434389 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.434386    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81ff471a-3fb8-4c64-bfff-fd1be577fe05-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.434573 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.434551    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81ff471a-3fb8-4c64-bfff-fd1be577fe05-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.434615 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.434585    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81ff471a-3fb8-4c64-bfff-fd1be577fe05-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.452663 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.452628    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81ff471a-3fb8-4c64-bfff-fd1be577fe05-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.454606 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.454587    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvfzz\" (UniqueName: \"kubernetes.io/projected/81ff471a-3fb8-4c64-bfff-fd1be577fe05-kube-api-access-vvfzz\") pod \"istiod-openshift-gateway-5677ccb49b-27bhq\" (UID: \"81ff471a-3fb8-4c64-bfff-fd1be577fe05\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.576380 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.576347    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:06.895235 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:06.895211    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"]
Sep 16 16:46:06.897127 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:46:06.897092    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ff471a_3fb8_4c64_bfff_fd1be577fe05.slice/crio-7671e309ea6ac6889b60b0ba1a85ae09ca991b32b20492e7f65cbf5a6d5f7fd6 WatchSource:0}: Error finding container 7671e309ea6ac6889b60b0ba1a85ae09ca991b32b20492e7f65cbf5a6d5f7fd6: Status 404 returned error can't find the container with id 7671e309ea6ac6889b60b0ba1a85ae09ca991b32b20492e7f65cbf5a6d5f7fd6
Sep 16 16:46:07.812835 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:07.812791    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq" event={"ID":"81ff471a-3fb8-4c64-bfff-fd1be577fe05","Type":"ContainerStarted","Data":"7671e309ea6ac6889b60b0ba1a85ae09ca991b32b20492e7f65cbf5a6d5f7fd6"}
Sep 16 16:46:09.366581 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:09.366538    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 16:46:09.366844 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:09.366609    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 16:46:09.823027 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:09.822985    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq" event={"ID":"81ff471a-3fb8-4c64-bfff-fd1be577fe05","Type":"ContainerStarted","Data":"93a285d7474fd965f5b4536fb5af083feb6e8204dd41a4df1238abe6d7875f59"}
Sep 16 16:46:09.823244 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:09.823098    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:09.858833 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:09.858768    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq" podStartSLOduration=1.391465668 podStartE2EDuration="3.858749156s" totalImagesPullingTime="2.467283488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:46:09.857684858 +0000 UTC m=+569.619646304" watchObservedRunningTime="2026-09-16 16:46:09.858749156 +0000 UTC m=+569.620710605"
Sep 16 16:46:10.828055 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:10.828024    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-27bhq"
Sep 16 16:46:15.803839 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:15.803808    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-rsbqk"
Sep 16 16:46:15.805825 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:46:15.805802    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-vdj6m"
Sep 16 16:47:16.873180 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:16.873144    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk"]
Sep 16 16:47:16.875507 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:16.875491    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk"
Sep 16 16:47:16.878015 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:16.877996    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 16:47:16.878119 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:16.878052    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 16:47:16.878761 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:16.878746    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-zfkt2\""
Sep 16 16:47:16.890171 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:16.890151    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk"]
Sep 16 16:47:16.958602 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:16.958571    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms6w5\" (UniqueName: \"kubernetes.io/projected/47399146-e232-4926-9f95-b78ce0a29ce9-kube-api-access-ms6w5\") pod \"limitador-operator-controller-manager-74495c69f8-j8dmk\" (UID: \"47399146-e232-4926-9f95-b78ce0a29ce9\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk"
Sep 16 16:47:17.059009 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:17.058979    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms6w5\" (UniqueName: \"kubernetes.io/projected/47399146-e232-4926-9f95-b78ce0a29ce9-kube-api-access-ms6w5\") pod \"limitador-operator-controller-manager-74495c69f8-j8dmk\" (UID: \"47399146-e232-4926-9f95-b78ce0a29ce9\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk"
Sep 16 16:47:17.070292 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:17.070264    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms6w5\" (UniqueName: \"kubernetes.io/projected/47399146-e232-4926-9f95-b78ce0a29ce9-kube-api-access-ms6w5\") pod \"limitador-operator-controller-manager-74495c69f8-j8dmk\" (UID: \"47399146-e232-4926-9f95-b78ce0a29ce9\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk"
Sep 16 16:47:17.185096 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:17.185074    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk"
Sep 16 16:47:17.459422 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:17.459357    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk"]
Sep 16 16:47:17.462081 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:47:17.462044    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47399146_e232_4926_9f95_b78ce0a29ce9.slice/crio-2bff1680a54cab73276278f4f7fa14a33e235d68e9fd2b338fdda8b9b4436ace WatchSource:0}: Error finding container 2bff1680a54cab73276278f4f7fa14a33e235d68e9fd2b338fdda8b9b4436ace: Status 404 returned error can't find the container with id 2bff1680a54cab73276278f4f7fa14a33e235d68e9fd2b338fdda8b9b4436ace
Sep 16 16:47:17.464261 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:17.464246    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 16:47:18.100626 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:18.100590    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk" event={"ID":"47399146-e232-4926-9f95-b78ce0a29ce9","Type":"ContainerStarted","Data":"2bff1680a54cab73276278f4f7fa14a33e235d68e9fd2b338fdda8b9b4436ace"}
Sep 16 16:47:21.115868 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:21.115830    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk" event={"ID":"47399146-e232-4926-9f95-b78ce0a29ce9","Type":"ContainerStarted","Data":"a88d5dfdb4d181412e10081f69efd173e185434b0f1734cbec2ac11a18d60636"}
Sep 16 16:47:21.116273 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:21.115922    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk"
Sep 16 16:47:21.138536 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:21.138492    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk" podStartSLOduration=2.280911532 podStartE2EDuration="5.138472068s" totalImagesPullingTime="2.857560536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:47:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:47:21.136338446 +0000 UTC m=+640.898299906" watchObservedRunningTime="2026-09-16 16:47:21.138472068 +0000 UTC m=+640.900433515"
Sep 16 16:47:23.480757 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:23.480722    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"]
Sep 16 16:47:23.482865 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:23.482848    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"
Sep 16 16:47:23.485194 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:23.485174    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ngnlg\""
Sep 16 16:47:23.502591 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:23.502568    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"]
Sep 16 16:47:23.611744 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:23.611706    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2ln6\" (UniqueName: \"kubernetes.io/projected/221b8665-b059-4f3a-818c-d94177611715-kube-api-access-l2ln6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fvnkp\" (UID: \"221b8665-b059-4f3a-818c-d94177611715\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"
Sep 16 16:47:23.611904 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:23.611795    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/221b8665-b059-4f3a-818c-d94177611715-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fvnkp\" (UID: \"221b8665-b059-4f3a-818c-d94177611715\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"
Sep 16 16:47:23.713147 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:23.713113    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/221b8665-b059-4f3a-818c-d94177611715-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fvnkp\" (UID: \"221b8665-b059-4f3a-818c-d94177611715\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"
Sep 16 16:47:23.713332 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:23.713168    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2ln6\" (UniqueName: \"kubernetes.io/projected/221b8665-b059-4f3a-818c-d94177611715-kube-api-access-l2ln6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fvnkp\" (UID: \"221b8665-b059-4f3a-818c-d94177611715\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"
Sep 16 16:47:23.713492 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:23.713474    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/221b8665-b059-4f3a-818c-d94177611715-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fvnkp\" (UID: \"221b8665-b059-4f3a-818c-d94177611715\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"
Sep 16 16:47:23.723435 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:23.723402    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2ln6\" (UniqueName: \"kubernetes.io/projected/221b8665-b059-4f3a-818c-d94177611715-kube-api-access-l2ln6\") pod \"kuadrant-operator-controller-manager-76f8f686bf-fvnkp\" (UID: \"221b8665-b059-4f3a-818c-d94177611715\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"
Sep 16 16:47:23.792180 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:23.792122    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"
Sep 16 16:47:24.080263 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:24.080239    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"]
Sep 16 16:47:24.083959 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:47:24.082601    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod221b8665_b059_4f3a_818c_d94177611715.slice/crio-63632692ab2a0b7b82cc71419258233623ed1de0455678d2fc24dfe591fa0d67 WatchSource:0}: Error finding container 63632692ab2a0b7b82cc71419258233623ed1de0455678d2fc24dfe591fa0d67: Status 404 returned error can't find the container with id 63632692ab2a0b7b82cc71419258233623ed1de0455678d2fc24dfe591fa0d67
Sep 16 16:47:24.128909 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:24.128879    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp" event={"ID":"221b8665-b059-4f3a-818c-d94177611715","Type":"ContainerStarted","Data":"63632692ab2a0b7b82cc71419258233623ed1de0455678d2fc24dfe591fa0d67"}
Sep 16 16:47:29.154480 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:29.154401    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp" event={"ID":"221b8665-b059-4f3a-818c-d94177611715","Type":"ContainerStarted","Data":"d3bac63cdb2a2597f7a44c8495b45721b864179215e7680eb45abea13e1a7c36"}
Sep 16 16:47:29.154835 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:29.154514    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"
Sep 16 16:47:29.175792 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:29.175744    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp" podStartSLOduration=1.5752787110000002 podStartE2EDuration="6.175729509s" totalImagesPullingTime="4.600450798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:47:29.174459955 +0000 UTC m=+648.936421602" watchObservedRunningTime="2026-09-16 16:47:29.175729509 +0000 UTC m=+648.937690953"
Sep 16 16:47:32.121997 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:32.121967    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk"
Sep 16 16:47:40.161271 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:40.161244    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"
Sep 16 16:47:41.547482 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:41.547450    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"]
Sep 16 16:47:41.549706 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:41.549689    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"
Sep 16 16:47:41.572708 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:41.572684    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"]
Sep 16 16:47:41.660607 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:41.660575    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-4k697\" (UID: \"7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"
Sep 16 16:47:41.660755 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:41.660636    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6k7w\" (UniqueName: \"kubernetes.io/projected/7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c-kube-api-access-c6k7w\") pod \"kuadrant-operator-controller-manager-5f685c496f-4k697\" (UID: \"7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"
Sep 16 16:47:41.761948 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:41.761910    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-4k697\" (UID: \"7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"
Sep 16 16:47:41.762055 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:41.762031    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6k7w\" (UniqueName: \"kubernetes.io/projected/7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c-kube-api-access-c6k7w\") pod \"kuadrant-operator-controller-manager-5f685c496f-4k697\" (UID: \"7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"
Sep 16 16:47:41.762337 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:41.762319    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-4k697\" (UID: \"7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"
Sep 16 16:47:41.773202 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:41.773182    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6k7w\" (UniqueName: \"kubernetes.io/projected/7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c-kube-api-access-c6k7w\") pod \"kuadrant-operator-controller-manager-5f685c496f-4k697\" (UID: \"7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"
Sep 16 16:47:41.859611 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:41.859526    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"
Sep 16 16:47:42.133534 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.133508    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"]
Sep 16 16:47:42.135951 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:47:42.135916    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5ee1dd_0ee5_4982_95d2_117c92d6bb5c.slice/crio-378d6e3e55ab20de3f53d302937a4bee98494874e0b80a5a8526f86dcf347c7e WatchSource:0}: Error finding container 378d6e3e55ab20de3f53d302937a4bee98494874e0b80a5a8526f86dcf347c7e: Status 404 returned error can't find the container with id 378d6e3e55ab20de3f53d302937a4bee98494874e0b80a5a8526f86dcf347c7e
Sep 16 16:47:42.163136 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.163111    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"]
Sep 16 16:47:42.172280 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.172257    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"]
Sep 16 16:47:42.190156 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.190134    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf"]
Sep 16 16:47:42.190481 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.190466    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c" containerName="manager"
Sep 16 16:47:42.190566 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.190482    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c" containerName="manager"
Sep 16 16:47:42.190624 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.190564    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c" containerName="manager"
Sep 16 16:47:42.192903 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.192567    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf"
Sep 16 16:47:42.194447 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.194426    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"]
Sep 16 16:47:42.194631 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.194595    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp" podUID="221b8665-b059-4f3a-818c-d94177611715" containerName="manager" containerID="cri-o://d3bac63cdb2a2597f7a44c8495b45721b864179215e7680eb45abea13e1a7c36" gracePeriod=2
Sep 16 16:47:42.201649 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.201553    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"]
Sep 16 16:47:42.211443 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.211421    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf"]
Sep 16 16:47:42.213973 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.213950    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk"]
Sep 16 16:47:42.214159 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.214131    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk" podUID="47399146-e232-4926-9f95-b78ce0a29ce9" containerName="manager" containerID="cri-o://a88d5dfdb4d181412e10081f69efd173e185434b0f1734cbec2ac11a18d60636" gracePeriod=2
Sep 16 16:47:42.214343 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.214312    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697" podUID="7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c" containerName="manager" containerID="cri-o://061ef750c9d4340e360a780c28bb8f31a0d3eb2b00e9b2250cda60f4c75572bb" gracePeriod=2
Sep 16 16:47:42.225460 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.225441    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk"]
Sep 16 16:47:42.240121 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.240101    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw"]
Sep 16 16:47:42.240487 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.240467    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="221b8665-b059-4f3a-818c-d94177611715" containerName="manager"
Sep 16 16:47:42.240487 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.240489    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="221b8665-b059-4f3a-818c-d94177611715" containerName="manager"
Sep 16 16:47:42.240626 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.240508    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47399146-e232-4926-9f95-b78ce0a29ce9" containerName="manager"
Sep 16 16:47:42.240626 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.240517    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="47399146-e232-4926-9f95-b78ce0a29ce9" containerName="manager"
Sep 16 16:47:42.240626 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.240580    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="221b8665-b059-4f3a-818c-d94177611715" containerName="manager"
Sep 16 16:47:42.240626 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.240592    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="47399146-e232-4926-9f95-b78ce0a29ce9" containerName="manager"
Sep 16 16:47:42.242275 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.242260    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw"
Sep 16 16:47:42.245278 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:47:42.245106    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-fvnkp\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="221b8665-b059-4f3a-818c-d94177611715" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"
Sep 16 16:47:42.248600 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:47:42.248567    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-j8dmk\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="47399146-e232-4926-9f95-b78ce0a29ce9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk"
Sep 16 16:47:42.254279 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:47:42.254253    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-4k697\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"
Sep 16 16:47:42.256266 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:47:42.256242    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-fvnkp\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="221b8665-b059-4f3a-818c-d94177611715" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"
Sep 16 16:47:42.258832 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:47:42.258625    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-j8dmk\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="47399146-e232-4926-9f95-b78ce0a29ce9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk"
Sep 16 16:47:42.259298 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.259278    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-82jzb"]
Sep 16 16:47:42.260920 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:47:42.260752    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-4k697\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"
Sep 16 16:47:42.261550 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.261328    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-82jzb"
Sep 16 16:47:42.276279 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.276254    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw"]
Sep 16 16:47:42.278049 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.278028    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-82jzb"]
Sep 16 16:47:42.302122 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:47:42.302091    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-j8dmk\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="47399146-e232-4926-9f95-b78ce0a29ce9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk"
Sep 16 16:47:42.304614 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:47:42.304581    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-4k697\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"
Sep 16 16:47:42.307008 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:47:42.306982    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-fvnkp\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="221b8665-b059-4f3a-818c-d94177611715" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"
Sep 16 16:47:42.366080 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.365958    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/784812d2-d5ea-4d0f-a92b-ef1dfab9f526-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-j79xf\" (UID: \"784812d2-d5ea-4d0f-a92b-ef1dfab9f526\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf"
Sep 16 16:47:42.366080 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.365999    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk4p6\" (UniqueName: \"kubernetes.io/projected/784812d2-d5ea-4d0f-a92b-ef1dfab9f526-kube-api-access-rk4p6\") pod \"kuadrant-operator-controller-manager-5f685c496f-j79xf\" (UID: \"784812d2-d5ea-4d0f-a92b-ef1dfab9f526\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf"
Sep 16 16:47:42.366080 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.366059    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26e6d99e-b725-4b3f-b799-9f804d651965-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d89fw\" (UID: \"26e6d99e-b725-4b3f-b799-9f804d651965\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw"
Sep 16 16:47:42.366282 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.366124    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gdh5\" (UniqueName: \"kubernetes.io/projected/26e6d99e-b725-4b3f-b799-9f804d651965-kube-api-access-2gdh5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d89fw\" (UID: \"26e6d99e-b725-4b3f-b799-9f804d651965\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw"
Sep 16 16:47:42.366282 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.366156    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-987xs\" (UniqueName: \"kubernetes.io/projected/49273cd9-7389-4997-ae50-4d38382174c8-kube-api-access-987xs\") pod \"limitador-operator-controller-manager-74495c69f8-82jzb\" (UID: \"49273cd9-7389-4997-ae50-4d38382174c8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-82jzb"
Sep 16 16:47:42.439358 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.439329    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"
Sep 16 16:47:42.441574 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:47:42.441547    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-4k697\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"
Sep 16 16:47:42.443409 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:47:42.443380    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-fvnkp\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="221b8665-b059-4f3a-818c-d94177611715" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"
Sep 16 16:47:42.445253 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:47:42.445231    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-j8dmk\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="47399146-e232-4926-9f95-b78ce0a29ce9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk"
Sep 16 16:47:42.445663 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.445645    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk"
Sep 16 16:47:42.447419 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:47:42.447396    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-j8dmk\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="47399146-e232-4926-9f95-b78ce0a29ce9" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk"
Sep 16 16:47:42.450086 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:47:42.450068    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-4k697\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"
Sep 16 16:47:42.451732 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:47:42.451715    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-fvnkp\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="221b8665-b059-4f3a-818c-d94177611715" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"
Sep 16 16:47:42.466653 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.466630    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26e6d99e-b725-4b3f-b799-9f804d651965-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d89fw\" (UID: \"26e6d99e-b725-4b3f-b799-9f804d651965\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw"
Sep 16 16:47:42.466717 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.466673    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gdh5\" (UniqueName: \"kubernetes.io/projected/26e6d99e-b725-4b3f-b799-9f804d651965-kube-api-access-2gdh5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d89fw\" (UID: \"26e6d99e-b725-4b3f-b799-9f804d651965\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw"
Sep 16 16:47:42.466717 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.466690    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-987xs\" (UniqueName: \"kubernetes.io/projected/49273cd9-7389-4997-ae50-4d38382174c8-kube-api-access-987xs\") pod \"limitador-operator-controller-manager-74495c69f8-82jzb\" (UID: \"49273cd9-7389-4997-ae50-4d38382174c8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-82jzb"
Sep 16 16:47:42.466717 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.466713    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/784812d2-d5ea-4d0f-a92b-ef1dfab9f526-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-j79xf\" (UID: \"784812d2-d5ea-4d0f-a92b-ef1dfab9f526\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf"
Sep 16 16:47:42.466824 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.466748    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk4p6\" (UniqueName: \"kubernetes.io/projected/784812d2-d5ea-4d0f-a92b-ef1dfab9f526-kube-api-access-rk4p6\") pod \"kuadrant-operator-controller-manager-5f685c496f-j79xf\" (UID: \"784812d2-d5ea-4d0f-a92b-ef1dfab9f526\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf"
Sep 16 16:47:42.467065 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.467047    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26e6d99e-b725-4b3f-b799-9f804d651965-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d89fw\" (UID: \"26e6d99e-b725-4b3f-b799-9f804d651965\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw"
Sep 16 16:47:42.467141 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.467111    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/784812d2-d5ea-4d0f-a92b-ef1dfab9f526-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-j79xf\" (UID: \"784812d2-d5ea-4d0f-a92b-ef1dfab9f526\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf"
Sep 16 16:47:42.477747 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.477723    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gdh5\" (UniqueName: \"kubernetes.io/projected/26e6d99e-b725-4b3f-b799-9f804d651965-kube-api-access-2gdh5\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d89fw\" (UID: \"26e6d99e-b725-4b3f-b799-9f804d651965\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw"
Sep 16 16:47:42.477922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.477905    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-987xs\" (UniqueName: \"kubernetes.io/projected/49273cd9-7389-4997-ae50-4d38382174c8-kube-api-access-987xs\") pod \"limitador-operator-controller-manager-74495c69f8-82jzb\" (UID: \"49273cd9-7389-4997-ae50-4d38382174c8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-82jzb"
Sep 16 16:47:42.478035 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.478018    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk4p6\" (UniqueName: \"kubernetes.io/projected/784812d2-d5ea-4d0f-a92b-ef1dfab9f526-kube-api-access-rk4p6\") pod \"kuadrant-operator-controller-manager-5f685c496f-j79xf\" (UID: \"784812d2-d5ea-4d0f-a92b-ef1dfab9f526\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf"
Sep 16 16:47:42.506047 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.506021    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf"
Sep 16 16:47:42.567396 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.567371    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2ln6\" (UniqueName: \"kubernetes.io/projected/221b8665-b059-4f3a-818c-d94177611715-kube-api-access-l2ln6\") pod \"221b8665-b059-4f3a-818c-d94177611715\" (UID: \"221b8665-b059-4f3a-818c-d94177611715\") "
Sep 16 16:47:42.567750 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.567456    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms6w5\" (UniqueName: \"kubernetes.io/projected/47399146-e232-4926-9f95-b78ce0a29ce9-kube-api-access-ms6w5\") pod \"47399146-e232-4926-9f95-b78ce0a29ce9\" (UID: \"47399146-e232-4926-9f95-b78ce0a29ce9\") "
Sep 16 16:47:42.567750 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.567478    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/221b8665-b059-4f3a-818c-d94177611715-extensions-socket-volume\") pod \"221b8665-b059-4f3a-818c-d94177611715\" (UID: \"221b8665-b059-4f3a-818c-d94177611715\") "
Sep 16 16:47:42.567922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.567899    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221b8665-b059-4f3a-818c-d94177611715-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "221b8665-b059-4f3a-818c-d94177611715" (UID: "221b8665-b059-4f3a-818c-d94177611715"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:47:42.570141 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.570117    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221b8665-b059-4f3a-818c-d94177611715-kube-api-access-l2ln6" (OuterVolumeSpecName: "kube-api-access-l2ln6") pod "221b8665-b059-4f3a-818c-d94177611715" (UID: "221b8665-b059-4f3a-818c-d94177611715"). InnerVolumeSpecName "kube-api-access-l2ln6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:47:42.570212 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.570132    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47399146-e232-4926-9f95-b78ce0a29ce9-kube-api-access-ms6w5" (OuterVolumeSpecName: "kube-api-access-ms6w5") pod "47399146-e232-4926-9f95-b78ce0a29ce9" (UID: "47399146-e232-4926-9f95-b78ce0a29ce9"). InnerVolumeSpecName "kube-api-access-ms6w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:47:42.629532 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.629492    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw"
Sep 16 16:47:42.637211 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.637188    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-82jzb"
Sep 16 16:47:42.668262 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.668230    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms6w5\" (UniqueName: \"kubernetes.io/projected/47399146-e232-4926-9f95-b78ce0a29ce9-kube-api-access-ms6w5\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:47:42.668410 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.668264    2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/221b8665-b059-4f3a-818c-d94177611715-extensions-socket-volume\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:47:42.668410 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.668281    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2ln6\" (UniqueName: \"kubernetes.io/projected/221b8665-b059-4f3a-818c-d94177611715-kube-api-access-l2ln6\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:47:42.824127 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.824098    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf"]
Sep 16 16:47:42.826926 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:47:42.826902    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784812d2_d5ea_4d0f_a92b_ef1dfab9f526.slice/crio-0cda5c3bdc07ec5191afe7d4fc7358ea7651a45c530b275877216f4eaaeb8a6a WatchSource:0}: Error finding container 0cda5c3bdc07ec5191afe7d4fc7358ea7651a45c530b275877216f4eaaeb8a6a: Status 404 returned error can't find the container with id 0cda5c3bdc07ec5191afe7d4fc7358ea7651a45c530b275877216f4eaaeb8a6a
Sep 16 16:47:42.849164 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.849141    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="221b8665-b059-4f3a-818c-d94177611715" path="/var/lib/kubelet/pods/221b8665-b059-4f3a-818c-d94177611715/volumes"
Sep 16 16:47:42.849449 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.849436    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47399146-e232-4926-9f95-b78ce0a29ce9" path="/var/lib/kubelet/pods/47399146-e232-4926-9f95-b78ce0a29ce9/volumes"
Sep 16 16:47:42.913582 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.913559    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw"]
Sep 16 16:47:42.915832 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:47:42.915807    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26e6d99e_b725_4b3f_b799_9f804d651965.slice/crio-72d2679be99f9359f4e447efe0e6390b96691a9a0420a9574a36ba27a5a01e11 WatchSource:0}: Error finding container 72d2679be99f9359f4e447efe0e6390b96691a9a0420a9574a36ba27a5a01e11: Status 404 returned error can't find the container with id 72d2679be99f9359f4e447efe0e6390b96691a9a0420a9574a36ba27a5a01e11
Sep 16 16:47:42.927057 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:42.927036    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-82jzb"]
Sep 16 16:47:42.930152 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:47:42.930132    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49273cd9_7389_4997_ae50_4d38382174c8.slice/crio-bdaa708017da6f15928299578511d17654792a0ea06d91ab187bc936b970d5ba WatchSource:0}: Error finding container bdaa708017da6f15928299578511d17654792a0ea06d91ab187bc936b970d5ba: Status 404 returned error can't find the container with id bdaa708017da6f15928299578511d17654792a0ea06d91ab187bc936b970d5ba
Sep 16 16:47:43.220677 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.220637    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-82jzb" event={"ID":"49273cd9-7389-4997-ae50-4d38382174c8","Type":"ContainerStarted","Data":"6eb2d051db4c26ca81e2032106c4941e3896d9c21e72cc8da27598fe11d0f441"}
Sep 16 16:47:43.220851 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.220701    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-82jzb" event={"ID":"49273cd9-7389-4997-ae50-4d38382174c8","Type":"ContainerStarted","Data":"bdaa708017da6f15928299578511d17654792a0ea06d91ab187bc936b970d5ba"}
Sep 16 16:47:43.220851 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.220732    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-82jzb"
Sep 16 16:47:43.222415 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.222384    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw" event={"ID":"26e6d99e-b725-4b3f-b799-9f804d651965","Type":"ContainerStarted","Data":"8104be8053e4828704e33cc66574458311325e4b0b442e44afc41a162c5d9de3"}
Sep 16 16:47:43.222538 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.222432    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw"
Sep 16 16:47:43.222538 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.222453    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw" event={"ID":"26e6d99e-b725-4b3f-b799-9f804d651965","Type":"ContainerStarted","Data":"72d2679be99f9359f4e447efe0e6390b96691a9a0420a9574a36ba27a5a01e11"}
Sep 16 16:47:43.223562 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.223534    2556 generic.go:356] "Generic (PLEG): container finished" podID="47399146-e232-4926-9f95-b78ce0a29ce9" containerID="a88d5dfdb4d181412e10081f69efd173e185434b0f1734cbec2ac11a18d60636" exitCode=0
Sep 16 16:47:43.223643 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.223574    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-j8dmk"
Sep 16 16:47:43.223643 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.223626    2556 scope.go:117] "RemoveContainer" containerID="a88d5dfdb4d181412e10081f69efd173e185434b0f1734cbec2ac11a18d60636"
Sep 16 16:47:43.225922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.225702    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf" event={"ID":"784812d2-d5ea-4d0f-a92b-ef1dfab9f526","Type":"ContainerStarted","Data":"377c9b78554e73d53a0e8e429e44d6891e31b41517e1561e553fad05f21f3c1d"}
Sep 16 16:47:43.225922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.225746    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf" event={"ID":"784812d2-d5ea-4d0f-a92b-ef1dfab9f526","Type":"ContainerStarted","Data":"0cda5c3bdc07ec5191afe7d4fc7358ea7651a45c530b275877216f4eaaeb8a6a"}
Sep 16 16:47:43.226094 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.225971    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf"
Sep 16 16:47:43.227276 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.227253    2556 generic.go:356] "Generic (PLEG): container finished" podID="221b8665-b059-4f3a-818c-d94177611715" containerID="d3bac63cdb2a2597f7a44c8495b45721b864179215e7680eb45abea13e1a7c36" exitCode=0
Sep 16 16:47:43.227366 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.227293    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-fvnkp"
Sep 16 16:47:43.233366 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.233352    2556 scope.go:117] "RemoveContainer" containerID="a88d5dfdb4d181412e10081f69efd173e185434b0f1734cbec2ac11a18d60636"
Sep 16 16:47:43.233633 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:47:43.233610    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a88d5dfdb4d181412e10081f69efd173e185434b0f1734cbec2ac11a18d60636\": container with ID starting with a88d5dfdb4d181412e10081f69efd173e185434b0f1734cbec2ac11a18d60636 not found: ID does not exist" containerID="a88d5dfdb4d181412e10081f69efd173e185434b0f1734cbec2ac11a18d60636"
Sep 16 16:47:43.233677 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.233643    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88d5dfdb4d181412e10081f69efd173e185434b0f1734cbec2ac11a18d60636"} err="failed to get container status \"a88d5dfdb4d181412e10081f69efd173e185434b0f1734cbec2ac11a18d60636\": rpc error: code = NotFound desc = could not find container \"a88d5dfdb4d181412e10081f69efd173e185434b0f1734cbec2ac11a18d60636\": container with ID starting with a88d5dfdb4d181412e10081f69efd173e185434b0f1734cbec2ac11a18d60636 not found: ID does not exist"
Sep 16 16:47:43.233677 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.233660    2556 scope.go:117] "RemoveContainer" containerID="d3bac63cdb2a2597f7a44c8495b45721b864179215e7680eb45abea13e1a7c36"
Sep 16 16:47:43.241026 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.240995    2556 scope.go:117] "RemoveContainer" containerID="d3bac63cdb2a2597f7a44c8495b45721b864179215e7680eb45abea13e1a7c36"
Sep 16 16:47:43.241272 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:47:43.241256    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3bac63cdb2a2597f7a44c8495b45721b864179215e7680eb45abea13e1a7c36\": container with ID starting with d3bac63cdb2a2597f7a44c8495b45721b864179215e7680eb45abea13e1a7c36 not found: ID does not exist" containerID="d3bac63cdb2a2597f7a44c8495b45721b864179215e7680eb45abea13e1a7c36"
Sep 16 16:47:43.241332 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.241278    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3bac63cdb2a2597f7a44c8495b45721b864179215e7680eb45abea13e1a7c36"} err="failed to get container status \"d3bac63cdb2a2597f7a44c8495b45721b864179215e7680eb45abea13e1a7c36\": rpc error: code = NotFound desc = could not find container \"d3bac63cdb2a2597f7a44c8495b45721b864179215e7680eb45abea13e1a7c36\": container with ID starting with d3bac63cdb2a2597f7a44c8495b45721b864179215e7680eb45abea13e1a7c36 not found: ID does not exist"
Sep 16 16:47:43.269190 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.269136    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-82jzb" podStartSLOduration=1.269120128 podStartE2EDuration="1.269120128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:47:43.247773515 +0000 UTC m=+663.009734964" watchObservedRunningTime="2026-09-16 16:47:43.269120128 +0000 UTC m=+663.031081574"
Sep 16 16:47:43.269793 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.269764    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf" podStartSLOduration=1.2697552619999999 podStartE2EDuration="1.269755262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:47:43.268326321 +0000 UTC m=+663.030287780" watchObservedRunningTime="2026-09-16 16:47:43.269755262 +0000 UTC m=+663.031716708"
Sep 16 16:47:43.289541 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:43.289497    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw" podStartSLOduration=1.289482958 podStartE2EDuration="1.289482958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:47:43.28914056 +0000 UTC m=+663.051102011" watchObservedRunningTime="2026-09-16 16:47:43.289482958 +0000 UTC m=+663.051444405"
Sep 16 16:47:44.387330 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:44.387305    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"
Sep 16 16:47:44.483745 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:44.483687    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c-extensions-socket-volume\") pod \"7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c\" (UID: \"7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c\") "
Sep 16 16:47:44.483845 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:44.483751    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6k7w\" (UniqueName: \"kubernetes.io/projected/7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c-kube-api-access-c6k7w\") pod \"7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c\" (UID: \"7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c\") "
Sep 16 16:47:44.483953 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:44.483916    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c" (UID: "7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:47:44.485611 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:44.485588    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c-kube-api-access-c6k7w" (OuterVolumeSpecName: "kube-api-access-c6k7w") pod "7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c" (UID: "7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c"). InnerVolumeSpecName "kube-api-access-c6k7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:47:44.584298 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:44.584270    2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c-extensions-socket-volume\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:47:44.584298 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:44.584294    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6k7w\" (UniqueName: \"kubernetes.io/projected/7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c-kube-api-access-c6k7w\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:47:44.848921 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:44.848851    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c" path="/var/lib/kubelet/pods/7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c/volumes"
Sep 16 16:47:45.242277 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:45.242236    2556 generic.go:356] "Generic (PLEG): container finished" podID="7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c" containerID="061ef750c9d4340e360a780c28bb8f31a0d3eb2b00e9b2250cda60f4c75572bb" exitCode=137
Sep 16 16:47:45.242456 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:45.242300    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"
Sep 16 16:47:45.242456 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:45.242331    2556 scope.go:117] "RemoveContainer" containerID="061ef750c9d4340e360a780c28bb8f31a0d3eb2b00e9b2250cda60f4c75572bb"
Sep 16 16:47:45.246490 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:47:45.246453    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-4k697\" is forbidden: User \"system:node:ip-10-0-140-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-120.ec2.internal' and this object" podUID="7b5ee1dd-0ee5-4982-95d2-117c92d6bb5c" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4k697"
Sep 16 16:47:45.250171 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:45.250151    2556 scope.go:117] "RemoveContainer" containerID="061ef750c9d4340e360a780c28bb8f31a0d3eb2b00e9b2250cda60f4c75572bb"
Sep 16 16:47:45.250427 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:47:45.250411    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"061ef750c9d4340e360a780c28bb8f31a0d3eb2b00e9b2250cda60f4c75572bb\": container with ID starting with 061ef750c9d4340e360a780c28bb8f31a0d3eb2b00e9b2250cda60f4c75572bb not found: ID does not exist" containerID="061ef750c9d4340e360a780c28bb8f31a0d3eb2b00e9b2250cda60f4c75572bb"
Sep 16 16:47:45.250481 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:45.250435    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061ef750c9d4340e360a780c28bb8f31a0d3eb2b00e9b2250cda60f4c75572bb"} err="failed to get container status \"061ef750c9d4340e360a780c28bb8f31a0d3eb2b00e9b2250cda60f4c75572bb\": rpc error: code = NotFound desc = could not find container \"061ef750c9d4340e360a780c28bb8f31a0d3eb2b00e9b2250cda60f4c75572bb\": container with ID starting with 061ef750c9d4340e360a780c28bb8f31a0d3eb2b00e9b2250cda60f4c75572bb not found: ID does not exist"
Sep 16 16:47:54.236250 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.236217    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf"
Sep 16 16:47:54.236730 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.236287    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-82jzb"
Sep 16 16:47:54.236730 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.236411    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw"
Sep 16 16:47:54.298046 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.298012    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw"]
Sep 16 16:47:54.298215 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.298195    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw" podUID="26e6d99e-b725-4b3f-b799-9f804d651965" containerName="manager" containerID="cri-o://8104be8053e4828704e33cc66574458311325e4b0b442e44afc41a162c5d9de3" gracePeriod=10
Sep 16 16:47:54.544055 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.544030    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv"]
Sep 16 16:47:54.544362 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.544344    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw"
Sep 16 16:47:54.547419 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.547404    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv"
Sep 16 16:47:54.570343 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.570279    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv"]
Sep 16 16:47:54.667739 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.667713    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gdh5\" (UniqueName: \"kubernetes.io/projected/26e6d99e-b725-4b3f-b799-9f804d651965-kube-api-access-2gdh5\") pod \"26e6d99e-b725-4b3f-b799-9f804d651965\" (UID: \"26e6d99e-b725-4b3f-b799-9f804d651965\") "
Sep 16 16:47:54.667862 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.667830    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26e6d99e-b725-4b3f-b799-9f804d651965-extensions-socket-volume\") pod \"26e6d99e-b725-4b3f-b799-9f804d651965\" (UID: \"26e6d99e-b725-4b3f-b799-9f804d651965\") "
Sep 16 16:47:54.667916 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.667899    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph572\" (UniqueName: \"kubernetes.io/projected/a7aeffa2-475a-47d0-9556-bee3e5534928-kube-api-access-ph572\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2mvqv\" (UID: \"a7aeffa2-475a-47d0-9556-bee3e5534928\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv"
Sep 16 16:47:54.667973 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.667935    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7aeffa2-475a-47d0-9556-bee3e5534928-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2mvqv\" (UID: \"a7aeffa2-475a-47d0-9556-bee3e5534928\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv"
Sep 16 16:47:54.668203 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.668176    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26e6d99e-b725-4b3f-b799-9f804d651965-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "26e6d99e-b725-4b3f-b799-9f804d651965" (UID: "26e6d99e-b725-4b3f-b799-9f804d651965"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:47:54.669480 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.669460    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26e6d99e-b725-4b3f-b799-9f804d651965-kube-api-access-2gdh5" (OuterVolumeSpecName: "kube-api-access-2gdh5") pod "26e6d99e-b725-4b3f-b799-9f804d651965" (UID: "26e6d99e-b725-4b3f-b799-9f804d651965"). InnerVolumeSpecName "kube-api-access-2gdh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:47:54.768853 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.768830    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph572\" (UniqueName: \"kubernetes.io/projected/a7aeffa2-475a-47d0-9556-bee3e5534928-kube-api-access-ph572\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2mvqv\" (UID: \"a7aeffa2-475a-47d0-9556-bee3e5534928\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv"
Sep 16 16:47:54.768995 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.768886    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7aeffa2-475a-47d0-9556-bee3e5534928-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2mvqv\" (UID: \"a7aeffa2-475a-47d0-9556-bee3e5534928\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv"
Sep 16 16:47:54.769056 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.768992    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gdh5\" (UniqueName: \"kubernetes.io/projected/26e6d99e-b725-4b3f-b799-9f804d651965-kube-api-access-2gdh5\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:47:54.769056 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.769012    2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/26e6d99e-b725-4b3f-b799-9f804d651965-extensions-socket-volume\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:47:54.769236 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.769219    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7aeffa2-475a-47d0-9556-bee3e5534928-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2mvqv\" (UID: \"a7aeffa2-475a-47d0-9556-bee3e5534928\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv"
Sep 16 16:47:54.783719 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.783666    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph572\" (UniqueName: \"kubernetes.io/projected/a7aeffa2-475a-47d0-9556-bee3e5534928-kube-api-access-ph572\") pod \"kuadrant-operator-controller-manager-5769f99d9f-2mvqv\" (UID: \"a7aeffa2-475a-47d0-9556-bee3e5534928\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv"
Sep 16 16:47:54.856183 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:54.856152    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv"
Sep 16 16:47:55.134698 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:55.134610    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv"]
Sep 16 16:47:55.137059 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:47:55.137026    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7aeffa2_475a_47d0_9556_bee3e5534928.slice/crio-0bc48070c3c7a64ec47b5bd86c433cbd8e7f8200abb2259df4bc4c6710012d18 WatchSource:0}: Error finding container 0bc48070c3c7a64ec47b5bd86c433cbd8e7f8200abb2259df4bc4c6710012d18: Status 404 returned error can't find the container with id 0bc48070c3c7a64ec47b5bd86c433cbd8e7f8200abb2259df4bc4c6710012d18
Sep 16 16:47:55.291441 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:55.291407    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv" event={"ID":"a7aeffa2-475a-47d0-9556-bee3e5534928","Type":"ContainerStarted","Data":"28b0ada8e48fc956ffec86876fe823896cb949f6f4ceb554168887461e96d617"}
Sep 16 16:47:55.291837 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:55.291465    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv" event={"ID":"a7aeffa2-475a-47d0-9556-bee3e5534928","Type":"ContainerStarted","Data":"0bc48070c3c7a64ec47b5bd86c433cbd8e7f8200abb2259df4bc4c6710012d18"}
Sep 16 16:47:55.291837 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:55.291509    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv"
Sep 16 16:47:55.292818 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:55.292793    2556 generic.go:356] "Generic (PLEG): container finished" podID="26e6d99e-b725-4b3f-b799-9f804d651965" containerID="8104be8053e4828704e33cc66574458311325e4b0b442e44afc41a162c5d9de3" exitCode=0
Sep 16 16:47:55.292922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:55.292843    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw"
Sep 16 16:47:55.292922 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:55.292875    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw" event={"ID":"26e6d99e-b725-4b3f-b799-9f804d651965","Type":"ContainerDied","Data":"8104be8053e4828704e33cc66574458311325e4b0b442e44afc41a162c5d9de3"}
Sep 16 16:47:55.293022 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:55.292921    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw" event={"ID":"26e6d99e-b725-4b3f-b799-9f804d651965","Type":"ContainerDied","Data":"72d2679be99f9359f4e447efe0e6390b96691a9a0420a9574a36ba27a5a01e11"}
Sep 16 16:47:55.293022 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:55.292966    2556 scope.go:117] "RemoveContainer" containerID="8104be8053e4828704e33cc66574458311325e4b0b442e44afc41a162c5d9de3"
Sep 16 16:47:55.301252 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:55.301080    2556 scope.go:117] "RemoveContainer" containerID="8104be8053e4828704e33cc66574458311325e4b0b442e44afc41a162c5d9de3"
Sep 16 16:47:55.301377 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:47:55.301360    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8104be8053e4828704e33cc66574458311325e4b0b442e44afc41a162c5d9de3\": container with ID starting with 8104be8053e4828704e33cc66574458311325e4b0b442e44afc41a162c5d9de3 not found: ID does not exist" containerID="8104be8053e4828704e33cc66574458311325e4b0b442e44afc41a162c5d9de3"
Sep 16 16:47:55.301427 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:55.301384    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8104be8053e4828704e33cc66574458311325e4b0b442e44afc41a162c5d9de3"} err="failed to get container status \"8104be8053e4828704e33cc66574458311325e4b0b442e44afc41a162c5d9de3\": rpc error: code = NotFound desc = could not find container \"8104be8053e4828704e33cc66574458311325e4b0b442e44afc41a162c5d9de3\": container with ID starting with 8104be8053e4828704e33cc66574458311325e4b0b442e44afc41a162c5d9de3 not found: ID does not exist"
Sep 16 16:47:55.334639 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:55.334599    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv" podStartSLOduration=1.334584171 podStartE2EDuration="1.334584171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:47:55.333152131 +0000 UTC m=+675.095113591" watchObservedRunningTime="2026-09-16 16:47:55.334584171 +0000 UTC m=+675.096545630"
Sep 16 16:47:55.364657 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:55.364633    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw"]
Sep 16 16:47:55.371107 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:55.371085    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d89fw"]
Sep 16 16:47:56.848105 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:47:56.848070    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26e6d99e-b725-4b3f-b799-9f804d651965" path="/var/lib/kubelet/pods/26e6d99e-b725-4b3f-b799-9f804d651965/volumes"
Sep 16 16:48:06.301609 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:06.301576    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv"
Sep 16 16:48:06.356817 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:06.356737    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf"]
Sep 16 16:48:06.357024 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:06.356979    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf" podUID="784812d2-d5ea-4d0f-a92b-ef1dfab9f526" containerName="manager" containerID="cri-o://377c9b78554e73d53a0e8e429e44d6891e31b41517e1561e553fad05f21f3c1d" gracePeriod=10
Sep 16 16:48:06.602615 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:06.602591    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf"
Sep 16 16:48:06.660867 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:06.660782    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/784812d2-d5ea-4d0f-a92b-ef1dfab9f526-extensions-socket-volume\") pod \"784812d2-d5ea-4d0f-a92b-ef1dfab9f526\" (UID: \"784812d2-d5ea-4d0f-a92b-ef1dfab9f526\") "
Sep 16 16:48:06.660867 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:06.660820    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk4p6\" (UniqueName: \"kubernetes.io/projected/784812d2-d5ea-4d0f-a92b-ef1dfab9f526-kube-api-access-rk4p6\") pod \"784812d2-d5ea-4d0f-a92b-ef1dfab9f526\" (UID: \"784812d2-d5ea-4d0f-a92b-ef1dfab9f526\") "
Sep 16 16:48:06.661197 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:06.661172    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784812d2-d5ea-4d0f-a92b-ef1dfab9f526-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "784812d2-d5ea-4d0f-a92b-ef1dfab9f526" (UID: "784812d2-d5ea-4d0f-a92b-ef1dfab9f526"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:48:06.662880 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:06.662855    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784812d2-d5ea-4d0f-a92b-ef1dfab9f526-kube-api-access-rk4p6" (OuterVolumeSpecName: "kube-api-access-rk4p6") pod "784812d2-d5ea-4d0f-a92b-ef1dfab9f526" (UID: "784812d2-d5ea-4d0f-a92b-ef1dfab9f526"). InnerVolumeSpecName "kube-api-access-rk4p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:48:06.761739 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:06.761707    2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/784812d2-d5ea-4d0f-a92b-ef1dfab9f526-extensions-socket-volume\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:48:06.761739 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:06.761733    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk4p6\" (UniqueName: \"kubernetes.io/projected/784812d2-d5ea-4d0f-a92b-ef1dfab9f526-kube-api-access-rk4p6\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:48:07.350724 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:07.350687    2556 generic.go:356] "Generic (PLEG): container finished" podID="784812d2-d5ea-4d0f-a92b-ef1dfab9f526" containerID="377c9b78554e73d53a0e8e429e44d6891e31b41517e1561e553fad05f21f3c1d" exitCode=0
Sep 16 16:48:07.351133 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:07.350734    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf"
Sep 16 16:48:07.351133 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:07.350771    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf" event={"ID":"784812d2-d5ea-4d0f-a92b-ef1dfab9f526","Type":"ContainerDied","Data":"377c9b78554e73d53a0e8e429e44d6891e31b41517e1561e553fad05f21f3c1d"}
Sep 16 16:48:07.351133 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:07.350821    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf" event={"ID":"784812d2-d5ea-4d0f-a92b-ef1dfab9f526","Type":"ContainerDied","Data":"0cda5c3bdc07ec5191afe7d4fc7358ea7651a45c530b275877216f4eaaeb8a6a"}
Sep 16 16:48:07.351133 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:07.350846    2556 scope.go:117] "RemoveContainer" containerID="377c9b78554e73d53a0e8e429e44d6891e31b41517e1561e553fad05f21f3c1d"
Sep 16 16:48:07.358870 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:07.358713    2556 scope.go:117] "RemoveContainer" containerID="377c9b78554e73d53a0e8e429e44d6891e31b41517e1561e553fad05f21f3c1d"
Sep 16 16:48:07.359027 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:48:07.359009    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377c9b78554e73d53a0e8e429e44d6891e31b41517e1561e553fad05f21f3c1d\": container with ID starting with 377c9b78554e73d53a0e8e429e44d6891e31b41517e1561e553fad05f21f3c1d not found: ID does not exist" containerID="377c9b78554e73d53a0e8e429e44d6891e31b41517e1561e553fad05f21f3c1d"
Sep 16 16:48:07.359067 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:07.359036    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377c9b78554e73d53a0e8e429e44d6891e31b41517e1561e553fad05f21f3c1d"} err="failed to get container status \"377c9b78554e73d53a0e8e429e44d6891e31b41517e1561e553fad05f21f3c1d\": rpc error: code = NotFound desc = could not find container \"377c9b78554e73d53a0e8e429e44d6891e31b41517e1561e553fad05f21f3c1d\": container with ID starting with 377c9b78554e73d53a0e8e429e44d6891e31b41517e1561e553fad05f21f3c1d not found: ID does not exist"
Sep 16 16:48:07.375556 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:07.375525    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf"]
Sep 16 16:48:07.382288 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:07.382266    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-j79xf"]
Sep 16 16:48:08.847499 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:08.847466    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="784812d2-d5ea-4d0f-a92b-ef1dfab9f526" path="/var/lib/kubelet/pods/784812d2-d5ea-4d0f-a92b-ef1dfab9f526/volumes"
Sep 16 16:48:27.107925 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:27.107889    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nkdwj"]
Sep 16 16:48:27.108464 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:27.108202    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26e6d99e-b725-4b3f-b799-9f804d651965" containerName="manager"
Sep 16 16:48:27.108464 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:27.108214    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="26e6d99e-b725-4b3f-b799-9f804d651965" containerName="manager"
Sep 16 16:48:27.108464 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:27.108225    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784812d2-d5ea-4d0f-a92b-ef1dfab9f526" containerName="manager"
Sep 16 16:48:27.108464 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:27.108231    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="784812d2-d5ea-4d0f-a92b-ef1dfab9f526" containerName="manager"
Sep 16 16:48:27.108464 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:27.108278    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="784812d2-d5ea-4d0f-a92b-ef1dfab9f526" containerName="manager"
Sep 16 16:48:27.108464 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:27.108288    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="26e6d99e-b725-4b3f-b799-9f804d651965" containerName="manager"
Sep 16 16:48:27.112713 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:27.112692    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-nkdwj"
Sep 16 16:48:27.115206 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:27.115185    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kkrlh\""
Sep 16 16:48:27.115319 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:27.115191    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 16:48:27.120829 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:27.120806    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nkdwj"]
Sep 16 16:48:27.206844 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:27.206807    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nkdwj"]
Sep 16 16:48:27.225477 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:27.225442    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzf5k\" (UniqueName: \"kubernetes.io/projected/0d9dcdd3-84da-4002-aa95-7aa5202626a3-kube-api-access-lzf5k\") pod \"limitador-limitador-8588749977-nkdwj\" (UID: \"0d9dcdd3-84da-4002-aa95-7aa5202626a3\") " pod="kuadrant-system/limitador-limitador-8588749977-nkdwj"
Sep 16 16:48:27.225608 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:27.225580    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0d9dcdd3-84da-4002-aa95-7aa5202626a3-config-file\") pod \"limitador-limitador-8588749977-nkdwj\" (UID: \"0d9dcdd3-84da-4002-aa95-7aa5202626a3\") " pod="kuadrant-system/limitador-limitador-8588749977-nkdwj"
Sep 16 16:48:27.326001 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:27.325968    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0d9dcdd3-84da-4002-aa95-7aa5202626a3-config-file\") pod \"limitador-limitador-8588749977-nkdwj\" (UID: \"0d9dcdd3-84da-4002-aa95-7aa5202626a3\") " pod="kuadrant-system/limitador-limitador-8588749977-nkdwj"
Sep 16 16:48:27.326157 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:27.326016    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzf5k\" (UniqueName: \"kubernetes.io/projected/0d9dcdd3-84da-4002-aa95-7aa5202626a3-kube-api-access-lzf5k\") pod \"limitador-limitador-8588749977-nkdwj\" (UID: \"0d9dcdd3-84da-4002-aa95-7aa5202626a3\") " pod="kuadrant-system/limitador-limitador-8588749977-nkdwj"
Sep 16 16:48:27.326625 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:27.326605    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0d9dcdd3-84da-4002-aa95-7aa5202626a3-config-file\") pod \"limitador-limitador-8588749977-nkdwj\" (UID: \"0d9dcdd3-84da-4002-aa95-7aa5202626a3\") " pod="kuadrant-system/limitador-limitador-8588749977-nkdwj"
Sep 16 16:48:27.334134 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:27.334097    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzf5k\" (UniqueName: \"kubernetes.io/projected/0d9dcdd3-84da-4002-aa95-7aa5202626a3-kube-api-access-lzf5k\") pod \"limitador-limitador-8588749977-nkdwj\" (UID: \"0d9dcdd3-84da-4002-aa95-7aa5202626a3\") " pod="kuadrant-system/limitador-limitador-8588749977-nkdwj"
Sep 16 16:48:27.423062 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:27.422987    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-nkdwj"
Sep 16 16:48:27.696855 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:27.696832    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nkdwj"]
Sep 16 16:48:27.698810 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:48:27.698782    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d9dcdd3_84da_4002_aa95_7aa5202626a3.slice/crio-8110a8fed85b2b1812ee1eb0ccfaf0f0105ea2a9e1b32cd3ba45ac8a54dd0f42 WatchSource:0}: Error finding container 8110a8fed85b2b1812ee1eb0ccfaf0f0105ea2a9e1b32cd3ba45ac8a54dd0f42: Status 404 returned error can't find the container with id 8110a8fed85b2b1812ee1eb0ccfaf0f0105ea2a9e1b32cd3ba45ac8a54dd0f42
Sep 16 16:48:28.446870 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:28.446836    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-nkdwj" event={"ID":"0d9dcdd3-84da-4002-aa95-7aa5202626a3","Type":"ContainerStarted","Data":"8110a8fed85b2b1812ee1eb0ccfaf0f0105ea2a9e1b32cd3ba45ac8a54dd0f42"}
Sep 16 16:48:29.452636 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:29.452600    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-nkdwj" event={"ID":"0d9dcdd3-84da-4002-aa95-7aa5202626a3","Type":"ContainerStarted","Data":"50c910745617e187a28eef1c5b451709917fa6b5357f544d81d39b521e3c8f2f"}
Sep 16 16:48:29.453015 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:29.452721    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-nkdwj"
Sep 16 16:48:29.469996 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:29.469935    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-nkdwj" podStartSLOduration=1.115376663 podStartE2EDuration="2.469920729s" totalImagesPullingTime="1.354544066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:48:29.46846955 +0000 UTC m=+709.230430996" watchObservedRunningTime="2026-09-16 16:48:29.469920729 +0000 UTC m=+709.231882176"
Sep 16 16:48:40.460148 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:40.460066    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-nkdwj"
Sep 16 16:48:41.196020 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:41.195984    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nkdwj"]
Sep 16 16:48:41.196178 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:41.196150    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-nkdwj" podUID="0d9dcdd3-84da-4002-aa95-7aa5202626a3" containerName="limitador" containerID="cri-o://50c910745617e187a28eef1c5b451709917fa6b5357f544d81d39b521e3c8f2f" gracePeriod=30
Sep 16 16:48:41.744619 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:41.744589    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-nkdwj"
Sep 16 16:48:41.835934 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:41.835845    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0d9dcdd3-84da-4002-aa95-7aa5202626a3-config-file\") pod \"0d9dcdd3-84da-4002-aa95-7aa5202626a3\" (UID: \"0d9dcdd3-84da-4002-aa95-7aa5202626a3\") "
Sep 16 16:48:41.835934 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:41.835917    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzf5k\" (UniqueName: \"kubernetes.io/projected/0d9dcdd3-84da-4002-aa95-7aa5202626a3-kube-api-access-lzf5k\") pod \"0d9dcdd3-84da-4002-aa95-7aa5202626a3\" (UID: \"0d9dcdd3-84da-4002-aa95-7aa5202626a3\") "
Sep 16 16:48:41.836269 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:41.836243    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d9dcdd3-84da-4002-aa95-7aa5202626a3-config-file" (OuterVolumeSpecName: "config-file") pod "0d9dcdd3-84da-4002-aa95-7aa5202626a3" (UID: "0d9dcdd3-84da-4002-aa95-7aa5202626a3"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:48:41.837918 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:41.837890    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d9dcdd3-84da-4002-aa95-7aa5202626a3-kube-api-access-lzf5k" (OuterVolumeSpecName: "kube-api-access-lzf5k") pod "0d9dcdd3-84da-4002-aa95-7aa5202626a3" (UID: "0d9dcdd3-84da-4002-aa95-7aa5202626a3"). InnerVolumeSpecName "kube-api-access-lzf5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:48:41.937291 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:41.937255    2556 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0d9dcdd3-84da-4002-aa95-7aa5202626a3-config-file\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:48:41.937291 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:41.937286    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzf5k\" (UniqueName: \"kubernetes.io/projected/0d9dcdd3-84da-4002-aa95-7aa5202626a3-kube-api-access-lzf5k\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:48:42.521796 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:42.521760    2556 generic.go:356] "Generic (PLEG): container finished" podID="0d9dcdd3-84da-4002-aa95-7aa5202626a3" containerID="50c910745617e187a28eef1c5b451709917fa6b5357f544d81d39b521e3c8f2f" exitCode=0
Sep 16 16:48:42.521989 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:42.521808    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-nkdwj"
Sep 16 16:48:42.521989 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:42.521845    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-nkdwj" event={"ID":"0d9dcdd3-84da-4002-aa95-7aa5202626a3","Type":"ContainerDied","Data":"50c910745617e187a28eef1c5b451709917fa6b5357f544d81d39b521e3c8f2f"}
Sep 16 16:48:42.521989 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:42.521897    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-nkdwj" event={"ID":"0d9dcdd3-84da-4002-aa95-7aa5202626a3","Type":"ContainerDied","Data":"8110a8fed85b2b1812ee1eb0ccfaf0f0105ea2a9e1b32cd3ba45ac8a54dd0f42"}
Sep 16 16:48:42.521989 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:42.521920    2556 scope.go:117] "RemoveContainer" containerID="50c910745617e187a28eef1c5b451709917fa6b5357f544d81d39b521e3c8f2f"
Sep 16 16:48:42.530655 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:42.530638    2556 scope.go:117] "RemoveContainer" containerID="50c910745617e187a28eef1c5b451709917fa6b5357f544d81d39b521e3c8f2f"
Sep 16 16:48:42.530905 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:48:42.530888    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50c910745617e187a28eef1c5b451709917fa6b5357f544d81d39b521e3c8f2f\": container with ID starting with 50c910745617e187a28eef1c5b451709917fa6b5357f544d81d39b521e3c8f2f not found: ID does not exist" containerID="50c910745617e187a28eef1c5b451709917fa6b5357f544d81d39b521e3c8f2f"
Sep 16 16:48:42.530969 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:42.530914    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50c910745617e187a28eef1c5b451709917fa6b5357f544d81d39b521e3c8f2f"} err="failed to get container status \"50c910745617e187a28eef1c5b451709917fa6b5357f544d81d39b521e3c8f2f\": rpc error: code = NotFound desc = could not find container \"50c910745617e187a28eef1c5b451709917fa6b5357f544d81d39b521e3c8f2f\": container with ID starting with 50c910745617e187a28eef1c5b451709917fa6b5357f544d81d39b521e3c8f2f not found: ID does not exist"
Sep 16 16:48:42.543563 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:42.543540    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nkdwj"]
Sep 16 16:48:42.548158 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:42.548139    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nkdwj"]
Sep 16 16:48:42.848723 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:42.848646    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d9dcdd3-84da-4002-aa95-7aa5202626a3" path="/var/lib/kubelet/pods/0d9dcdd3-84da-4002-aa95-7aa5202626a3/volumes"
Sep 16 16:48:44.838069 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:44.838038    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-c4m7h"]
Sep 16 16:48:44.838434 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:44.838352    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d9dcdd3-84da-4002-aa95-7aa5202626a3" containerName="limitador"
Sep 16 16:48:44.838434 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:44.838362    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d9dcdd3-84da-4002-aa95-7aa5202626a3" containerName="limitador"
Sep 16 16:48:44.838434 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:44.838423    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d9dcdd3-84da-4002-aa95-7aa5202626a3" containerName="limitador"
Sep 16 16:48:44.842772 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:44.842744    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c4m7h"
Sep 16 16:48:44.845404 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:44.845381    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 16 16:48:44.845535 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:44.845384    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 16 16:48:44.845535 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:44.845384    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 16 16:48:44.846086 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:44.846067    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-56lt4\""
Sep 16 16:48:44.849437 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:44.849417    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-c4m7h"]
Sep 16 16:48:44.960847 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:44.960818    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7df80f59-1527-4045-98d0-32cbb77dc7db-data\") pod \"postgres-69fb4594cf-c4m7h\" (UID: \"7df80f59-1527-4045-98d0-32cbb77dc7db\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c4m7h"
Sep 16 16:48:44.961017 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:44.960857    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfpkp\" (UniqueName: \"kubernetes.io/projected/7df80f59-1527-4045-98d0-32cbb77dc7db-kube-api-access-gfpkp\") pod \"postgres-69fb4594cf-c4m7h\" (UID: \"7df80f59-1527-4045-98d0-32cbb77dc7db\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c4m7h"
Sep 16 16:48:45.061463 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:45.061425    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7df80f59-1527-4045-98d0-32cbb77dc7db-data\") pod \"postgres-69fb4594cf-c4m7h\" (UID: \"7df80f59-1527-4045-98d0-32cbb77dc7db\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c4m7h"
Sep 16 16:48:45.061463 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:45.061470    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfpkp\" (UniqueName: \"kubernetes.io/projected/7df80f59-1527-4045-98d0-32cbb77dc7db-kube-api-access-gfpkp\") pod \"postgres-69fb4594cf-c4m7h\" (UID: \"7df80f59-1527-4045-98d0-32cbb77dc7db\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c4m7h"
Sep 16 16:48:45.061812 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:45.061793    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7df80f59-1527-4045-98d0-32cbb77dc7db-data\") pod \"postgres-69fb4594cf-c4m7h\" (UID: \"7df80f59-1527-4045-98d0-32cbb77dc7db\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c4m7h"
Sep 16 16:48:45.070113 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:45.070086    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfpkp\" (UniqueName: \"kubernetes.io/projected/7df80f59-1527-4045-98d0-32cbb77dc7db-kube-api-access-gfpkp\") pod \"postgres-69fb4594cf-c4m7h\" (UID: \"7df80f59-1527-4045-98d0-32cbb77dc7db\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c4m7h"
Sep 16 16:48:45.154687 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:45.154604    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c4m7h"
Sep 16 16:48:45.423863 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:45.423834    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-c4m7h"]
Sep 16 16:48:45.425669 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:48:45.425640    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df80f59_1527_4045_98d0_32cbb77dc7db.slice/crio-687c47cb6b54961b47132a53b5ebe71a82876215256d1ee39d2c3aa876b06727 WatchSource:0}: Error finding container 687c47cb6b54961b47132a53b5ebe71a82876215256d1ee39d2c3aa876b06727: Status 404 returned error can't find the container with id 687c47cb6b54961b47132a53b5ebe71a82876215256d1ee39d2c3aa876b06727
Sep 16 16:48:45.536640 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:45.536606    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c4m7h" event={"ID":"7df80f59-1527-4045-98d0-32cbb77dc7db","Type":"ContainerStarted","Data":"687c47cb6b54961b47132a53b5ebe71a82876215256d1ee39d2c3aa876b06727"}
Sep 16 16:48:50.564134 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:50.564089    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c4m7h" event={"ID":"7df80f59-1527-4045-98d0-32cbb77dc7db","Type":"ContainerStarted","Data":"9a2a3b83016f0e53ca002d1ebf1e50f29649c5c6705302db6add2f4b85629fe8"}
Sep 16 16:48:50.564519 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:50.564168    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c4m7h"
Sep 16 16:48:50.579914 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:50.579869    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c4m7h" podStartSLOduration=1.8268360719999999 podStartE2EDuration="6.57985372s" totalImagesPullingTime="4.753017648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:48:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:48:50.578542659 +0000 UTC m=+730.340504105" watchObservedRunningTime="2026-09-16 16:48:50.57985372 +0000 UTC m=+730.341815166"
Sep 16 16:48:56.598167 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:56.598129    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c4m7h"
Sep 16 16:48:57.210093 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:57.210060    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-hlrm8"]
Sep 16 16:48:57.215241 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:57.215220    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-hlrm8"
Sep 16 16:48:57.217968 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:57.217886    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 16 16:48:57.218106 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:57.217975    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-96js7\""
Sep 16 16:48:57.219888 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:57.219869    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-hlrm8"]
Sep 16 16:48:57.341973 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:57.341929    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-hlrm8"]
Sep 16 16:48:57.342169 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:48:57.342150    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-28sjp tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-hlrm8" podUID="17c32959-a2ad-4c1b-b584-db68739f2547"
Sep 16 16:48:57.361747 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:57.361718    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28sjp\" (UniqueName: \"kubernetes.io/projected/17c32959-a2ad-4c1b-b584-db68739f2547-kube-api-access-28sjp\") pod \"authorino-6d4dfc649f-hlrm8\" (UID: \"17c32959-a2ad-4c1b-b584-db68739f2547\") " pod="kuadrant-system/authorino-6d4dfc649f-hlrm8"
Sep 16 16:48:57.361860 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:57.361817    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/17c32959-a2ad-4c1b-b584-db68739f2547-tls-cert\") pod \"authorino-6d4dfc649f-hlrm8\" (UID: \"17c32959-a2ad-4c1b-b584-db68739f2547\") " pod="kuadrant-system/authorino-6d4dfc649f-hlrm8"
Sep 16 16:48:57.463096 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:57.463023    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/17c32959-a2ad-4c1b-b584-db68739f2547-tls-cert\") pod \"authorino-6d4dfc649f-hlrm8\" (UID: \"17c32959-a2ad-4c1b-b584-db68739f2547\") " pod="kuadrant-system/authorino-6d4dfc649f-hlrm8"
Sep 16 16:48:57.463096 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:57.463072    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28sjp\" (UniqueName: \"kubernetes.io/projected/17c32959-a2ad-4c1b-b584-db68739f2547-kube-api-access-28sjp\") pod \"authorino-6d4dfc649f-hlrm8\" (UID: \"17c32959-a2ad-4c1b-b584-db68739f2547\") " pod="kuadrant-system/authorino-6d4dfc649f-hlrm8"
Sep 16 16:48:57.465439 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:57.465413    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/17c32959-a2ad-4c1b-b584-db68739f2547-tls-cert\") pod \"authorino-6d4dfc649f-hlrm8\" (UID: \"17c32959-a2ad-4c1b-b584-db68739f2547\") " pod="kuadrant-system/authorino-6d4dfc649f-hlrm8"
Sep 16 16:48:57.471038 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:57.471015    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28sjp\" (UniqueName: \"kubernetes.io/projected/17c32959-a2ad-4c1b-b584-db68739f2547-kube-api-access-28sjp\") pod \"authorino-6d4dfc649f-hlrm8\" (UID: \"17c32959-a2ad-4c1b-b584-db68739f2547\") " pod="kuadrant-system/authorino-6d4dfc649f-hlrm8"
Sep 16 16:48:57.594048 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:57.594024    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-hlrm8"
Sep 16 16:48:57.600712 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:57.600688    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-hlrm8"
Sep 16 16:48:57.766375 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:57.766309    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/17c32959-a2ad-4c1b-b584-db68739f2547-tls-cert\") pod \"17c32959-a2ad-4c1b-b584-db68739f2547\" (UID: \"17c32959-a2ad-4c1b-b584-db68739f2547\") "
Sep 16 16:48:57.766375 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:57.766342    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28sjp\" (UniqueName: \"kubernetes.io/projected/17c32959-a2ad-4c1b-b584-db68739f2547-kube-api-access-28sjp\") pod \"17c32959-a2ad-4c1b-b584-db68739f2547\" (UID: \"17c32959-a2ad-4c1b-b584-db68739f2547\") "
Sep 16 16:48:57.768419 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:57.768388    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c32959-a2ad-4c1b-b584-db68739f2547-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "17c32959-a2ad-4c1b-b584-db68739f2547" (UID: "17c32959-a2ad-4c1b-b584-db68739f2547"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:48:57.768503 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:57.768473    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c32959-a2ad-4c1b-b584-db68739f2547-kube-api-access-28sjp" (OuterVolumeSpecName: "kube-api-access-28sjp") pod "17c32959-a2ad-4c1b-b584-db68739f2547" (UID: "17c32959-a2ad-4c1b-b584-db68739f2547"). InnerVolumeSpecName "kube-api-access-28sjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:48:57.867558 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:57.867528    2556 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/17c32959-a2ad-4c1b-b584-db68739f2547-tls-cert\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:48:57.867558 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:57.867556    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28sjp\" (UniqueName: \"kubernetes.io/projected/17c32959-a2ad-4c1b-b584-db68739f2547-kube-api-access-28sjp\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:48:58.599064 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:58.599033    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-hlrm8"
Sep 16 16:48:58.629376 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:58.629348    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-hlrm8"]
Sep 16 16:48:58.633242 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:58.633218    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-hlrm8"]
Sep 16 16:48:58.848907 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:48:58.848871    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17c32959-a2ad-4c1b-b584-db68739f2547" path="/var/lib/kubelet/pods/17c32959-a2ad-4c1b-b584-db68739f2547/volumes"
Sep 16 16:49:03.783657 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:03.783625    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-65cbd6bdf6-c2ckz"]
Sep 16 16:49:03.849758 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:03.849724    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65cbd6bdf6-c2ckz"]
Sep 16 16:49:03.849925 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:03.849836    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz"
Sep 16 16:49:03.853322 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:03.853275    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 16 16:49:03.853467 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:03.853328    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 16 16:49:03.853467 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:03.853344    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 16 16:49:03.853467 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:03.853278    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-8kj7n\""
Sep 16 16:49:03.917096 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:03.917068    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce772390-8023-4e87-b7b5-ed49bc609d59-metrics-certs\") pod \"maas-controller-65cbd6bdf6-c2ckz\" (UID: \"ce772390-8023-4e87-b7b5-ed49bc609d59\") " pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz"
Sep 16 16:49:03.917279 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:03.917113    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce772390-8023-4e87-b7b5-ed49bc609d59-webhook-cert\") pod \"maas-controller-65cbd6bdf6-c2ckz\" (UID: \"ce772390-8023-4e87-b7b5-ed49bc609d59\") " pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz"
Sep 16 16:49:03.917279 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:03.917151    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqmsl\" (UniqueName: \"kubernetes.io/projected/ce772390-8023-4e87-b7b5-ed49bc609d59-kube-api-access-xqmsl\") pod \"maas-controller-65cbd6bdf6-c2ckz\" (UID: \"ce772390-8023-4e87-b7b5-ed49bc609d59\") " pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz"
Sep 16 16:49:04.017548 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:04.017519    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce772390-8023-4e87-b7b5-ed49bc609d59-metrics-certs\") pod \"maas-controller-65cbd6bdf6-c2ckz\" (UID: \"ce772390-8023-4e87-b7b5-ed49bc609d59\") " pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz"
Sep 16 16:49:04.017718 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:04.017564    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce772390-8023-4e87-b7b5-ed49bc609d59-webhook-cert\") pod \"maas-controller-65cbd6bdf6-c2ckz\" (UID: \"ce772390-8023-4e87-b7b5-ed49bc609d59\") " pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz"
Sep 16 16:49:04.017718 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:04.017603    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqmsl\" (UniqueName: \"kubernetes.io/projected/ce772390-8023-4e87-b7b5-ed49bc609d59-kube-api-access-xqmsl\") pod \"maas-controller-65cbd6bdf6-c2ckz\" (UID: \"ce772390-8023-4e87-b7b5-ed49bc609d59\") " pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz"
Sep 16 16:49:04.019990 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:04.019961    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce772390-8023-4e87-b7b5-ed49bc609d59-webhook-cert\") pod \"maas-controller-65cbd6bdf6-c2ckz\" (UID: \"ce772390-8023-4e87-b7b5-ed49bc609d59\") " pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz"
Sep 16 16:49:04.019990 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:04.019977    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce772390-8023-4e87-b7b5-ed49bc609d59-metrics-certs\") pod \"maas-controller-65cbd6bdf6-c2ckz\" (UID: \"ce772390-8023-4e87-b7b5-ed49bc609d59\") " pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz"
Sep 16 16:49:04.026371 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:04.026351    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqmsl\" (UniqueName: \"kubernetes.io/projected/ce772390-8023-4e87-b7b5-ed49bc609d59-kube-api-access-xqmsl\") pod \"maas-controller-65cbd6bdf6-c2ckz\" (UID: \"ce772390-8023-4e87-b7b5-ed49bc609d59\") " pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz"
Sep 16 16:49:04.160112 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:04.160026    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz"
Sep 16 16:49:04.430271 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:04.430246    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-65cbd6bdf6-c2ckz"]
Sep 16 16:49:04.432431 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:49:04.432403    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce772390_8023_4e87_b7b5_ed49bc609d59.slice/crio-407d4590c84bbb7d8f13652406d8d7abed1c16d7bc000b234d793420c81d4a84 WatchSource:0}: Error finding container 407d4590c84bbb7d8f13652406d8d7abed1c16d7bc000b234d793420c81d4a84: Status 404 returned error can't find the container with id 407d4590c84bbb7d8f13652406d8d7abed1c16d7bc000b234d793420c81d4a84
Sep 16 16:49:04.624556 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:04.624522    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz" event={"ID":"ce772390-8023-4e87-b7b5-ed49bc609d59","Type":"ContainerStarted","Data":"407d4590c84bbb7d8f13652406d8d7abed1c16d7bc000b234d793420c81d4a84"}
Sep 16 16:49:07.647998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:07.647933    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz" event={"ID":"ce772390-8023-4e87-b7b5-ed49bc609d59","Type":"ContainerStarted","Data":"033c78c72caf3f344b26571a1fc87b0b82e031af5f8a424a7058dff786180569"}
Sep 16 16:49:07.648393 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:07.648082    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz"
Sep 16 16:49:18.654599 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:18.654564    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz"
Sep 16 16:49:18.672708 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:18.672658    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz" podStartSLOduration=13.035544769 podStartE2EDuration="15.672645021s" totalImagesPullingTime="2.637100252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:49:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:49:07.680281219 +0000 UTC m=+747.442242666" watchObservedRunningTime="2026-09-16 16:49:18.672645021 +0000 UTC m=+758.434606468"
Sep 16 16:49:19.231183 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.231146    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65cbd6bdf6-c2ckz"]
Sep 16 16:49:19.231426 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.231399    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz" podUID="ce772390-8023-4e87-b7b5-ed49bc609d59" containerName="manager" containerID="cri-o://033c78c72caf3f344b26571a1fc87b0b82e031af5f8a424a7058dff786180569" gracePeriod=30
Sep 16 16:49:19.476603 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.476579    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz"
Sep 16 16:49:19.548998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.548902    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce772390-8023-4e87-b7b5-ed49bc609d59-metrics-certs\") pod \"ce772390-8023-4e87-b7b5-ed49bc609d59\" (UID: \"ce772390-8023-4e87-b7b5-ed49bc609d59\") "
Sep 16 16:49:19.548998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.548978    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqmsl\" (UniqueName: \"kubernetes.io/projected/ce772390-8023-4e87-b7b5-ed49bc609d59-kube-api-access-xqmsl\") pod \"ce772390-8023-4e87-b7b5-ed49bc609d59\" (UID: \"ce772390-8023-4e87-b7b5-ed49bc609d59\") "
Sep 16 16:49:19.549214 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.549091    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce772390-8023-4e87-b7b5-ed49bc609d59-webhook-cert\") pod \"ce772390-8023-4e87-b7b5-ed49bc609d59\" (UID: \"ce772390-8023-4e87-b7b5-ed49bc609d59\") "
Sep 16 16:49:19.551146 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.551120    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce772390-8023-4e87-b7b5-ed49bc609d59-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ce772390-8023-4e87-b7b5-ed49bc609d59" (UID: "ce772390-8023-4e87-b7b5-ed49bc609d59"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:49:19.551240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.551155    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce772390-8023-4e87-b7b5-ed49bc609d59-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ce772390-8023-4e87-b7b5-ed49bc609d59" (UID: "ce772390-8023-4e87-b7b5-ed49bc609d59"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:49:19.551306 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.551286    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce772390-8023-4e87-b7b5-ed49bc609d59-kube-api-access-xqmsl" (OuterVolumeSpecName: "kube-api-access-xqmsl") pod "ce772390-8023-4e87-b7b5-ed49bc609d59" (UID: "ce772390-8023-4e87-b7b5-ed49bc609d59"). InnerVolumeSpecName "kube-api-access-xqmsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:49:19.649920 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.649884    2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce772390-8023-4e87-b7b5-ed49bc609d59-webhook-cert\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:49:19.649920 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.649912    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce772390-8023-4e87-b7b5-ed49bc609d59-metrics-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:49:19.649920 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.649922    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqmsl\" (UniqueName: \"kubernetes.io/projected/ce772390-8023-4e87-b7b5-ed49bc609d59-kube-api-access-xqmsl\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:49:19.702703 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.702665    2556 generic.go:356] "Generic (PLEG): container finished" podID="ce772390-8023-4e87-b7b5-ed49bc609d59" containerID="033c78c72caf3f344b26571a1fc87b0b82e031af5f8a424a7058dff786180569" exitCode=0
Sep 16 16:49:19.703154 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.702713    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz"
Sep 16 16:49:19.703154 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.702752    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz" event={"ID":"ce772390-8023-4e87-b7b5-ed49bc609d59","Type":"ContainerDied","Data":"033c78c72caf3f344b26571a1fc87b0b82e031af5f8a424a7058dff786180569"}
Sep 16 16:49:19.703154 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.702799    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-65cbd6bdf6-c2ckz" event={"ID":"ce772390-8023-4e87-b7b5-ed49bc609d59","Type":"ContainerDied","Data":"407d4590c84bbb7d8f13652406d8d7abed1c16d7bc000b234d793420c81d4a84"}
Sep 16 16:49:19.703154 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.702825    2556 scope.go:117] "RemoveContainer" containerID="033c78c72caf3f344b26571a1fc87b0b82e031af5f8a424a7058dff786180569"
Sep 16 16:49:19.711173 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.711149    2556 scope.go:117] "RemoveContainer" containerID="033c78c72caf3f344b26571a1fc87b0b82e031af5f8a424a7058dff786180569"
Sep 16 16:49:19.711432 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:49:19.711415    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"033c78c72caf3f344b26571a1fc87b0b82e031af5f8a424a7058dff786180569\": container with ID starting with 033c78c72caf3f344b26571a1fc87b0b82e031af5f8a424a7058dff786180569 not found: ID does not exist" containerID="033c78c72caf3f344b26571a1fc87b0b82e031af5f8a424a7058dff786180569"
Sep 16 16:49:19.711496 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.711440    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"033c78c72caf3f344b26571a1fc87b0b82e031af5f8a424a7058dff786180569"} err="failed to get container status \"033c78c72caf3f344b26571a1fc87b0b82e031af5f8a424a7058dff786180569\": rpc error: code = NotFound desc = could not find container \"033c78c72caf3f344b26571a1fc87b0b82e031af5f8a424a7058dff786180569\": container with ID starting with 033c78c72caf3f344b26571a1fc87b0b82e031af5f8a424a7058dff786180569 not found: ID does not exist"
Sep 16 16:49:19.722077 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.722055    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-65cbd6bdf6-c2ckz"]
Sep 16 16:49:19.725782 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:19.725760    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-65cbd6bdf6-c2ckz"]
Sep 16 16:49:20.847681 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:20.847648    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce772390-8023-4e87-b7b5-ed49bc609d59" path="/var/lib/kubelet/pods/ce772390-8023-4e87-b7b5-ed49bc609d59/volumes"
Sep 16 16:49:40.526909 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.526876    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"]
Sep 16 16:49:40.527355 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.527255    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce772390-8023-4e87-b7b5-ed49bc609d59" containerName="manager"
Sep 16 16:49:40.527355 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.527270    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce772390-8023-4e87-b7b5-ed49bc609d59" containerName="manager"
Sep 16 16:49:40.527428 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.527357    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce772390-8023-4e87-b7b5-ed49bc609d59" containerName="manager"
Sep 16 16:49:40.548151 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.548120    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"]
Sep 16 16:49:40.548275 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.548254    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"
Sep 16 16:49:40.550996 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.550972    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 16:49:40.550996 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.550986    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5qc55\""
Sep 16 16:49:40.551207 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.550993    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 16:49:40.718975 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.718929    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-494fb\" (UniqueName: \"kubernetes.io/projected/5df3cb22-5542-468b-b4e8-53f3dbadf022-kube-api-access-494fb\") pod \"maas-api-5665c7f4f4-mpw4f\" (UID: \"5df3cb22-5542-468b-b4e8-53f3dbadf022\") " pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"
Sep 16 16:49:40.719094 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.718985    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df3cb22-5542-468b-b4e8-53f3dbadf022-openshift-service-ca\") pod \"maas-api-5665c7f4f4-mpw4f\" (UID: \"5df3cb22-5542-468b-b4e8-53f3dbadf022\") " pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"
Sep 16 16:49:40.719094 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.719006    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5df3cb22-5542-468b-b4e8-53f3dbadf022-metrics-certs\") pod \"maas-api-5665c7f4f4-mpw4f\" (UID: \"5df3cb22-5542-468b-b4e8-53f3dbadf022\") " pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"
Sep 16 16:49:40.719094 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.719046    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5df3cb22-5542-468b-b4e8-53f3dbadf022-maas-api-tls\") pod \"maas-api-5665c7f4f4-mpw4f\" (UID: \"5df3cb22-5542-468b-b4e8-53f3dbadf022\") " pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"
Sep 16 16:49:40.819472 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.819392    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5df3cb22-5542-468b-b4e8-53f3dbadf022-maas-api-tls\") pod \"maas-api-5665c7f4f4-mpw4f\" (UID: \"5df3cb22-5542-468b-b4e8-53f3dbadf022\") " pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"
Sep 16 16:49:40.819614 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.819499    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-494fb\" (UniqueName: \"kubernetes.io/projected/5df3cb22-5542-468b-b4e8-53f3dbadf022-kube-api-access-494fb\") pod \"maas-api-5665c7f4f4-mpw4f\" (UID: \"5df3cb22-5542-468b-b4e8-53f3dbadf022\") " pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"
Sep 16 16:49:40.819614 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.819530    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df3cb22-5542-468b-b4e8-53f3dbadf022-openshift-service-ca\") pod \"maas-api-5665c7f4f4-mpw4f\" (UID: \"5df3cb22-5542-468b-b4e8-53f3dbadf022\") " pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"
Sep 16 16:49:40.819614 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.819554    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5df3cb22-5542-468b-b4e8-53f3dbadf022-metrics-certs\") pod \"maas-api-5665c7f4f4-mpw4f\" (UID: \"5df3cb22-5542-468b-b4e8-53f3dbadf022\") " pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"
Sep 16 16:49:40.820238 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.820216    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df3cb22-5542-468b-b4e8-53f3dbadf022-openshift-service-ca\") pod \"maas-api-5665c7f4f4-mpw4f\" (UID: \"5df3cb22-5542-468b-b4e8-53f3dbadf022\") " pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"
Sep 16 16:49:40.821639 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.821619    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 16 16:49:40.821752 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.821623    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 16 16:49:40.828263 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:40.828241    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-494fb\" (UniqueName: \"kubernetes.io/projected/5df3cb22-5542-468b-b4e8-53f3dbadf022-kube-api-access-494fb\") pod \"maas-api-5665c7f4f4-mpw4f\" (UID: \"5df3cb22-5542-468b-b4e8-53f3dbadf022\") " pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"
Sep 16 16:49:40.829627 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:49:40.829598    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Sep 16 16:49:40.829716 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:49:40.829662    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5df3cb22-5542-468b-b4e8-53f3dbadf022-maas-api-tls podName:5df3cb22-5542-468b-b4e8-53f3dbadf022 nodeName:}" failed. No retries permitted until 2026-09-16 16:49:41.329644059 +0000 UTC m=+781.091605486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5df3cb22-5542-468b-b4e8-53f3dbadf022-maas-api-tls") pod "maas-api-5665c7f4f4-mpw4f" (UID: "5df3cb22-5542-468b-b4e8-53f3dbadf022") : secret "maas-api-serving-cert" not found
Sep 16 16:49:40.829779 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:49:40.829726    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 16 16:49:40.829827 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:49:40.829789    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5df3cb22-5542-468b-b4e8-53f3dbadf022-metrics-certs podName:5df3cb22-5542-468b-b4e8-53f3dbadf022 nodeName:}" failed. No retries permitted until 2026-09-16 16:49:41.329771848 +0000 UTC m=+781.091733287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5df3cb22-5542-468b-b4e8-53f3dbadf022-metrics-certs") pod "maas-api-5665c7f4f4-mpw4f" (UID: "5df3cb22-5542-468b-b4e8-53f3dbadf022") : secret "maas-api-metrics-tls" not found
Sep 16 16:49:41.031992 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.031959    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-mpcpf"]
Sep 16 16:49:41.035228 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.035208    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-mpcpf"
Sep 16 16:49:41.037792 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.037770    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 16 16:49:41.038178 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.038159    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-g4st5\""
Sep 16 16:49:41.048454 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.048433    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-mpcpf"]
Sep 16 16:49:41.121912 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.121841    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb67f300-0327-4e84-9998-045e6b646f34-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-mpcpf\" (UID: \"bb67f300-0327-4e84-9998-045e6b646f34\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mpcpf"
Sep 16 16:49:41.122056 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.121970    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz4bm\" (UniqueName: \"kubernetes.io/projected/bb67f300-0327-4e84-9998-045e6b646f34-kube-api-access-jz4bm\") pod \"payload-processing-5b54b7fbb7-mpcpf\" (UID: \"bb67f300-0327-4e84-9998-045e6b646f34\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mpcpf"
Sep 16 16:49:41.222363 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.222329    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb67f300-0327-4e84-9998-045e6b646f34-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-mpcpf\" (UID: \"bb67f300-0327-4e84-9998-045e6b646f34\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mpcpf"
Sep 16 16:49:41.222501 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.222408    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz4bm\" (UniqueName: \"kubernetes.io/projected/bb67f300-0327-4e84-9998-045e6b646f34-kube-api-access-jz4bm\") pod \"payload-processing-5b54b7fbb7-mpcpf\" (UID: \"bb67f300-0327-4e84-9998-045e6b646f34\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mpcpf"
Sep 16 16:49:41.223062 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.223044    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb67f300-0327-4e84-9998-045e6b646f34-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-mpcpf\" (UID: \"bb67f300-0327-4e84-9998-045e6b646f34\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mpcpf"
Sep 16 16:49:41.230642 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.230606    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz4bm\" (UniqueName: \"kubernetes.io/projected/bb67f300-0327-4e84-9998-045e6b646f34-kube-api-access-jz4bm\") pod \"payload-processing-5b54b7fbb7-mpcpf\" (UID: \"bb67f300-0327-4e84-9998-045e6b646f34\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mpcpf"
Sep 16 16:49:41.346525 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.346491    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-mpcpf"
Sep 16 16:49:41.424414 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.424384    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5df3cb22-5542-468b-b4e8-53f3dbadf022-maas-api-tls\") pod \"maas-api-5665c7f4f4-mpw4f\" (UID: \"5df3cb22-5542-468b-b4e8-53f3dbadf022\") " pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"
Sep 16 16:49:41.424552 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.424534    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5df3cb22-5542-468b-b4e8-53f3dbadf022-metrics-certs\") pod \"maas-api-5665c7f4f4-mpw4f\" (UID: \"5df3cb22-5542-468b-b4e8-53f3dbadf022\") " pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"
Sep 16 16:49:41.426764 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.426724    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5df3cb22-5542-468b-b4e8-53f3dbadf022-metrics-certs\") pod \"maas-api-5665c7f4f4-mpw4f\" (UID: \"5df3cb22-5542-468b-b4e8-53f3dbadf022\") " pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"
Sep 16 16:49:41.426933 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.426900    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5df3cb22-5542-468b-b4e8-53f3dbadf022-maas-api-tls\") pod \"maas-api-5665c7f4f4-mpw4f\" (UID: \"5df3cb22-5542-468b-b4e8-53f3dbadf022\") " pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"
Sep 16 16:49:41.460914 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.460890    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5qc55\""
Sep 16 16:49:41.469397 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.469375    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"
Sep 16 16:49:41.641713 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.641681    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-mpcpf"]
Sep 16 16:49:41.644639 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:49:41.644580    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb67f300_0327_4e84_9998_045e6b646f34.slice/crio-81f8d5a6c355724e75976ad0346f1657bf863de8e2c8174f5e9a14ca556e66c2 WatchSource:0}: Error finding container 81f8d5a6c355724e75976ad0346f1657bf863de8e2c8174f5e9a14ca556e66c2: Status 404 returned error can't find the container with id 81f8d5a6c355724e75976ad0346f1657bf863de8e2c8174f5e9a14ca556e66c2
Sep 16 16:49:41.760672 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.760647    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"]
Sep 16 16:49:41.762883 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:49:41.762850    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5df3cb22_5542_468b_b4e8_53f3dbadf022.slice/crio-606736a05dcff0a2891163279654cda08fe359ddc4941bd07592fdce0def1875 WatchSource:0}: Error finding container 606736a05dcff0a2891163279654cda08fe359ddc4941bd07592fdce0def1875: Status 404 returned error can't find the container with id 606736a05dcff0a2891163279654cda08fe359ddc4941bd07592fdce0def1875
Sep 16 16:49:41.796840 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.796812    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-mpcpf" event={"ID":"bb67f300-0327-4e84-9998-045e6b646f34","Type":"ContainerStarted","Data":"81f8d5a6c355724e75976ad0346f1657bf863de8e2c8174f5e9a14ca556e66c2"}
Sep 16 16:49:41.798122 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:41.798098    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f" event={"ID":"5df3cb22-5542-468b-b4e8-53f3dbadf022","Type":"ContainerStarted","Data":"606736a05dcff0a2891163279654cda08fe359ddc4941bd07592fdce0def1875"}
Sep 16 16:49:43.810784 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:43.810744    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f" event={"ID":"5df3cb22-5542-468b-b4e8-53f3dbadf022","Type":"ContainerStarted","Data":"0b96b72294528bb5393fc5435fdae1c0101ff906084c73aabef82776c8ef7478"}
Sep 16 16:49:43.811234 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:43.810846    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"
Sep 16 16:49:43.812455 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:43.812430    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-mpcpf" event={"ID":"bb67f300-0327-4e84-9998-045e6b646f34","Type":"ContainerStarted","Data":"c1095aeadd4e59e1206a6ca16247eed797cf478b44f28c03327fba8eccb1e566"}
Sep 16 16:49:43.812592 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:43.812579    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-mpcpf"
Sep 16 16:49:43.829665 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:43.829610    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f" podStartSLOduration=2.229504111 podStartE2EDuration="3.829590493s" totalImagesPullingTime="1.600086382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:49:43.827104651 +0000 UTC m=+783.589066113" watchObservedRunningTime="2026-09-16 16:49:43.829590493 +0000 UTC m=+783.591551943"
Sep 16 16:49:43.843854 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:43.843801    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-mpcpf" podStartSLOduration=1.170246776 podStartE2EDuration="2.843788299s" totalImagesPullingTime="1.673541523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:49:43.843128565 +0000 UTC m=+783.605090012" watchObservedRunningTime="2026-09-16 16:49:43.843788299 +0000 UTC m=+783.605749746"
Sep 16 16:49:49.823186 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:49.823154    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"
Sep 16 16:49:54.820121 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:49:54.820093    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-mpcpf"
Sep 16 16:50:25.475248 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.475210    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"]
Sep 16 16:50:25.480139 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.480118    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:25.483759 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.483735    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 16 16:50:25.483883 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.483772    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 16 16:50:25.483883 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.483773    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cxtbk\""
Sep 16 16:50:25.483883 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.483787    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 16 16:50:25.489195 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.489170    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"]
Sep 16 16:50:25.573894 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.573861    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf92945-7a38-4b2c-86bc-761d2187a2aa-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg\" (UID: \"2cf92945-7a38-4b2c-86bc-761d2187a2aa\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:25.574068 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.573903    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf92945-7a38-4b2c-86bc-761d2187a2aa-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg\" (UID: \"2cf92945-7a38-4b2c-86bc-761d2187a2aa\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:25.574068 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.573931    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf92945-7a38-4b2c-86bc-761d2187a2aa-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg\" (UID: \"2cf92945-7a38-4b2c-86bc-761d2187a2aa\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:25.574068 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.574047    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf92945-7a38-4b2c-86bc-761d2187a2aa-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg\" (UID: \"2cf92945-7a38-4b2c-86bc-761d2187a2aa\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:25.574228 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.574101    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf92945-7a38-4b2c-86bc-761d2187a2aa-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg\" (UID: \"2cf92945-7a38-4b2c-86bc-761d2187a2aa\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:25.574228 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.574133    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kcc6\" (UniqueName: \"kubernetes.io/projected/2cf92945-7a38-4b2c-86bc-761d2187a2aa-kube-api-access-6kcc6\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg\" (UID: \"2cf92945-7a38-4b2c-86bc-761d2187a2aa\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:25.674603 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.674571    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf92945-7a38-4b2c-86bc-761d2187a2aa-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg\" (UID: \"2cf92945-7a38-4b2c-86bc-761d2187a2aa\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:25.674727 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.674612    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf92945-7a38-4b2c-86bc-761d2187a2aa-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg\" (UID: \"2cf92945-7a38-4b2c-86bc-761d2187a2aa\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:25.674727 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.674631    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kcc6\" (UniqueName: \"kubernetes.io/projected/2cf92945-7a38-4b2c-86bc-761d2187a2aa-kube-api-access-6kcc6\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg\" (UID: \"2cf92945-7a38-4b2c-86bc-761d2187a2aa\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:25.674727 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.674678    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf92945-7a38-4b2c-86bc-761d2187a2aa-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg\" (UID: \"2cf92945-7a38-4b2c-86bc-761d2187a2aa\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:25.674727 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.674720    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf92945-7a38-4b2c-86bc-761d2187a2aa-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg\" (UID: \"2cf92945-7a38-4b2c-86bc-761d2187a2aa\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:25.674879 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.674742    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf92945-7a38-4b2c-86bc-761d2187a2aa-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg\" (UID: \"2cf92945-7a38-4b2c-86bc-761d2187a2aa\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:25.675072 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.675050    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf92945-7a38-4b2c-86bc-761d2187a2aa-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg\" (UID: \"2cf92945-7a38-4b2c-86bc-761d2187a2aa\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:25.675131 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.675095    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2cf92945-7a38-4b2c-86bc-761d2187a2aa-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg\" (UID: \"2cf92945-7a38-4b2c-86bc-761d2187a2aa\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:25.675173 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.675122    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2cf92945-7a38-4b2c-86bc-761d2187a2aa-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg\" (UID: \"2cf92945-7a38-4b2c-86bc-761d2187a2aa\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:25.676986 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.676968    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2cf92945-7a38-4b2c-86bc-761d2187a2aa-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg\" (UID: \"2cf92945-7a38-4b2c-86bc-761d2187a2aa\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:25.677193 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.677179    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cf92945-7a38-4b2c-86bc-761d2187a2aa-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg\" (UID: \"2cf92945-7a38-4b2c-86bc-761d2187a2aa\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:25.683337 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.683315    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kcc6\" (UniqueName: \"kubernetes.io/projected/2cf92945-7a38-4b2c-86bc-761d2187a2aa-kube-api-access-6kcc6\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg\" (UID: \"2cf92945-7a38-4b2c-86bc-761d2187a2aa\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:25.791518 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:25.791446    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:26.062179 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:26.062104    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"]
Sep 16 16:50:26.065413 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:50:26.065383    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf92945_7a38_4b2c_86bc_761d2187a2aa.slice/crio-07f807e3d4cfbe105aa37d18c4c59370d5e16f95a4f772aa2a6f0b5fb3cc0378 WatchSource:0}: Error finding container 07f807e3d4cfbe105aa37d18c4c59370d5e16f95a4f772aa2a6f0b5fb3cc0378: Status 404 returned error can't find the container with id 07f807e3d4cfbe105aa37d18c4c59370d5e16f95a4f772aa2a6f0b5fb3cc0378
Sep 16 16:50:27.002119 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:27.002073    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg" event={"ID":"2cf92945-7a38-4b2c-86bc-761d2187a2aa","Type":"ContainerStarted","Data":"07f807e3d4cfbe105aa37d18c4c59370d5e16f95a4f772aa2a6f0b5fb3cc0378"}
Sep 16 16:50:28.007988 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:28.007935    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg" event={"ID":"2cf92945-7a38-4b2c-86bc-761d2187a2aa","Type":"ContainerStarted","Data":"b925247b16cd6e7f3c2a5239e9d9fdef0cd6b3ea489568f9c37a580ba2933cc5"}
Sep 16 16:50:28.024656 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:28.024615    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg" podStartSLOduration=2.02232229 podStartE2EDuration="3.0246002s" totalImagesPullingTime="1.00227791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:50:28.023644007 +0000 UTC m=+827.785605448" watchObservedRunningTime="2026-09-16 16:50:28.0246002 +0000 UTC m=+827.786561647"
Sep 16 16:50:35.791978 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:35.791921    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:35.792534 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:35.792001    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:35.804659 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:35.804635    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:36.053382 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:36.053304    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg"
Sep 16 16:50:45.981437 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:45.981400    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"]
Sep 16 16:50:45.984977 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:45.984957    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:45.987258 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:45.987239    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 16 16:50:45.996934 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:45.996910    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"]
Sep 16 16:50:46.150270 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:46.150235    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee454389-f8a1-4351-ae5b-2c342a3c63a0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gh99\" (UID: \"ee454389-f8a1-4351-ae5b-2c342a3c63a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:46.150429 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:46.150303    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee454389-f8a1-4351-ae5b-2c342a3c63a0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gh99\" (UID: \"ee454389-f8a1-4351-ae5b-2c342a3c63a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:46.150429 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:46.150330    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb264\" (UniqueName: \"kubernetes.io/projected/ee454389-f8a1-4351-ae5b-2c342a3c63a0-kube-api-access-cb264\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gh99\" (UID: \"ee454389-f8a1-4351-ae5b-2c342a3c63a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:46.150429 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:46.150373    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee454389-f8a1-4351-ae5b-2c342a3c63a0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gh99\" (UID: \"ee454389-f8a1-4351-ae5b-2c342a3c63a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:46.150429 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:46.150425    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee454389-f8a1-4351-ae5b-2c342a3c63a0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gh99\" (UID: \"ee454389-f8a1-4351-ae5b-2c342a3c63a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:46.150610 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:46.150450    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee454389-f8a1-4351-ae5b-2c342a3c63a0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gh99\" (UID: \"ee454389-f8a1-4351-ae5b-2c342a3c63a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:46.251095 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:46.251005    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee454389-f8a1-4351-ae5b-2c342a3c63a0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gh99\" (UID: \"ee454389-f8a1-4351-ae5b-2c342a3c63a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:46.251095 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:46.251058    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee454389-f8a1-4351-ae5b-2c342a3c63a0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gh99\" (UID: \"ee454389-f8a1-4351-ae5b-2c342a3c63a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:46.251095 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:46.251082    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb264\" (UniqueName: \"kubernetes.io/projected/ee454389-f8a1-4351-ae5b-2c342a3c63a0-kube-api-access-cb264\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gh99\" (UID: \"ee454389-f8a1-4351-ae5b-2c342a3c63a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:46.251369 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:46.251126    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee454389-f8a1-4351-ae5b-2c342a3c63a0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gh99\" (UID: \"ee454389-f8a1-4351-ae5b-2c342a3c63a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:46.251369 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:46.251153    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee454389-f8a1-4351-ae5b-2c342a3c63a0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gh99\" (UID: \"ee454389-f8a1-4351-ae5b-2c342a3c63a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:46.251369 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:46.251178    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee454389-f8a1-4351-ae5b-2c342a3c63a0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gh99\" (UID: \"ee454389-f8a1-4351-ae5b-2c342a3c63a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:46.251522 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:46.251447    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee454389-f8a1-4351-ae5b-2c342a3c63a0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gh99\" (UID: \"ee454389-f8a1-4351-ae5b-2c342a3c63a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:46.251522 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:46.251482    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee454389-f8a1-4351-ae5b-2c342a3c63a0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gh99\" (UID: \"ee454389-f8a1-4351-ae5b-2c342a3c63a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:46.251522 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:46.251510    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee454389-f8a1-4351-ae5b-2c342a3c63a0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gh99\" (UID: \"ee454389-f8a1-4351-ae5b-2c342a3c63a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:46.253378 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:46.253358    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee454389-f8a1-4351-ae5b-2c342a3c63a0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gh99\" (UID: \"ee454389-f8a1-4351-ae5b-2c342a3c63a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:46.253638 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:46.253618    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee454389-f8a1-4351-ae5b-2c342a3c63a0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gh99\" (UID: \"ee454389-f8a1-4351-ae5b-2c342a3c63a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:46.261709 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:46.261679    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb264\" (UniqueName: \"kubernetes.io/projected/ee454389-f8a1-4351-ae5b-2c342a3c63a0-kube-api-access-cb264\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9gh99\" (UID: \"ee454389-f8a1-4351-ae5b-2c342a3c63a0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:46.295055 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:46.295023    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:46.568108 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:46.567929    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"]
Sep 16 16:50:46.570187 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:50:46.570151    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee454389_f8a1_4351_ae5b_2c342a3c63a0.slice/crio-ab465298c8c6f801465884c357af38e47c90272c91c4ab340fbc551882870ae4 WatchSource:0}: Error finding container ab465298c8c6f801465884c357af38e47c90272c91c4ab340fbc551882870ae4: Status 404 returned error can't find the container with id ab465298c8c6f801465884c357af38e47c90272c91c4ab340fbc551882870ae4
Sep 16 16:50:47.091548 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:47.091515    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99" event={"ID":"ee454389-f8a1-4351-ae5b-2c342a3c63a0","Type":"ContainerStarted","Data":"0af211fa52baf6bbf898e2b323cc8b1868bcbd51428117c33f531853220629a8"}
Sep 16 16:50:47.091998 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:47.091568    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99" event={"ID":"ee454389-f8a1-4351-ae5b-2c342a3c63a0","Type":"ContainerStarted","Data":"ab465298c8c6f801465884c357af38e47c90272c91c4ab340fbc551882870ae4"}
Sep 16 16:50:47.111986 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:47.111925    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99" podStartSLOduration=1.9286447340000001 podStartE2EDuration="2.111910971s" totalImagesPullingTime="183.266237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:50:47.109752376 +0000 UTC m=+846.871713823" watchObservedRunningTime="2026-09-16 16:50:47.111910971 +0000 UTC m=+846.873872417"
Sep 16 16:50:56.295462 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:56.295422    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:56.295911 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:56.295516    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:56.307893 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:56.307865    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:50:57.148590 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:50:57.148560    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9gh99"
Sep 16 16:52:03.862191 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:03.862154    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"]
Sep 16 16:52:03.865672 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:03.865649    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:03.869543 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:03.869522    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f604b4-openshift-default-dockercfg-hdjnk\""
Sep 16 16:52:03.882283 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:03.882261    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"]
Sep 16 16:52:03.979279 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:03.979243    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d617c78-e49d-4cc3-b439-092189aaea66-istio-token\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:03.979279 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:03.979280    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-istio-data\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:03.979502 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:03.979332    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-credential-socket\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:03.979502 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:03.979374    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d617c78-e49d-4cc3-b439-092189aaea66-istiod-ca-cert\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:03.979502 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:03.979478    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-workload-socket\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:03.979502 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:03.979502    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4725f\" (UniqueName: \"kubernetes.io/projected/7d617c78-e49d-4cc3-b439-092189aaea66-kube-api-access-4725f\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:03.979717 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:03.979525    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-workload-certs\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:03.979717 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:03.979643    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-istio-envoy\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:03.979717 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:03.979670    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d617c78-e49d-4cc3-b439-092189aaea66-istio-podinfo\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:04.080398 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:04.080366    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-workload-socket\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:04.080398 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:04.080403    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4725f\" (UniqueName: \"kubernetes.io/projected/7d617c78-e49d-4cc3-b439-092189aaea66-kube-api-access-4725f\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:04.080623 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:04.080421    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-workload-certs\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:04.080623 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:04.080440    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-istio-envoy\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:04.080623 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:04.080552    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d617c78-e49d-4cc3-b439-092189aaea66-istio-podinfo\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:04.080807 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:04.080627    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d617c78-e49d-4cc3-b439-092189aaea66-istio-token\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:04.080807 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:04.080661    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-istio-data\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:04.080807 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:04.080697    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-credential-socket\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:04.080807 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:04.080726    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d617c78-e49d-4cc3-b439-092189aaea66-istiod-ca-cert\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:04.080807 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:04.080795    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-workload-socket\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:04.081043 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:04.080865    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-workload-certs\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:04.081127 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:04.081090    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-istio-data\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:04.081127 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:04.081108    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-credential-socket\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:04.081433 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:04.081410    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d617c78-e49d-4cc3-b439-092189aaea66-istiod-ca-cert\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:04.082742 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:04.082720    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-istio-envoy\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:04.082928 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:04.082912    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d617c78-e49d-4cc3-b439-092189aaea66-istio-podinfo\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:04.088082 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:04.088063    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4725f\" (UniqueName: \"kubernetes.io/projected/7d617c78-e49d-4cc3-b439-092189aaea66-kube-api-access-4725f\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:04.088194 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:04.088112    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d617c78-e49d-4cc3-b439-092189aaea66-istio-token\") pod \"e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") " pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:04.177776 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:04.177714    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:04.470893 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:04.470871    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"]
Sep 16 16:52:04.473047 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:52:04.473002    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d617c78_e49d_4cc3_b439_092189aaea66.slice/crio-c7b2766a41d224fa4f9325ddaa5cb0cea656355fdd967768402a6f7fc7355767 WatchSource:0}: Error finding container c7b2766a41d224fa4f9325ddaa5cb0cea656355fdd967768402a6f7fc7355767: Status 404 returned error can't find the container with id c7b2766a41d224fa4f9325ddaa5cb0cea656355fdd967768402a6f7fc7355767
Sep 16 16:52:05.444365 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:05.444319    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9" event={"ID":"7d617c78-e49d-4cc3-b439-092189aaea66","Type":"ContainerStarted","Data":"c7b2766a41d224fa4f9325ddaa5cb0cea656355fdd967768402a6f7fc7355767"}
Sep 16 16:52:07.322867 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:07.322830    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 16:52:07.323185 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:07.322900    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 16:52:07.323185 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:07.322930    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 16:52:07.462272 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:07.462219    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9" event={"ID":"7d617c78-e49d-4cc3-b439-092189aaea66","Type":"ContainerStarted","Data":"8dc2567b03f7650153c22add0081cc0835cb631ea38dbc7aa49d37f1ad5930c9"}
Sep 16 16:52:07.486145 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:07.486087    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9" podStartSLOduration=1.63893458 podStartE2EDuration="4.486063962s" totalImagesPullingTime="2.847129382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:52:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:52:07.483764395 +0000 UTC m=+927.245725825" watchObservedRunningTime="2026-09-16 16:52:07.486063962 +0000 UTC m=+927.248025411"
Sep 16 16:52:07.580336 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:07.580255    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"]
Sep 16 16:52:07.583927 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:07.583905    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"
Sep 16 16:52:07.587572 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:07.587548    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-eb2ab2-serving-cert\""
Sep 16 16:52:07.595672 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:07.595645    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"]
Sep 16 16:52:07.610700 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:07.610673    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a45e34-74c4-4028-b1bc-06d220aab39a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs\" (UID: \"e3a45e34-74c4-4028-b1bc-06d220aab39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"
Sep 16 16:52:07.610821 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:07.610721    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3a45e34-74c4-4028-b1bc-06d220aab39a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs\" (UID: \"e3a45e34-74c4-4028-b1bc-06d220aab39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"
Sep 16 16:52:07.610821 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:07.610738    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3a45e34-74c4-4028-b1bc-06d220aab39a-metrics-certs\") pod \"maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs\" (UID: \"e3a45e34-74c4-4028-b1bc-06d220aab39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"
Sep 16 16:52:07.610964 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:07.610850    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqcf5\" (UniqueName: \"kubernetes.io/projected/e3a45e34-74c4-4028-b1bc-06d220aab39a-kube-api-access-kqcf5\") pod \"maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs\" (UID: \"e3a45e34-74c4-4028-b1bc-06d220aab39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"
Sep 16 16:52:07.712148 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:07.712117    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3a45e34-74c4-4028-b1bc-06d220aab39a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs\" (UID: \"e3a45e34-74c4-4028-b1bc-06d220aab39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"
Sep 16 16:52:07.712148 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:07.712152    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3a45e34-74c4-4028-b1bc-06d220aab39a-metrics-certs\") pod \"maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs\" (UID: \"e3a45e34-74c4-4028-b1bc-06d220aab39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"
Sep 16 16:52:07.712391 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:07.712192    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqcf5\" (UniqueName: \"kubernetes.io/projected/e3a45e34-74c4-4028-b1bc-06d220aab39a-kube-api-access-kqcf5\") pod \"maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs\" (UID: \"e3a45e34-74c4-4028-b1bc-06d220aab39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"
Sep 16 16:52:07.712391 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:07.712241    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a45e34-74c4-4028-b1bc-06d220aab39a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs\" (UID: \"e3a45e34-74c4-4028-b1bc-06d220aab39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"
Sep 16 16:52:07.712391 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:52:07.712282    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-serving-cert: secret "maas-api-e2e-worker-w0-eb2ab2-serving-cert" not found
Sep 16 16:52:07.712391 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:52:07.712366    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3a45e34-74c4-4028-b1bc-06d220aab39a-maas-api-tls podName:e3a45e34-74c4-4028-b1bc-06d220aab39a nodeName:}" failed. No retries permitted until 2026-09-16 16:52:08.21234254 +0000 UTC m=+927.974303966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e3a45e34-74c4-4028-b1bc-06d220aab39a-maas-api-tls") pod "maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs" (UID: "e3a45e34-74c4-4028-b1bc-06d220aab39a") : secret "maas-api-e2e-worker-w0-eb2ab2-serving-cert" not found
Sep 16 16:52:07.712956 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:07.712923    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a45e34-74c4-4028-b1bc-06d220aab39a-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs\" (UID: \"e3a45e34-74c4-4028-b1bc-06d220aab39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"
Sep 16 16:52:07.714561 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:07.714540    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3a45e34-74c4-4028-b1bc-06d220aab39a-metrics-certs\") pod \"maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs\" (UID: \"e3a45e34-74c4-4028-b1bc-06d220aab39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"
Sep 16 16:52:07.722649 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:07.722631    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqcf5\" (UniqueName: \"kubernetes.io/projected/e3a45e34-74c4-4028-b1bc-06d220aab39a-kube-api-access-kqcf5\") pod \"maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs\" (UID: \"e3a45e34-74c4-4028-b1bc-06d220aab39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"
Sep 16 16:52:08.178351 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:08.178312    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:08.183367 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:08.183340    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:08.217855 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:08.217820    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3a45e34-74c4-4028-b1bc-06d220aab39a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs\" (UID: \"e3a45e34-74c4-4028-b1bc-06d220aab39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"
Sep 16 16:52:08.220193 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:08.220171    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3a45e34-74c4-4028-b1bc-06d220aab39a-maas-api-tls\") pod \"maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs\" (UID: \"e3a45e34-74c4-4028-b1bc-06d220aab39a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"
Sep 16 16:52:08.466849 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:08.466815    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:08.467772 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:08.467746    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:08.494669 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:08.494641    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"
Sep 16 16:52:08.782378 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:08.782352    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"]
Sep 16 16:52:08.784894 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:52:08.784867    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a45e34_74c4_4028_b1bc_06d220aab39a.slice/crio-cab2dad190b2b710aa14e2dc85f813e8725d810972edf120bcbd1c0468dc30c9 WatchSource:0}: Error finding container cab2dad190b2b710aa14e2dc85f813e8725d810972edf120bcbd1c0468dc30c9: Status 404 returned error can't find the container with id cab2dad190b2b710aa14e2dc85f813e8725d810972edf120bcbd1c0468dc30c9
Sep 16 16:52:08.957803 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:08.957767    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"]
Sep 16 16:52:08.963066 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:08.963043    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"
Sep 16 16:52:08.965361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:08.965333    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e2ea3e-serving-cert\""
Sep 16 16:52:08.974504 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:08.974481    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"]
Sep 16 16:52:09.025777 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.025748    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd879f74-608b-4626-a084-6af9bb94f68d-metrics-certs\") pod \"maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b\" (UID: \"cd879f74-608b-4626-a084-6af9bb94f68d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"
Sep 16 16:52:09.025965 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.025800    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd879f74-608b-4626-a084-6af9bb94f68d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b\" (UID: \"cd879f74-608b-4626-a084-6af9bb94f68d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"
Sep 16 16:52:09.025965 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.025827    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjhf4\" (UniqueName: \"kubernetes.io/projected/cd879f74-608b-4626-a084-6af9bb94f68d-kube-api-access-sjhf4\") pod \"maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b\" (UID: \"cd879f74-608b-4626-a084-6af9bb94f68d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"
Sep 16 16:52:09.025965 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.025893    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd879f74-608b-4626-a084-6af9bb94f68d-maas-api-tls\") pod \"maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b\" (UID: \"cd879f74-608b-4626-a084-6af9bb94f68d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"
Sep 16 16:52:09.126657 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.126578    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd879f74-608b-4626-a084-6af9bb94f68d-maas-api-tls\") pod \"maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b\" (UID: \"cd879f74-608b-4626-a084-6af9bb94f68d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"
Sep 16 16:52:09.126657 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.126647    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd879f74-608b-4626-a084-6af9bb94f68d-metrics-certs\") pod \"maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b\" (UID: \"cd879f74-608b-4626-a084-6af9bb94f68d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"
Sep 16 16:52:09.126842 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.126675    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd879f74-608b-4626-a084-6af9bb94f68d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b\" (UID: \"cd879f74-608b-4626-a084-6af9bb94f68d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"
Sep 16 16:52:09.126842 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.126698    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjhf4\" (UniqueName: \"kubernetes.io/projected/cd879f74-608b-4626-a084-6af9bb94f68d-kube-api-access-sjhf4\") pod \"maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b\" (UID: \"cd879f74-608b-4626-a084-6af9bb94f68d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"
Sep 16 16:52:09.126842 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:52:09.126724    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-serving-cert: secret "maas-api-e2e-inf-a-e2ea3e-serving-cert" not found
Sep 16 16:52:09.126842 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:52:09.126790    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd879f74-608b-4626-a084-6af9bb94f68d-maas-api-tls podName:cd879f74-608b-4626-a084-6af9bb94f68d nodeName:}" failed. No retries permitted until 2026-09-16 16:52:09.626773808 +0000 UTC m=+929.388735233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cd879f74-608b-4626-a084-6af9bb94f68d-maas-api-tls") pod "maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b" (UID: "cd879f74-608b-4626-a084-6af9bb94f68d") : secret "maas-api-e2e-inf-a-e2ea3e-serving-cert" not found
Sep 16 16:52:09.127432 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.127414    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd879f74-608b-4626-a084-6af9bb94f68d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b\" (UID: \"cd879f74-608b-4626-a084-6af9bb94f68d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"
Sep 16 16:52:09.129156 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.129135    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd879f74-608b-4626-a084-6af9bb94f68d-metrics-certs\") pod \"maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b\" (UID: \"cd879f74-608b-4626-a084-6af9bb94f68d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"
Sep 16 16:52:09.135834 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.135812    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjhf4\" (UniqueName: \"kubernetes.io/projected/cd879f74-608b-4626-a084-6af9bb94f68d-kube-api-access-sjhf4\") pod \"maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b\" (UID: \"cd879f74-608b-4626-a084-6af9bb94f68d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"
Sep 16 16:52:09.472880 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.472843    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs" event={"ID":"e3a45e34-74c4-4028-b1bc-06d220aab39a","Type":"ContainerStarted","Data":"6ed5d74388426d0c373d3f6eda2a9c98edede042f476253c9e9610b61de74710"}
Sep 16 16:52:09.473303 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.472899    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs" event={"ID":"e3a45e34-74c4-4028-b1bc-06d220aab39a","Type":"ContainerStarted","Data":"cab2dad190b2b710aa14e2dc85f813e8725d810972edf120bcbd1c0468dc30c9"}
Sep 16 16:52:09.473303 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.472979    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"
Sep 16 16:52:09.490609 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.490554    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs" podStartSLOduration=2.015361321 podStartE2EDuration="2.490536916s" totalImagesPullingTime="475.175595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:52:09.489552499 +0000 UTC m=+929.251513945" watchObservedRunningTime="2026-09-16 16:52:09.490536916 +0000 UTC m=+929.252498367"
Sep 16 16:52:09.630236 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.630202    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd879f74-608b-4626-a084-6af9bb94f68d-maas-api-tls\") pod \"maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b\" (UID: \"cd879f74-608b-4626-a084-6af9bb94f68d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"
Sep 16 16:52:09.632504 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.632482    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd879f74-608b-4626-a084-6af9bb94f68d-maas-api-tls\") pod \"maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b\" (UID: \"cd879f74-608b-4626-a084-6af9bb94f68d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"
Sep 16 16:52:09.825926 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.825847    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz"]
Sep 16 16:52:09.829287 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.829267    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz"
Sep 16 16:52:09.831653 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.831630    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e2ea3e-dockercfg-q8h2c\""
Sep 16 16:52:09.831843 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.831829    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e2ea3e\""
Sep 16 16:52:09.843469 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.843445    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz"]
Sep 16 16:52:09.876266 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.876234    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"
Sep 16 16:52:09.933555 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.933036    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49jmq\" (UniqueName: \"kubernetes.io/projected/dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a-kube-api-access-49jmq\") pod \"payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz\" (UID: \"dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz"
Sep 16 16:52:09.933555 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:09.933203    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz\" (UID: \"dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz"
Sep 16 16:52:10.033923 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:10.033894    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49jmq\" (UniqueName: \"kubernetes.io/projected/dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a-kube-api-access-49jmq\") pod \"payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz\" (UID: \"dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz"
Sep 16 16:52:10.034100 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:10.033967    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz\" (UID: \"dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz"
Sep 16 16:52:10.034616 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:10.034598    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz\" (UID: \"dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz"
Sep 16 16:52:10.048802 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:10.048770    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49jmq\" (UniqueName: \"kubernetes.io/projected/dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a-kube-api-access-49jmq\") pod \"payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz\" (UID: \"dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz"
Sep 16 16:52:10.139115 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:10.139042    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz"
Sep 16 16:52:10.166097 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:52:10.166068    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd879f74_608b_4626_a084_6af9bb94f68d.slice/crio-ba27bd5fca870487dc61c2bd5019cc8f482089f6664647edca082889ae77bc1e WatchSource:0}: Error finding container ba27bd5fca870487dc61c2bd5019cc8f482089f6664647edca082889ae77bc1e: Status 404 returned error can't find the container with id ba27bd5fca870487dc61c2bd5019cc8f482089f6664647edca082889ae77bc1e
Sep 16 16:52:10.166472 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:10.166448    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"]
Sep 16 16:52:10.413041 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:10.413015    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz"]
Sep 16 16:52:10.415026 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:52:10.414998    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc36aba_a11d_4ed2_a5c9_9d875af8cb4a.slice/crio-374fea5a7c007598c9ebdef0a6e272435d6a4165354bf23e96e7d9ddbdf3cff5 WatchSource:0}: Error finding container 374fea5a7c007598c9ebdef0a6e272435d6a4165354bf23e96e7d9ddbdf3cff5: Status 404 returned error can't find the container with id 374fea5a7c007598c9ebdef0a6e272435d6a4165354bf23e96e7d9ddbdf3cff5
Sep 16 16:52:10.479801 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:10.479775    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz" event={"ID":"dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a","Type":"ContainerStarted","Data":"c021c05d067627186f90a66f959b39d3e1f89a612dc318ab40ff82e346693e13"}
Sep 16 16:52:10.480196 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:10.479827    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz" event={"ID":"dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a","Type":"ContainerStarted","Data":"374fea5a7c007598c9ebdef0a6e272435d6a4165354bf23e96e7d9ddbdf3cff5"}
Sep 16 16:52:10.480196 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:10.479857    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz"
Sep 16 16:52:10.481066 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:10.481041    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b" event={"ID":"cd879f74-608b-4626-a084-6af9bb94f68d","Type":"ContainerStarted","Data":"ba27bd5fca870487dc61c2bd5019cc8f482089f6664647edca082889ae77bc1e"}
Sep 16 16:52:11.487496 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:11.487459    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b" event={"ID":"cd879f74-608b-4626-a084-6af9bb94f68d","Type":"ContainerStarted","Data":"ad22e3cfd9777c7ef7dbcb9e49402b72b9cdb346c4ab81cdf937d0903093e495"}
Sep 16 16:52:11.487919 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:11.487565    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"
Sep 16 16:52:11.506843 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:11.506794    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b" podStartSLOduration=3.156389632 podStartE2EDuration="3.506780014s" totalImagesPullingTime="350.390382ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:52:11.504644268 +0000 UTC m=+931.266605728" watchObservedRunningTime="2026-09-16 16:52:11.506780014 +0000 UTC m=+931.268741461"
Sep 16 16:52:11.507001 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:11.506924    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz" podStartSLOduration=2.506918931 podStartE2EDuration="2.506918931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:52:10.504510408 +0000 UTC m=+930.266471856" watchObservedRunningTime="2026-09-16 16:52:11.506918931 +0000 UTC m=+931.268880377"
Sep 16 16:52:13.185248 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:13.185210    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"]
Sep 16 16:52:13.190020 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:13.189965    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"
Sep 16 16:52:13.193130 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:13.192862    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-f604b4-serving-cert\""
Sep 16 16:52:13.229821 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:13.229789    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"]
Sep 16 16:52:13.264826 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:13.264792    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-metrics-certs\") pod \"maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6\" (UID: \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"
Sep 16 16:52:13.265029 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:13.264842    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js4pz\" (UniqueName: \"kubernetes.io/projected/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-kube-api-access-js4pz\") pod \"maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6\" (UID: \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"
Sep 16 16:52:13.265029 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:13.264933    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6\" (UID: \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"
Sep 16 16:52:13.265029 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:13.264982    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6\" (UID: \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"
Sep 16 16:52:13.366171 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:13.366135    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-metrics-certs\") pod \"maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6\" (UID: \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"
Sep 16 16:52:13.366374 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:13.366193    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js4pz\" (UniqueName: \"kubernetes.io/projected/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-kube-api-access-js4pz\") pod \"maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6\" (UID: \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"
Sep 16 16:52:13.366374 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:13.366266    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6\" (UID: \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"
Sep 16 16:52:13.366374 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:52:13.366342    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-serving-cert: secret "maas-api-e2e-worker-w4-f604b4-serving-cert" not found
Sep 16 16:52:13.366537 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:13.366393    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6\" (UID: \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"
Sep 16 16:52:13.366537 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:52:13.366405    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-maas-api-tls podName:4385870d-6cac-4730-a6bd-f0a69ea8ee0f nodeName:}" failed. No retries permitted until 2026-09-16 16:52:13.8663853 +0000 UTC m=+933.628346725 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-maas-api-tls") pod "maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6" (UID: "4385870d-6cac-4730-a6bd-f0a69ea8ee0f") : secret "maas-api-e2e-worker-w4-f604b4-serving-cert" not found
Sep 16 16:52:13.367035 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:13.367016    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6\" (UID: \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"
Sep 16 16:52:13.368644 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:13.368625    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-metrics-certs\") pod \"maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6\" (UID: \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"
Sep 16 16:52:13.374903 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:13.374879    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js4pz\" (UniqueName: \"kubernetes.io/projected/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-kube-api-access-js4pz\") pod \"maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6\" (UID: \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"
Sep 16 16:52:13.870294 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:13.870248    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6\" (UID: \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"
Sep 16 16:52:13.872654 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:13.872631    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6\" (UID: \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"
Sep 16 16:52:14.102416 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:14.102380    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"
Sep 16 16:52:14.376702 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:14.376676    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"]
Sep 16 16:52:14.378959 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:52:14.378915    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4385870d_6cac_4730_a6bd_f0a69ea8ee0f.slice/crio-365533a493748a8929ad48819089166f20919d47cc4791789c9709f1a22bb1c3 WatchSource:0}: Error finding container 365533a493748a8929ad48819089166f20919d47cc4791789c9709f1a22bb1c3: Status 404 returned error can't find the container with id 365533a493748a8929ad48819089166f20919d47cc4791789c9709f1a22bb1c3
Sep 16 16:52:14.503966 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:14.503920    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6" event={"ID":"4385870d-6cac-4730-a6bd-f0a69ea8ee0f","Type":"ContainerStarted","Data":"365533a493748a8929ad48819089166f20919d47cc4791789c9709f1a22bb1c3"}
Sep 16 16:52:15.486694 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:15.486657    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"
Sep 16 16:52:15.509832 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:15.509803    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6" event={"ID":"4385870d-6cac-4730-a6bd-f0a69ea8ee0f","Type":"ContainerStarted","Data":"47795b553fa32d6f273b8cd0a074ee7fcebac2620e46d5b95a738d2260b44d2d"}
Sep 16 16:52:15.509997 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:15.509868    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"
Sep 16 16:52:15.580958 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:15.580893    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6" podStartSLOduration=1.884188749 podStartE2EDuration="2.580877311s" totalImagesPullingTime="696.688562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:52:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:52:15.577903948 +0000 UTC m=+935.339865396" watchObservedRunningTime="2026-09-16 16:52:15.580877311 +0000 UTC m=+935.342838808"
Sep 16 16:52:17.498402 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:17.498368    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"
Sep 16 16:52:19.060529 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:19.060488    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"]
Sep 16 16:52:19.064996 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:19.064965    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"
Sep 16 16:52:19.067762 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:19.067736    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-708c6a-serving-cert\""
Sep 16 16:52:19.076761 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:19.076685    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"]
Sep 16 16:52:19.121306 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:19.121273    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn7pt\" (UniqueName: \"kubernetes.io/projected/5ff37461-4b79-454d-a0bb-ec6a55b8e302-kube-api-access-jn7pt\") pod \"maas-api-e2e-autores-708c6a-79568ff975-bpw5l\" (UID: \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"
Sep 16 16:52:19.121479 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:19.121328    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff37461-4b79-454d-a0bb-ec6a55b8e302-openshift-service-ca\") pod \"maas-api-e2e-autores-708c6a-79568ff975-bpw5l\" (UID: \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"
Sep 16 16:52:19.121479 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:19.121377    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ff37461-4b79-454d-a0bb-ec6a55b8e302-maas-api-tls\") pod \"maas-api-e2e-autores-708c6a-79568ff975-bpw5l\" (UID: \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"
Sep 16 16:52:19.121479 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:19.121410    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff37461-4b79-454d-a0bb-ec6a55b8e302-metrics-certs\") pod \"maas-api-e2e-autores-708c6a-79568ff975-bpw5l\" (UID: \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"
Sep 16 16:52:19.222274 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:19.222242    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ff37461-4b79-454d-a0bb-ec6a55b8e302-maas-api-tls\") pod \"maas-api-e2e-autores-708c6a-79568ff975-bpw5l\" (UID: \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"
Sep 16 16:52:19.222471 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:19.222296    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff37461-4b79-454d-a0bb-ec6a55b8e302-metrics-certs\") pod \"maas-api-e2e-autores-708c6a-79568ff975-bpw5l\" (UID: \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"
Sep 16 16:52:19.222471 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:19.222330    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn7pt\" (UniqueName: \"kubernetes.io/projected/5ff37461-4b79-454d-a0bb-ec6a55b8e302-kube-api-access-jn7pt\") pod \"maas-api-e2e-autores-708c6a-79568ff975-bpw5l\" (UID: \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"
Sep 16 16:52:19.222471 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:19.222374    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff37461-4b79-454d-a0bb-ec6a55b8e302-openshift-service-ca\") pod \"maas-api-e2e-autores-708c6a-79568ff975-bpw5l\" (UID: \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"
Sep 16 16:52:19.222471 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:52:19.222430    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-serving-cert: secret "maas-api-e2e-autores-708c6a-serving-cert" not found
Sep 16 16:52:19.222671 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:52:19.222523    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ff37461-4b79-454d-a0bb-ec6a55b8e302-maas-api-tls podName:5ff37461-4b79-454d-a0bb-ec6a55b8e302 nodeName:}" failed. No retries permitted until 2026-09-16 16:52:19.722501836 +0000 UTC m=+939.484463266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5ff37461-4b79-454d-a0bb-ec6a55b8e302-maas-api-tls") pod "maas-api-e2e-autores-708c6a-79568ff975-bpw5l" (UID: "5ff37461-4b79-454d-a0bb-ec6a55b8e302") : secret "maas-api-e2e-autores-708c6a-serving-cert" not found
Sep 16 16:52:19.223047 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:19.223018    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff37461-4b79-454d-a0bb-ec6a55b8e302-openshift-service-ca\") pod \"maas-api-e2e-autores-708c6a-79568ff975-bpw5l\" (UID: \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"
Sep 16 16:52:19.224654 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:19.224637    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff37461-4b79-454d-a0bb-ec6a55b8e302-metrics-certs\") pod \"maas-api-e2e-autores-708c6a-79568ff975-bpw5l\" (UID: \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"
Sep 16 16:52:19.232873 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:19.232849    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn7pt\" (UniqueName: \"kubernetes.io/projected/5ff37461-4b79-454d-a0bb-ec6a55b8e302-kube-api-access-jn7pt\") pod \"maas-api-e2e-autores-708c6a-79568ff975-bpw5l\" (UID: \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"
Sep 16 16:52:19.727246 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:19.727209    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ff37461-4b79-454d-a0bb-ec6a55b8e302-maas-api-tls\") pod \"maas-api-e2e-autores-708c6a-79568ff975-bpw5l\" (UID: \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"
Sep 16 16:52:19.729569 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:19.729544    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ff37461-4b79-454d-a0bb-ec6a55b8e302-maas-api-tls\") pod \"maas-api-e2e-autores-708c6a-79568ff975-bpw5l\" (UID: \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"
Sep 16 16:52:19.978632 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:19.978539    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"
Sep 16 16:52:20.250580 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:20.250556    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"]
Sep 16 16:52:20.252781 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:52:20.252753    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff37461_4b79_454d_a0bb_ec6a55b8e302.slice/crio-1f03866b287b3a0d105f900fc5619aa87e411137f6f569f8c783c26f35a45792 WatchSource:0}: Error finding container 1f03866b287b3a0d105f900fc5619aa87e411137f6f569f8c783c26f35a45792: Status 404 returned error can't find the container with id 1f03866b287b3a0d105f900fc5619aa87e411137f6f569f8c783c26f35a45792
Sep 16 16:52:20.254207 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:20.254186    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 16:52:20.534058 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:20.533976    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l" event={"ID":"5ff37461-4b79-454d-a0bb-ec6a55b8e302","Type":"ContainerStarted","Data":"1f03866b287b3a0d105f900fc5619aa87e411137f6f569f8c783c26f35a45792"}
Sep 16 16:52:20.899966 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:20.899883    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz"]
Sep 16 16:52:20.903536 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:20.903521    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz"
Sep 16 16:52:20.906508 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:20.906490    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-41cc4972-dockercfg-wsbgd\""
Sep 16 16:52:20.910190 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:20.910169    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-41cc4972\""
Sep 16 16:52:20.932149 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:20.932129    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz"]
Sep 16 16:52:21.038560 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:21.038526    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fst5q\" (UniqueName: \"kubernetes.io/projected/0d976837-1d20-4819-bd50-6b01ceb255b6-kube-api-access-fst5q\") pod \"payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz\" (UID: \"0d976837-1d20-4819-bd50-6b01ceb255b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz"
Sep 16 16:52:21.038735 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:21.038582    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d976837-1d20-4819-bd50-6b01ceb255b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz\" (UID: \"0d976837-1d20-4819-bd50-6b01ceb255b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz"
Sep 16 16:52:21.139822 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:21.139783    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d976837-1d20-4819-bd50-6b01ceb255b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz\" (UID: \"0d976837-1d20-4819-bd50-6b01ceb255b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz"
Sep 16 16:52:21.140029 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:21.139885    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fst5q\" (UniqueName: \"kubernetes.io/projected/0d976837-1d20-4819-bd50-6b01ceb255b6-kube-api-access-fst5q\") pod \"payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz\" (UID: \"0d976837-1d20-4819-bd50-6b01ceb255b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz"
Sep 16 16:52:21.140633 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:21.140607    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d976837-1d20-4819-bd50-6b01ceb255b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz\" (UID: \"0d976837-1d20-4819-bd50-6b01ceb255b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz"
Sep 16 16:52:21.161681 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:21.161624    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fst5q\" (UniqueName: \"kubernetes.io/projected/0d976837-1d20-4819-bd50-6b01ceb255b6-kube-api-access-fst5q\") pod \"payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz\" (UID: \"0d976837-1d20-4819-bd50-6b01ceb255b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz"
Sep 16 16:52:21.213554 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:21.213526    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz"
Sep 16 16:52:21.489874 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:21.489840    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz"
Sep 16 16:52:21.498428 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:21.498404    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz"]
Sep 16 16:52:21.501008 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:52:21.500981    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d976837_1d20_4819_bd50_6b01ceb255b6.slice/crio-8f6b9f6f37b39218bbafd40bc77f2209c040cb2d0cfe159eb00e32df6dcdcb1b WatchSource:0}: Error finding container 8f6b9f6f37b39218bbafd40bc77f2209c040cb2d0cfe159eb00e32df6dcdcb1b: Status 404 returned error can't find the container with id 8f6b9f6f37b39218bbafd40bc77f2209c040cb2d0cfe159eb00e32df6dcdcb1b
Sep 16 16:52:21.520194 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:21.520173    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"
Sep 16 16:52:21.541631 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:21.541588    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz" event={"ID":"0d976837-1d20-4819-bd50-6b01ceb255b6","Type":"ContainerStarted","Data":"8f6b9f6f37b39218bbafd40bc77f2209c040cb2d0cfe159eb00e32df6dcdcb1b"}
Sep 16 16:52:21.543477 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:21.543447    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l" event={"ID":"5ff37461-4b79-454d-a0bb-ec6a55b8e302","Type":"ContainerStarted","Data":"80af277bf76881e6f786e985ef094eb31ed4d3e938d7da186d13be4ffec97d4e"}
Sep 16 16:52:21.543583 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:21.543521    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"
Sep 16 16:52:21.611150 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:21.611094    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l" podStartSLOduration=2.263047112 podStartE2EDuration="2.611072859s" totalImagesPullingTime="348.025747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:52:21.610401604 +0000 UTC m=+941.372363052" watchObservedRunningTime="2026-09-16 16:52:21.611072859 +0000 UTC m=+941.373034324"
Sep 16 16:52:22.551481 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:22.551439    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz" event={"ID":"0d976837-1d20-4819-bd50-6b01ceb255b6","Type":"ContainerStarted","Data":"2a7f2024bf0037a3c4819f4d4c6b3f1d619b8f281f24fd939c2ccbbb5f12cb59"}
Sep 16 16:52:22.552017 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:22.551624    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz"
Sep 16 16:52:22.625644 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:22.625591    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz" podStartSLOduration=2.625574597 podStartE2EDuration="2.625574597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:52:22.624476434 +0000 UTC m=+942.386437882" watchObservedRunningTime="2026-09-16 16:52:22.625574597 +0000 UTC m=+942.387536044"
Sep 16 16:52:27.567261 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:27.567223    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"
Sep 16 16:52:28.411628 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:28.411587    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"]
Sep 16 16:52:28.416077 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:28.416052    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"
Sep 16 16:52:28.418979 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:28.418959    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ea8f09-serving-cert\""
Sep 16 16:52:28.426584 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:28.426559    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"]
Sep 16 16:52:28.512778 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:28.512743    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn5xh\" (UniqueName: \"kubernetes.io/projected/9b0eac9c-d48e-494b-b43a-839b9923dad4-kube-api-access-gn5xh\") pod \"maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6\" (UID: \"9b0eac9c-d48e-494b-b43a-839b9923dad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"
Sep 16 16:52:28.512900 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:28.512786    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0eac9c-d48e-494b-b43a-839b9923dad4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6\" (UID: \"9b0eac9c-d48e-494b-b43a-839b9923dad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"
Sep 16 16:52:28.512900 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:28.512820    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b0eac9c-d48e-494b-b43a-839b9923dad4-maas-api-tls\") pod \"maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6\" (UID: \"9b0eac9c-d48e-494b-b43a-839b9923dad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"
Sep 16 16:52:28.513031 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:28.512897    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0eac9c-d48e-494b-b43a-839b9923dad4-metrics-certs\") pod \"maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6\" (UID: \"9b0eac9c-d48e-494b-b43a-839b9923dad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"
Sep 16 16:52:28.614336 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:28.614302    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn5xh\" (UniqueName: \"kubernetes.io/projected/9b0eac9c-d48e-494b-b43a-839b9923dad4-kube-api-access-gn5xh\") pod \"maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6\" (UID: \"9b0eac9c-d48e-494b-b43a-839b9923dad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"
Sep 16 16:52:28.614675 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:28.614344    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0eac9c-d48e-494b-b43a-839b9923dad4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6\" (UID: \"9b0eac9c-d48e-494b-b43a-839b9923dad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"
Sep 16 16:52:28.614675 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:28.614388    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b0eac9c-d48e-494b-b43a-839b9923dad4-maas-api-tls\") pod \"maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6\" (UID: \"9b0eac9c-d48e-494b-b43a-839b9923dad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"
Sep 16 16:52:28.614675 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:28.614433    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0eac9c-d48e-494b-b43a-839b9923dad4-metrics-certs\") pod \"maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6\" (UID: \"9b0eac9c-d48e-494b-b43a-839b9923dad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"
Sep 16 16:52:28.615042 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:28.615013    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0eac9c-d48e-494b-b43a-839b9923dad4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6\" (UID: \"9b0eac9c-d48e-494b-b43a-839b9923dad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"
Sep 16 16:52:28.617012 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:28.616988    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b0eac9c-d48e-494b-b43a-839b9923dad4-maas-api-tls\") pod \"maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6\" (UID: \"9b0eac9c-d48e-494b-b43a-839b9923dad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"
Sep 16 16:52:28.617106 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:28.617034    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0eac9c-d48e-494b-b43a-839b9923dad4-metrics-certs\") pod \"maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6\" (UID: \"9b0eac9c-d48e-494b-b43a-839b9923dad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"
Sep 16 16:52:28.632515 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:28.632486    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn5xh\" (UniqueName: \"kubernetes.io/projected/9b0eac9c-d48e-494b-b43a-839b9923dad4-kube-api-access-gn5xh\") pod \"maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6\" (UID: \"9b0eac9c-d48e-494b-b43a-839b9923dad4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"
Sep 16 16:52:28.729961 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:28.729907    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"
Sep 16 16:52:29.011140 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:29.011108    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"]
Sep 16 16:52:29.012262 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:52:29.012229    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0eac9c_d48e_494b_b43a_839b9923dad4.slice/crio-40c37c359c10737a1be63437a1632d3b1660ebc9a8decb71da146e47b3e678f5 WatchSource:0}: Error finding container 40c37c359c10737a1be63437a1632d3b1660ebc9a8decb71da146e47b3e678f5: Status 404 returned error can't find the container with id 40c37c359c10737a1be63437a1632d3b1660ebc9a8decb71da146e47b3e678f5
Sep 16 16:52:29.173501 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:29.173462    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc"]
Sep 16 16:52:29.178127 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:29.178105    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc"
Sep 16 16:52:29.180841 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:29.180813    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ea8f09-dockercfg-fdxfk\""
Sep 16 16:52:29.180993 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:29.180867    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ea8f09\""
Sep 16 16:52:29.187686 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:29.187662    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc"]
Sep 16 16:52:29.322581 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:29.322498    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec165c6-71a1-4b89-9fcf-2aa247a5f475-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc\" (UID: \"2ec165c6-71a1-4b89-9fcf-2aa247a5f475\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc"
Sep 16 16:52:29.322783 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:29.322583    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vpqn\" (UniqueName: \"kubernetes.io/projected/2ec165c6-71a1-4b89-9fcf-2aa247a5f475-kube-api-access-5vpqn\") pod \"payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc\" (UID: \"2ec165c6-71a1-4b89-9fcf-2aa247a5f475\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc"
Sep 16 16:52:29.424057 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:29.424029    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vpqn\" (UniqueName: \"kubernetes.io/projected/2ec165c6-71a1-4b89-9fcf-2aa247a5f475-kube-api-access-5vpqn\") pod \"payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc\" (UID: \"2ec165c6-71a1-4b89-9fcf-2aa247a5f475\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc"
Sep 16 16:52:29.424218 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:29.424129    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec165c6-71a1-4b89-9fcf-2aa247a5f475-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc\" (UID: \"2ec165c6-71a1-4b89-9fcf-2aa247a5f475\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc"
Sep 16 16:52:29.425026 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:29.425003    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec165c6-71a1-4b89-9fcf-2aa247a5f475-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc\" (UID: \"2ec165c6-71a1-4b89-9fcf-2aa247a5f475\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc"
Sep 16 16:52:29.432577 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:29.432547    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vpqn\" (UniqueName: \"kubernetes.io/projected/2ec165c6-71a1-4b89-9fcf-2aa247a5f475-kube-api-access-5vpqn\") pod \"payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc\" (UID: \"2ec165c6-71a1-4b89-9fcf-2aa247a5f475\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc"
Sep 16 16:52:29.490401 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:29.490371    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc"
Sep 16 16:52:29.607417 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:29.607328    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6" event={"ID":"9b0eac9c-d48e-494b-b43a-839b9923dad4","Type":"ContainerStarted","Data":"87b17000cdd74104ae3e330a8e952b7cc387c62192b6dc001b2987970a1f7671"}
Sep 16 16:52:29.607417 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:29.607396    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6" event={"ID":"9b0eac9c-d48e-494b-b43a-839b9923dad4","Type":"ContainerStarted","Data":"40c37c359c10737a1be63437a1632d3b1660ebc9a8decb71da146e47b3e678f5"}
Sep 16 16:52:29.607633 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:29.607506    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"
Sep 16 16:52:29.625697 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:29.625636    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6" podStartSLOduration=1.266965034 podStartE2EDuration="1.625615038s" totalImagesPullingTime="358.650004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:52:29.623750268 +0000 UTC m=+949.385711724" watchObservedRunningTime="2026-09-16 16:52:29.625615038 +0000 UTC m=+949.387576486"
Sep 16 16:52:29.767503 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:29.767475    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc"]
Sep 16 16:52:29.770294 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:52:29.770261    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec165c6_71a1_4b89_9fcf_2aa247a5f475.slice/crio-b1235f05bd1cf92411c35496b27cc9bc5d05f0091e018baf6ced004872e1f167 WatchSource:0}: Error finding container b1235f05bd1cf92411c35496b27cc9bc5d05f0091e018baf6ced004872e1f167: Status 404 returned error can't find the container with id b1235f05bd1cf92411c35496b27cc9bc5d05f0091e018baf6ced004872e1f167
Sep 16 16:52:30.614646 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:30.614612    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc" event={"ID":"2ec165c6-71a1-4b89-9fcf-2aa247a5f475","Type":"ContainerStarted","Data":"d9f838b76222aa1353a09d4045ca1d6e3937fd6f850f198d16a8cc3adaa4f44e"}
Sep 16 16:52:30.614826 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:30.614667    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc" event={"ID":"2ec165c6-71a1-4b89-9fcf-2aa247a5f475","Type":"ContainerStarted","Data":"b1235f05bd1cf92411c35496b27cc9bc5d05f0091e018baf6ced004872e1f167"}
Sep 16 16:52:30.614913 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:30.614881    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc"
Sep 16 16:52:30.631658 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:30.631601    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc" podStartSLOduration=1.6315889540000001 podStartE2EDuration="1.631588954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:52:30.630444645 +0000 UTC m=+950.392406093" watchObservedRunningTime="2026-09-16 16:52:30.631588954 +0000 UTC m=+950.393550454"
Sep 16 16:52:32.625991 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:32.625956    2556 generic.go:356] "Generic (PLEG): container finished" podID="7d617c78-e49d-4cc3-b439-092189aaea66" containerID="8dc2567b03f7650153c22add0081cc0835cb631ea38dbc7aa49d37f1ad5930c9" exitCode=0
Sep 16 16:52:32.626400 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:32.625969    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9" event={"ID":"7d617c78-e49d-4cc3-b439-092189aaea66","Type":"ContainerDied","Data":"8dc2567b03f7650153c22add0081cc0835cb631ea38dbc7aa49d37f1ad5930c9"}
Sep 16 16:52:32.626400 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:32.626357    2556 scope.go:117] "RemoveContainer" containerID="8dc2567b03f7650153c22add0081cc0835cb631ea38dbc7aa49d37f1ad5930c9"
Sep 16 16:52:32.627532 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:32.627495    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 16:52:32.627609 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:32.627596    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 16:52:32.627665 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:32.627643    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 16:52:33.571133 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:33.571099    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz"
Sep 16 16:52:33.633436 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:33.633405    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9" event={"ID":"7d617c78-e49d-4cc3-b439-092189aaea66","Type":"ContainerStarted","Data":"68fe1ef39177260ebbab8a6852b29f348d26b42ca07009978808077a8b4d3e69"}
Sep 16 16:52:34.178604 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:34.178560    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:34.179552 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:34.179527    2556 patch_prober.go:28] interesting pod/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 16 16:52:34.179681 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:34.179586    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9" podUID="7d617c78-e49d-4cc3-b439-092189aaea66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 16 16:52:35.178555 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:35.178514    2556 patch_prober.go:28] interesting pod/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 16 16:52:35.179032 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:35.178597    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9" podUID="7d617c78-e49d-4cc3-b439-092189aaea66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 16 16:52:35.619666 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:35.619635    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"
Sep 16 16:52:36.178682 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:36.178642    2556 patch_prober.go:28] interesting pod/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 16 16:52:36.179076 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:36.178722    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9" podUID="7d617c78-e49d-4cc3-b439-092189aaea66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 16 16:52:37.178620 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:37.178587    2556 patch_prober.go:28] interesting pod/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 16 16:52:37.178799 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:37.178641    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9" podUID="7d617c78-e49d-4cc3-b439-092189aaea66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 16 16:52:38.178370 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:38.178330    2556 patch_prober.go:28] interesting pod/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 16 16:52:38.178580 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:38.178397    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9" podUID="7d617c78-e49d-4cc3-b439-092189aaea66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 16 16:52:38.467315 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:38.467268    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:39.182421 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:39.182392    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:39.183493 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:39.183469    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:52:41.622324 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:41.622295    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc"
Sep 16 16:52:48.125469 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:48.125432    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz"]
Sep 16 16:52:48.126034 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:48.125707    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz" podUID="0d976837-1d20-4819-bd50-6b01ceb255b6" containerName="payload-pre-processing" containerID="cri-o://2a7f2024bf0037a3c4819f4d4c6b3f1d619b8f281f24fd939c2ccbbb5f12cb59" gracePeriod=30
Sep 16 16:52:48.389847 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:48.389792    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz"
Sep 16 16:52:48.404586 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:48.404564    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d976837-1d20-4819-bd50-6b01ceb255b6-plugins-config-volume\") pod \"0d976837-1d20-4819-bd50-6b01ceb255b6\" (UID: \"0d976837-1d20-4819-bd50-6b01ceb255b6\") "
Sep 16 16:52:48.404727 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:48.404632    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fst5q\" (UniqueName: \"kubernetes.io/projected/0d976837-1d20-4819-bd50-6b01ceb255b6-kube-api-access-fst5q\") pod \"0d976837-1d20-4819-bd50-6b01ceb255b6\" (UID: \"0d976837-1d20-4819-bd50-6b01ceb255b6\") "
Sep 16 16:52:48.405042 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:48.405012    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d976837-1d20-4819-bd50-6b01ceb255b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0d976837-1d20-4819-bd50-6b01ceb255b6" (UID: "0d976837-1d20-4819-bd50-6b01ceb255b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:52:48.406892 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:48.406866    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d976837-1d20-4819-bd50-6b01ceb255b6-kube-api-access-fst5q" (OuterVolumeSpecName: "kube-api-access-fst5q") pod "0d976837-1d20-4819-bd50-6b01ceb255b6" (UID: "0d976837-1d20-4819-bd50-6b01ceb255b6"). InnerVolumeSpecName "kube-api-access-fst5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:52:48.506018 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:48.505985    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d976837-1d20-4819-bd50-6b01ceb255b6-plugins-config-volume\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:52:48.506018 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:48.506019    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fst5q\" (UniqueName: \"kubernetes.io/projected/0d976837-1d20-4819-bd50-6b01ceb255b6-kube-api-access-fst5q\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:52:48.719598 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:48.719560    2556 generic.go:356] "Generic (PLEG): container finished" podID="0d976837-1d20-4819-bd50-6b01ceb255b6" containerID="2a7f2024bf0037a3c4819f4d4c6b3f1d619b8f281f24fd939c2ccbbb5f12cb59" exitCode=0
Sep 16 16:52:48.719795 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:48.719621    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz"
Sep 16 16:52:48.719795 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:48.719641    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz" event={"ID":"0d976837-1d20-4819-bd50-6b01ceb255b6","Type":"ContainerDied","Data":"2a7f2024bf0037a3c4819f4d4c6b3f1d619b8f281f24fd939c2ccbbb5f12cb59"}
Sep 16 16:52:48.719795 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:48.719701    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz" event={"ID":"0d976837-1d20-4819-bd50-6b01ceb255b6","Type":"ContainerDied","Data":"8f6b9f6f37b39218bbafd40bc77f2209c040cb2d0cfe159eb00e32df6dcdcb1b"}
Sep 16 16:52:48.719795 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:48.719725    2556 scope.go:117] "RemoveContainer" containerID="2a7f2024bf0037a3c4819f4d4c6b3f1d619b8f281f24fd939c2ccbbb5f12cb59"
Sep 16 16:52:48.732504 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:48.732484    2556 scope.go:117] "RemoveContainer" containerID="2a7f2024bf0037a3c4819f4d4c6b3f1d619b8f281f24fd939c2ccbbb5f12cb59"
Sep 16 16:52:48.732789 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:52:48.732769    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a7f2024bf0037a3c4819f4d4c6b3f1d619b8f281f24fd939c2ccbbb5f12cb59\": container with ID starting with 2a7f2024bf0037a3c4819f4d4c6b3f1d619b8f281f24fd939c2ccbbb5f12cb59 not found: ID does not exist" containerID="2a7f2024bf0037a3c4819f4d4c6b3f1d619b8f281f24fd939c2ccbbb5f12cb59"
Sep 16 16:52:48.732838 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:48.732799    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a7f2024bf0037a3c4819f4d4c6b3f1d619b8f281f24fd939c2ccbbb5f12cb59"} err="failed to get container status \"2a7f2024bf0037a3c4819f4d4c6b3f1d619b8f281f24fd939c2ccbbb5f12cb59\": rpc error: code = NotFound desc = could not find container \"2a7f2024bf0037a3c4819f4d4c6b3f1d619b8f281f24fd939c2ccbbb5f12cb59\": container with ID starting with 2a7f2024bf0037a3c4819f4d4c6b3f1d619b8f281f24fd939c2ccbbb5f12cb59 not found: ID does not exist"
Sep 16 16:52:48.748876 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:48.748853    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz"]
Sep 16 16:52:48.754893 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:48.754874    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-41cc4972-fb476d958-6kwfz"]
Sep 16 16:52:48.849889 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:52:48.849858    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d976837-1d20-4819-bd50-6b01ceb255b6" path="/var/lib/kubelet/pods/0d976837-1d20-4819-bd50-6b01ceb255b6/volumes"
Sep 16 16:54:42.081831 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.081757    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"]
Sep 16 16:54:42.082313 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.082203    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d976837-1d20-4819-bd50-6b01ceb255b6" containerName="payload-pre-processing"
Sep 16 16:54:42.082313 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.082217    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d976837-1d20-4819-bd50-6b01ceb255b6" containerName="payload-pre-processing"
Sep 16 16:54:42.082313 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.082285    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d976837-1d20-4819-bd50-6b01ceb255b6" containerName="payload-pre-processing"
Sep 16 16:54:42.085352 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.085331    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:42.088408 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.088387    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-708c6a\"/\"kube-root-ca.crt\""
Sep 16 16:54:42.088858 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.088834    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-708c6a\"/\"default-dockercfg-bsjvn\""
Sep 16 16:54:42.089002 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.088976    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-708c6a\"/\"autores-model-708c6a-kserve-self-signed-certs\""
Sep 16 16:54:42.089146 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.089126    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-708c6a\"/\"openshift-service-ca.crt\""
Sep 16 16:54:42.095304 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.095281    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"]
Sep 16 16:54:42.234356 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.234326    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rls8\" (UniqueName: \"kubernetes.io/projected/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-kube-api-access-6rls8\") pod \"autores-model-708c6a-kserve-955c84bb7-hkwwk\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") " pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:42.234546 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.234387    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-home\") pod \"autores-model-708c6a-kserve-955c84bb7-hkwwk\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") " pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:42.234546 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.234405    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-model-cache\") pod \"autores-model-708c6a-kserve-955c84bb7-hkwwk\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") " pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:42.234546 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.234422    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-tmp-dir\") pod \"autores-model-708c6a-kserve-955c84bb7-hkwwk\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") " pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:42.234546 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.234500    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-tls-certs\") pod \"autores-model-708c6a-kserve-955c84bb7-hkwwk\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") " pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:42.234688 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.234553    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-dshm\") pod \"autores-model-708c6a-kserve-955c84bb7-hkwwk\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") " pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:42.336079 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.335975    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rls8\" (UniqueName: \"kubernetes.io/projected/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-kube-api-access-6rls8\") pod \"autores-model-708c6a-kserve-955c84bb7-hkwwk\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") " pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:42.336221 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.336109    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-home\") pod \"autores-model-708c6a-kserve-955c84bb7-hkwwk\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") " pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:42.336221 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.336135    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-model-cache\") pod \"autores-model-708c6a-kserve-955c84bb7-hkwwk\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") " pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:42.336221 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.336159    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-tmp-dir\") pod \"autores-model-708c6a-kserve-955c84bb7-hkwwk\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") " pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:42.336221 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.336198    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-tls-certs\") pod \"autores-model-708c6a-kserve-955c84bb7-hkwwk\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") " pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:42.336450 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.336243    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-dshm\") pod \"autores-model-708c6a-kserve-955c84bb7-hkwwk\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") " pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:42.336590 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.336564    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-home\") pod \"autores-model-708c6a-kserve-955c84bb7-hkwwk\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") " pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:42.336649 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.336590    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-tmp-dir\") pod \"autores-model-708c6a-kserve-955c84bb7-hkwwk\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") " pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:42.336649 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.336623    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-model-cache\") pod \"autores-model-708c6a-kserve-955c84bb7-hkwwk\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") " pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:42.338676 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.338653    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-dshm\") pod \"autores-model-708c6a-kserve-955c84bb7-hkwwk\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") " pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:42.338770 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.338660    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-tls-certs\") pod \"autores-model-708c6a-kserve-955c84bb7-hkwwk\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") " pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:42.350620 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.350585    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rls8\" (UniqueName: \"kubernetes.io/projected/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-kube-api-access-6rls8\") pod \"autores-model-708c6a-kserve-955c84bb7-hkwwk\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") " pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:42.395359 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.395330    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:42.668249 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:42.668198    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"]
Sep 16 16:54:42.670783 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:54:42.670755    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcef0a0d2_9293_47bb_a2e5_6d84ee858f6b.slice/crio-5370bf63a9a96409efe06b0186c9860c43db45a4090df590897d957ac640336f WatchSource:0}: Error finding container 5370bf63a9a96409efe06b0186c9860c43db45a4090df590897d957ac640336f: Status 404 returned error can't find the container with id 5370bf63a9a96409efe06b0186c9860c43db45a4090df590897d957ac640336f
Sep 16 16:54:43.285018 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:43.284976    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk" event={"ID":"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b","Type":"ContainerStarted","Data":"78afd4307a738ab43cc541a5199e22345e6f15c9d6efd5d51981997bdbd9ed0d"}
Sep 16 16:54:43.285400 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:43.285030    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk" event={"ID":"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b","Type":"ContainerStarted","Data":"5370bf63a9a96409efe06b0186c9860c43db45a4090df590897d957ac640336f"}
Sep 16 16:54:43.305049 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:43.305003    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk" podStartSLOduration=1.3049879899999999 podStartE2EDuration="1.30498799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:54:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:54:43.303112216 +0000 UTC m=+1083.065073662" watchObservedRunningTime="2026-09-16 16:54:43.30498799 +0000 UTC m=+1083.066949436"
Sep 16 16:54:52.395525 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:52.395486    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:52.396089 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:52.395540    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:52.408306 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:52.408280    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:54:53.343743 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:54:53.343714    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:55:24.018221 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.018186    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"]
Sep 16 16:55:24.018728 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.018385    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l" podUID="5ff37461-4b79-454d-a0bb-ec6a55b8e302" containerName="maas-api" containerID="cri-o://80af277bf76881e6f786e985ef094eb31ed4d3e938d7da186d13be4ffec97d4e" gracePeriod=30
Sep 16 16:55:24.276599 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.276544    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"
Sep 16 16:55:24.433682 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.433650    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn7pt\" (UniqueName: \"kubernetes.io/projected/5ff37461-4b79-454d-a0bb-ec6a55b8e302-kube-api-access-jn7pt\") pod \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\" (UID: \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\") "
Sep 16 16:55:24.433850 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.433695    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ff37461-4b79-454d-a0bb-ec6a55b8e302-maas-api-tls\") pod \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\" (UID: \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\") "
Sep 16 16:55:24.433850 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.433829    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff37461-4b79-454d-a0bb-ec6a55b8e302-metrics-certs\") pod \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\" (UID: \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\") "
Sep 16 16:55:24.433962 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.433927    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff37461-4b79-454d-a0bb-ec6a55b8e302-openshift-service-ca\") pod \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\" (UID: \"5ff37461-4b79-454d-a0bb-ec6a55b8e302\") "
Sep 16 16:55:24.434256 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.434231    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff37461-4b79-454d-a0bb-ec6a55b8e302-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5ff37461-4b79-454d-a0bb-ec6a55b8e302" (UID: "5ff37461-4b79-454d-a0bb-ec6a55b8e302"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:55:24.435989 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.435960    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff37461-4b79-454d-a0bb-ec6a55b8e302-kube-api-access-jn7pt" (OuterVolumeSpecName: "kube-api-access-jn7pt") pod "5ff37461-4b79-454d-a0bb-ec6a55b8e302" (UID: "5ff37461-4b79-454d-a0bb-ec6a55b8e302"). InnerVolumeSpecName "kube-api-access-jn7pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:55:24.436270 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.436247    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff37461-4b79-454d-a0bb-ec6a55b8e302-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5ff37461-4b79-454d-a0bb-ec6a55b8e302" (UID: "5ff37461-4b79-454d-a0bb-ec6a55b8e302"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:55:24.436334 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.436266    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff37461-4b79-454d-a0bb-ec6a55b8e302-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5ff37461-4b79-454d-a0bb-ec6a55b8e302" (UID: "5ff37461-4b79-454d-a0bb-ec6a55b8e302"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:55:24.497954 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.497909    2556 generic.go:356] "Generic (PLEG): container finished" podID="5ff37461-4b79-454d-a0bb-ec6a55b8e302" containerID="80af277bf76881e6f786e985ef094eb31ed4d3e938d7da186d13be4ffec97d4e" exitCode=0
Sep 16 16:55:24.498117 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.497975    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"
Sep 16 16:55:24.498117 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.497996    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l" event={"ID":"5ff37461-4b79-454d-a0bb-ec6a55b8e302","Type":"ContainerDied","Data":"80af277bf76881e6f786e985ef094eb31ed4d3e938d7da186d13be4ffec97d4e"}
Sep 16 16:55:24.498117 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.498074    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l" event={"ID":"5ff37461-4b79-454d-a0bb-ec6a55b8e302","Type":"ContainerDied","Data":"1f03866b287b3a0d105f900fc5619aa87e411137f6f569f8c783c26f35a45792"}
Sep 16 16:55:24.498117 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.498097    2556 scope.go:117] "RemoveContainer" containerID="80af277bf76881e6f786e985ef094eb31ed4d3e938d7da186d13be4ffec97d4e"
Sep 16 16:55:24.507236 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.507219    2556 scope.go:117] "RemoveContainer" containerID="80af277bf76881e6f786e985ef094eb31ed4d3e938d7da186d13be4ffec97d4e"
Sep 16 16:55:24.507509 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:55:24.507492    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80af277bf76881e6f786e985ef094eb31ed4d3e938d7da186d13be4ffec97d4e\": container with ID starting with 80af277bf76881e6f786e985ef094eb31ed4d3e938d7da186d13be4ffec97d4e not found: ID does not exist" containerID="80af277bf76881e6f786e985ef094eb31ed4d3e938d7da186d13be4ffec97d4e"
Sep 16 16:55:24.507567 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.507521    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80af277bf76881e6f786e985ef094eb31ed4d3e938d7da186d13be4ffec97d4e"} err="failed to get container status \"80af277bf76881e6f786e985ef094eb31ed4d3e938d7da186d13be4ffec97d4e\": rpc error: code = NotFound desc = could not find container \"80af277bf76881e6f786e985ef094eb31ed4d3e938d7da186d13be4ffec97d4e\": container with ID starting with 80af277bf76881e6f786e985ef094eb31ed4d3e938d7da186d13be4ffec97d4e not found: ID does not exist"
Sep 16 16:55:24.523070 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.523044    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"]
Sep 16 16:55:24.525772 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.525753    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-708c6a-79568ff975-bpw5l"]
Sep 16 16:55:24.535312 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.535254    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff37461-4b79-454d-a0bb-ec6a55b8e302-metrics-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:24.535312 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.535275    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff37461-4b79-454d-a0bb-ec6a55b8e302-openshift-service-ca\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:24.535312 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.535290    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn7pt\" (UniqueName: \"kubernetes.io/projected/5ff37461-4b79-454d-a0bb-ec6a55b8e302-kube-api-access-jn7pt\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:24.535453 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.535303    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ff37461-4b79-454d-a0bb-ec6a55b8e302-maas-api-tls\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:24.848828 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:24.848747    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ff37461-4b79-454d-a0bb-ec6a55b8e302" path="/var/lib/kubelet/pods/5ff37461-4b79-454d-a0bb-ec6a55b8e302/volumes"
Sep 16 16:55:29.616959 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:29.616918    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"]
Sep 16 16:55:37.455690 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:37.455643    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"]
Sep 16 16:55:37.456146 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:37.455896    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs" podUID="e3a45e34-74c4-4028-b1bc-06d220aab39a" containerName="maas-api" containerID="cri-o://6ed5d74388426d0c373d3f6eda2a9c98edede042f476253c9e9610b61de74710" gracePeriod=30
Sep 16 16:55:37.722209 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:37.722139    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"
Sep 16 16:55:37.755431 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:37.755396    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a45e34-74c4-4028-b1bc-06d220aab39a-openshift-service-ca\") pod \"e3a45e34-74c4-4028-b1bc-06d220aab39a\" (UID: \"e3a45e34-74c4-4028-b1bc-06d220aab39a\") "
Sep 16 16:55:37.755431 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:37.755432    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3a45e34-74c4-4028-b1bc-06d220aab39a-metrics-certs\") pod \"e3a45e34-74c4-4028-b1bc-06d220aab39a\" (UID: \"e3a45e34-74c4-4028-b1bc-06d220aab39a\") "
Sep 16 16:55:37.755693 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:37.755465    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3a45e34-74c4-4028-b1bc-06d220aab39a-maas-api-tls\") pod \"e3a45e34-74c4-4028-b1bc-06d220aab39a\" (UID: \"e3a45e34-74c4-4028-b1bc-06d220aab39a\") "
Sep 16 16:55:37.755693 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:37.755534    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqcf5\" (UniqueName: \"kubernetes.io/projected/e3a45e34-74c4-4028-b1bc-06d220aab39a-kube-api-access-kqcf5\") pod \"e3a45e34-74c4-4028-b1bc-06d220aab39a\" (UID: \"e3a45e34-74c4-4028-b1bc-06d220aab39a\") "
Sep 16 16:55:37.755810 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:37.755764    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3a45e34-74c4-4028-b1bc-06d220aab39a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e3a45e34-74c4-4028-b1bc-06d220aab39a" (UID: "e3a45e34-74c4-4028-b1bc-06d220aab39a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:55:37.757723 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:37.757697    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3a45e34-74c4-4028-b1bc-06d220aab39a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e3a45e34-74c4-4028-b1bc-06d220aab39a" (UID: "e3a45e34-74c4-4028-b1bc-06d220aab39a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:55:37.757723 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:37.757715    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3a45e34-74c4-4028-b1bc-06d220aab39a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e3a45e34-74c4-4028-b1bc-06d220aab39a" (UID: "e3a45e34-74c4-4028-b1bc-06d220aab39a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:55:37.758034 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:37.758000    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a45e34-74c4-4028-b1bc-06d220aab39a-kube-api-access-kqcf5" (OuterVolumeSpecName: "kube-api-access-kqcf5") pod "e3a45e34-74c4-4028-b1bc-06d220aab39a" (UID: "e3a45e34-74c4-4028-b1bc-06d220aab39a"). InnerVolumeSpecName "kube-api-access-kqcf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:55:37.856308 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:37.856271    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqcf5\" (UniqueName: \"kubernetes.io/projected/e3a45e34-74c4-4028-b1bc-06d220aab39a-kube-api-access-kqcf5\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:37.856308 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:37.856310    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3a45e34-74c4-4028-b1bc-06d220aab39a-openshift-service-ca\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:37.856499 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:37.856326    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3a45e34-74c4-4028-b1bc-06d220aab39a-metrics-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:37.856499 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:37.856339    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e3a45e34-74c4-4028-b1bc-06d220aab39a-maas-api-tls\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:38.569432 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:38.569391    2556 generic.go:356] "Generic (PLEG): container finished" podID="e3a45e34-74c4-4028-b1bc-06d220aab39a" containerID="6ed5d74388426d0c373d3f6eda2a9c98edede042f476253c9e9610b61de74710" exitCode=0
Sep 16 16:55:38.569825 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:38.569442    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"
Sep 16 16:55:38.569825 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:38.569483    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs" event={"ID":"e3a45e34-74c4-4028-b1bc-06d220aab39a","Type":"ContainerDied","Data":"6ed5d74388426d0c373d3f6eda2a9c98edede042f476253c9e9610b61de74710"}
Sep 16 16:55:38.569825 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:38.569538    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs" event={"ID":"e3a45e34-74c4-4028-b1bc-06d220aab39a","Type":"ContainerDied","Data":"cab2dad190b2b710aa14e2dc85f813e8725d810972edf120bcbd1c0468dc30c9"}
Sep 16 16:55:38.569825 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:38.569563    2556 scope.go:117] "RemoveContainer" containerID="6ed5d74388426d0c373d3f6eda2a9c98edede042f476253c9e9610b61de74710"
Sep 16 16:55:38.578850 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:38.578836    2556 scope.go:117] "RemoveContainer" containerID="6ed5d74388426d0c373d3f6eda2a9c98edede042f476253c9e9610b61de74710"
Sep 16 16:55:38.579106 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:55:38.579090    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ed5d74388426d0c373d3f6eda2a9c98edede042f476253c9e9610b61de74710\": container with ID starting with 6ed5d74388426d0c373d3f6eda2a9c98edede042f476253c9e9610b61de74710 not found: ID does not exist" containerID="6ed5d74388426d0c373d3f6eda2a9c98edede042f476253c9e9610b61de74710"
Sep 16 16:55:38.579158 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:38.579114    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ed5d74388426d0c373d3f6eda2a9c98edede042f476253c9e9610b61de74710"} err="failed to get container status \"6ed5d74388426d0c373d3f6eda2a9c98edede042f476253c9e9610b61de74710\": rpc error: code = NotFound desc = could not find container \"6ed5d74388426d0c373d3f6eda2a9c98edede042f476253c9e9610b61de74710\": container with ID starting with 6ed5d74388426d0c373d3f6eda2a9c98edede042f476253c9e9610b61de74710 not found: ID does not exist"
Sep 16 16:55:38.601542 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:38.601490    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"]
Sep 16 16:55:38.603751 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:38.603728    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-eb2ab2-6d6b57ff4f-nswfs"]
Sep 16 16:55:38.851316 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:38.851230    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3a45e34-74c4-4028-b1bc-06d220aab39a" path="/var/lib/kubelet/pods/e3a45e34-74c4-4028-b1bc-06d220aab39a/volumes"
Sep 16 16:55:44.636111 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:44.636065    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk" podUID="cef0a0d2-9293-47bb-a2e5-6d84ee858f6b" containerName="main" containerID="cri-o://78afd4307a738ab43cc541a5199e22345e6f15c9d6efd5d51981997bdbd9ed0d" gracePeriod=45
Sep 16 16:55:44.896497 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:44.896436    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:55:44.920918 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:44.920891    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-dshm\") pod \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") "
Sep 16 16:55:44.921059 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:44.920951    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-model-cache\") pod \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") "
Sep 16 16:55:44.921059 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:44.920993    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-home\") pod \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") "
Sep 16 16:55:44.921059 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:44.921045    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-tmp-dir\") pod \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") "
Sep 16 16:55:44.921205 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:44.921069    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-tls-certs\") pod \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") "
Sep 16 16:55:44.921205 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:44.921124    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rls8\" (UniqueName: \"kubernetes.io/projected/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-kube-api-access-6rls8\") pod \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\" (UID: \"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b\") "
Sep 16 16:55:44.921312 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:44.921249    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-model-cache" (OuterVolumeSpecName: "model-cache") pod "cef0a0d2-9293-47bb-a2e5-6d84ee858f6b" (UID: "cef0a0d2-9293-47bb-a2e5-6d84ee858f6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:55:44.921312 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:44.921298    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-home" (OuterVolumeSpecName: "home") pod "cef0a0d2-9293-47bb-a2e5-6d84ee858f6b" (UID: "cef0a0d2-9293-47bb-a2e5-6d84ee858f6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:55:44.921404 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:44.921333    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cef0a0d2-9293-47bb-a2e5-6d84ee858f6b" (UID: "cef0a0d2-9293-47bb-a2e5-6d84ee858f6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:55:44.921493 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:44.921473    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-model-cache\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:44.921565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:44.921501    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-home\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:44.921565 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:44.921516    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-tmp-dir\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:44.923144 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:44.923115    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-dshm" (OuterVolumeSpecName: "dshm") pod "cef0a0d2-9293-47bb-a2e5-6d84ee858f6b" (UID: "cef0a0d2-9293-47bb-a2e5-6d84ee858f6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:55:44.923242 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:44.923220    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-kube-api-access-6rls8" (OuterVolumeSpecName: "kube-api-access-6rls8") pod "cef0a0d2-9293-47bb-a2e5-6d84ee858f6b" (UID: "cef0a0d2-9293-47bb-a2e5-6d84ee858f6b"). InnerVolumeSpecName "kube-api-access-6rls8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:55:44.923306 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:44.923284    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cef0a0d2-9293-47bb-a2e5-6d84ee858f6b" (UID: "cef0a0d2-9293-47bb-a2e5-6d84ee858f6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:55:45.022422 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:45.022385    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-dshm\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:45.022422 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:45.022415    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-tls-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:45.022422 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:45.022426    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rls8\" (UniqueName: \"kubernetes.io/projected/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b-kube-api-access-6rls8\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:45.604886 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:45.604848    2556 generic.go:356] "Generic (PLEG): container finished" podID="cef0a0d2-9293-47bb-a2e5-6d84ee858f6b" containerID="78afd4307a738ab43cc541a5199e22345e6f15c9d6efd5d51981997bdbd9ed0d" exitCode=0
Sep 16 16:55:45.605088 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:45.604909    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"
Sep 16 16:55:45.605088 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:45.604907    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk" event={"ID":"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b","Type":"ContainerDied","Data":"78afd4307a738ab43cc541a5199e22345e6f15c9d6efd5d51981997bdbd9ed0d"}
Sep 16 16:55:45.605088 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:45.605001    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk" event={"ID":"cef0a0d2-9293-47bb-a2e5-6d84ee858f6b","Type":"ContainerDied","Data":"5370bf63a9a96409efe06b0186c9860c43db45a4090df590897d957ac640336f"}
Sep 16 16:55:45.605088 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:45.605025    2556 scope.go:117] "RemoveContainer" containerID="78afd4307a738ab43cc541a5199e22345e6f15c9d6efd5d51981997bdbd9ed0d"
Sep 16 16:55:45.613783 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:45.613764    2556 scope.go:117] "RemoveContainer" containerID="78afd4307a738ab43cc541a5199e22345e6f15c9d6efd5d51981997bdbd9ed0d"
Sep 16 16:55:45.614092 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:55:45.614073    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78afd4307a738ab43cc541a5199e22345e6f15c9d6efd5d51981997bdbd9ed0d\": container with ID starting with 78afd4307a738ab43cc541a5199e22345e6f15c9d6efd5d51981997bdbd9ed0d not found: ID does not exist" containerID="78afd4307a738ab43cc541a5199e22345e6f15c9d6efd5d51981997bdbd9ed0d"
Sep 16 16:55:45.614149 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:45.614101    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78afd4307a738ab43cc541a5199e22345e6f15c9d6efd5d51981997bdbd9ed0d"} err="failed to get container status \"78afd4307a738ab43cc541a5199e22345e6f15c9d6efd5d51981997bdbd9ed0d\": rpc error: code = NotFound desc = could not find container \"78afd4307a738ab43cc541a5199e22345e6f15c9d6efd5d51981997bdbd9ed0d\": container with ID starting with 78afd4307a738ab43cc541a5199e22345e6f15c9d6efd5d51981997bdbd9ed0d not found: ID does not exist"
Sep 16 16:55:45.625246 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:45.625216    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"]
Sep 16 16:55:45.631054 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:45.631033    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-708c6a/autores-model-708c6a-kserve-955c84bb7-hkwwk"]
Sep 16 16:55:45.859167 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:45.859074    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"]
Sep 16 16:55:45.859650 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:45.859322    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6" podUID="4385870d-6cac-4730-a6bd-f0a69ea8ee0f" containerName="maas-api" containerID="cri-o://47795b553fa32d6f273b8cd0a074ee7fcebac2620e46d5b95a738d2260b44d2d" gracePeriod=30
Sep 16 16:55:46.110953 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.110869    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"
Sep 16 16:55:46.234400 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.234357    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-maas-api-tls\") pod \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\" (UID: \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\") "
Sep 16 16:55:46.234593 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.234420    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-js4pz\" (UniqueName: \"kubernetes.io/projected/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-kube-api-access-js4pz\") pod \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\" (UID: \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\") "
Sep 16 16:55:46.234593 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.234445    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-metrics-certs\") pod \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\" (UID: \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\") "
Sep 16 16:55:46.234593 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.234496    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-openshift-service-ca\") pod \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\" (UID: \"4385870d-6cac-4730-a6bd-f0a69ea8ee0f\") "
Sep 16 16:55:46.235034 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.235001    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4385870d-6cac-4730-a6bd-f0a69ea8ee0f" (UID: "4385870d-6cac-4730-a6bd-f0a69ea8ee0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:55:46.236809 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.236783    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4385870d-6cac-4730-a6bd-f0a69ea8ee0f" (UID: "4385870d-6cac-4730-a6bd-f0a69ea8ee0f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:55:46.236981 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.236807    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-kube-api-access-js4pz" (OuterVolumeSpecName: "kube-api-access-js4pz") pod "4385870d-6cac-4730-a6bd-f0a69ea8ee0f" (UID: "4385870d-6cac-4730-a6bd-f0a69ea8ee0f"). InnerVolumeSpecName "kube-api-access-js4pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:55:46.237204 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.237173    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4385870d-6cac-4730-a6bd-f0a69ea8ee0f" (UID: "4385870d-6cac-4730-a6bd-f0a69ea8ee0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:55:46.335539 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.335507    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js4pz\" (UniqueName: \"kubernetes.io/projected/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-kube-api-access-js4pz\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:46.335539 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.335536    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-metrics-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:46.335737 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.335549    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-openshift-service-ca\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:46.335737 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.335561    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4385870d-6cac-4730-a6bd-f0a69ea8ee0f-maas-api-tls\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:46.612656 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.612624    2556 generic.go:356] "Generic (PLEG): container finished" podID="4385870d-6cac-4730-a6bd-f0a69ea8ee0f" containerID="47795b553fa32d6f273b8cd0a074ee7fcebac2620e46d5b95a738d2260b44d2d" exitCode=0
Sep 16 16:55:46.612820 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.612665    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"
Sep 16 16:55:46.612820 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.612665    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6" event={"ID":"4385870d-6cac-4730-a6bd-f0a69ea8ee0f","Type":"ContainerDied","Data":"47795b553fa32d6f273b8cd0a074ee7fcebac2620e46d5b95a738d2260b44d2d"}
Sep 16 16:55:46.612820 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.612747    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6" event={"ID":"4385870d-6cac-4730-a6bd-f0a69ea8ee0f","Type":"ContainerDied","Data":"365533a493748a8929ad48819089166f20919d47cc4791789c9709f1a22bb1c3"}
Sep 16 16:55:46.612820 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.612773    2556 scope.go:117] "RemoveContainer" containerID="47795b553fa32d6f273b8cd0a074ee7fcebac2620e46d5b95a738d2260b44d2d"
Sep 16 16:55:46.621741 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.621724    2556 scope.go:117] "RemoveContainer" containerID="47795b553fa32d6f273b8cd0a074ee7fcebac2620e46d5b95a738d2260b44d2d"
Sep 16 16:55:46.622036 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:55:46.622013    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47795b553fa32d6f273b8cd0a074ee7fcebac2620e46d5b95a738d2260b44d2d\": container with ID starting with 47795b553fa32d6f273b8cd0a074ee7fcebac2620e46d5b95a738d2260b44d2d not found: ID does not exist" containerID="47795b553fa32d6f273b8cd0a074ee7fcebac2620e46d5b95a738d2260b44d2d"
Sep 16 16:55:46.622132 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.622041    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47795b553fa32d6f273b8cd0a074ee7fcebac2620e46d5b95a738d2260b44d2d"} err="failed to get container status \"47795b553fa32d6f273b8cd0a074ee7fcebac2620e46d5b95a738d2260b44d2d\": rpc error: code = NotFound desc = could not find container \"47795b553fa32d6f273b8cd0a074ee7fcebac2620e46d5b95a738d2260b44d2d\": container with ID starting with 47795b553fa32d6f273b8cd0a074ee7fcebac2620e46d5b95a738d2260b44d2d not found: ID does not exist"
Sep 16 16:55:46.638712 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.638685    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"]
Sep 16 16:55:46.641096 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.641074    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f604b4-67c5b966f-p8jb6"]
Sep 16 16:55:46.849299 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.849259    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4385870d-6cac-4730-a6bd-f0a69ea8ee0f" path="/var/lib/kubelet/pods/4385870d-6cac-4730-a6bd-f0a69ea8ee0f/volumes"
Sep 16 16:55:46.849732 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:46.849711    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cef0a0d2-9293-47bb-a2e5-6d84ee858f6b" path="/var/lib/kubelet/pods/cef0a0d2-9293-47bb-a2e5-6d84ee858f6b/volumes"
Sep 16 16:55:52.888758 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:52.888727    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"]
Sep 16 16:55:52.889184 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:52.888972    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9" podUID="7d617c78-e49d-4cc3-b439-092189aaea66" containerName="istio-proxy" containerID="cri-o://68fe1ef39177260ebbab8a6852b29f348d26b42ca07009978808077a8b4d3e69" gracePeriod=30
Sep 16 16:55:54.185629 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:54.185585    2556 patch_prober.go:28] interesting pod/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 16:55:54.186028 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:54.185654    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9" podUID="7d617c78-e49d-4cc3-b439-092189aaea66" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 16:55:58.131074 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.131043    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:55:58.238744 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.238718    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-istio-data\") pod \"7d617c78-e49d-4cc3-b439-092189aaea66\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") "
Sep 16 16:55:58.238924 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.238765    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d617c78-e49d-4cc3-b439-092189aaea66-istio-podinfo\") pod \"7d617c78-e49d-4cc3-b439-092189aaea66\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") "
Sep 16 16:55:58.238924 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.238796    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d617c78-e49d-4cc3-b439-092189aaea66-istio-token\") pod \"7d617c78-e49d-4cc3-b439-092189aaea66\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") "
Sep 16 16:55:58.238924 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.238821    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-credential-socket\") pod \"7d617c78-e49d-4cc3-b439-092189aaea66\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") "
Sep 16 16:55:58.239122 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.238970    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-istio-envoy\") pod \"7d617c78-e49d-4cc3-b439-092189aaea66\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") "
Sep 16 16:55:58.239122 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.239032    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-workload-certs\") pod \"7d617c78-e49d-4cc3-b439-092189aaea66\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") "
Sep 16 16:55:58.239122 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.239076    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-istio-data" (OuterVolumeSpecName: "istio-data") pod "7d617c78-e49d-4cc3-b439-092189aaea66" (UID: "7d617c78-e49d-4cc3-b439-092189aaea66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:55:58.239122 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.239113    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7d617c78-e49d-4cc3-b439-092189aaea66" (UID: "7d617c78-e49d-4cc3-b439-092189aaea66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:55:58.239306 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.239123    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d617c78-e49d-4cc3-b439-092189aaea66-istiod-ca-cert\") pod \"7d617c78-e49d-4cc3-b439-092189aaea66\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") "
Sep 16 16:55:58.239306 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.239148    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-workload-socket\") pod \"7d617c78-e49d-4cc3-b439-092189aaea66\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") "
Sep 16 16:55:58.239306 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.239184    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4725f\" (UniqueName: \"kubernetes.io/projected/7d617c78-e49d-4cc3-b439-092189aaea66-kube-api-access-4725f\") pod \"7d617c78-e49d-4cc3-b439-092189aaea66\" (UID: \"7d617c78-e49d-4cc3-b439-092189aaea66\") "
Sep 16 16:55:58.239306 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.239269    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7d617c78-e49d-4cc3-b439-092189aaea66" (UID: "7d617c78-e49d-4cc3-b439-092189aaea66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:55:58.239504 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.239400    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7d617c78-e49d-4cc3-b439-092189aaea66" (UID: "7d617c78-e49d-4cc3-b439-092189aaea66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:55:58.239504 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.239457    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d617c78-e49d-4cc3-b439-092189aaea66-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7d617c78-e49d-4cc3-b439-092189aaea66" (UID: "7d617c78-e49d-4cc3-b439-092189aaea66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:55:58.239504 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.239498    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-credential-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:58.239625 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.239517    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-workload-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:58.239625 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.239531    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-workload-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:58.239625 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.239544    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-istio-data\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:58.241083 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.241060    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d617c78-e49d-4cc3-b439-092189aaea66-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7d617c78-e49d-4cc3-b439-092189aaea66" (UID: "7d617c78-e49d-4cc3-b439-092189aaea66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 16:55:58.241083 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.241058    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7d617c78-e49d-4cc3-b439-092189aaea66" (UID: "7d617c78-e49d-4cc3-b439-092189aaea66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:55:58.241370 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.241346    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d617c78-e49d-4cc3-b439-092189aaea66-istio-token" (OuterVolumeSpecName: "istio-token") pod "7d617c78-e49d-4cc3-b439-092189aaea66" (UID: "7d617c78-e49d-4cc3-b439-092189aaea66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:55:58.241458 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.241414    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d617c78-e49d-4cc3-b439-092189aaea66-kube-api-access-4725f" (OuterVolumeSpecName: "kube-api-access-4725f") pod "7d617c78-e49d-4cc3-b439-092189aaea66" (UID: "7d617c78-e49d-4cc3-b439-092189aaea66"). InnerVolumeSpecName "kube-api-access-4725f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:55:58.340762 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.340727    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d617c78-e49d-4cc3-b439-092189aaea66-istiod-ca-cert\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:58.340762 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.340760    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4725f\" (UniqueName: \"kubernetes.io/projected/7d617c78-e49d-4cc3-b439-092189aaea66-kube-api-access-4725f\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:58.340762 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.340770    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d617c78-e49d-4cc3-b439-092189aaea66-istio-podinfo\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:58.340928 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.340778    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d617c78-e49d-4cc3-b439-092189aaea66-istio-token\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:58.340928 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.340786    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d617c78-e49d-4cc3-b439-092189aaea66-istio-envoy\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:55:58.674748 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.674666    2556 generic.go:356] "Generic (PLEG): container finished" podID="7d617c78-e49d-4cc3-b439-092189aaea66" containerID="68fe1ef39177260ebbab8a6852b29f348d26b42ca07009978808077a8b4d3e69" exitCode=0
Sep 16 16:55:58.674748 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.674716    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"
Sep 16 16:55:58.674912 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.674751    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9" event={"ID":"7d617c78-e49d-4cc3-b439-092189aaea66","Type":"ContainerDied","Data":"68fe1ef39177260ebbab8a6852b29f348d26b42ca07009978808077a8b4d3e69"}
Sep 16 16:55:58.674912 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.674806    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9" event={"ID":"7d617c78-e49d-4cc3-b439-092189aaea66","Type":"ContainerDied","Data":"c7b2766a41d224fa4f9325ddaa5cb0cea656355fdd967768402a6f7fc7355767"}
Sep 16 16:55:58.674912 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.674829    2556 scope.go:117] "RemoveContainer" containerID="68fe1ef39177260ebbab8a6852b29f348d26b42ca07009978808077a8b4d3e69"
Sep 16 16:55:58.684700 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.684687    2556 scope.go:117] "RemoveContainer" containerID="8dc2567b03f7650153c22add0081cc0835cb631ea38dbc7aa49d37f1ad5930c9"
Sep 16 16:55:58.692257 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.692235    2556 scope.go:117] "RemoveContainer" containerID="68fe1ef39177260ebbab8a6852b29f348d26b42ca07009978808077a8b4d3e69"
Sep 16 16:55:58.692491 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:55:58.692475    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68fe1ef39177260ebbab8a6852b29f348d26b42ca07009978808077a8b4d3e69\": container with ID starting with 68fe1ef39177260ebbab8a6852b29f348d26b42ca07009978808077a8b4d3e69 not found: ID does not exist" containerID="68fe1ef39177260ebbab8a6852b29f348d26b42ca07009978808077a8b4d3e69"
Sep 16 16:55:58.692545 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.692498    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68fe1ef39177260ebbab8a6852b29f348d26b42ca07009978808077a8b4d3e69"} err="failed to get container status \"68fe1ef39177260ebbab8a6852b29f348d26b42ca07009978808077a8b4d3e69\": rpc error: code = NotFound desc = could not find container \"68fe1ef39177260ebbab8a6852b29f348d26b42ca07009978808077a8b4d3e69\": container with ID starting with 68fe1ef39177260ebbab8a6852b29f348d26b42ca07009978808077a8b4d3e69 not found: ID does not exist"
Sep 16 16:55:58.692545 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.692512    2556 scope.go:117] "RemoveContainer" containerID="8dc2567b03f7650153c22add0081cc0835cb631ea38dbc7aa49d37f1ad5930c9"
Sep 16 16:55:58.692751 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:55:58.692734    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc2567b03f7650153c22add0081cc0835cb631ea38dbc7aa49d37f1ad5930c9\": container with ID starting with 8dc2567b03f7650153c22add0081cc0835cb631ea38dbc7aa49d37f1ad5930c9 not found: ID does not exist" containerID="8dc2567b03f7650153c22add0081cc0835cb631ea38dbc7aa49d37f1ad5930c9"
Sep 16 16:55:58.692795 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.692756    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc2567b03f7650153c22add0081cc0835cb631ea38dbc7aa49d37f1ad5930c9"} err="failed to get container status \"8dc2567b03f7650153c22add0081cc0835cb631ea38dbc7aa49d37f1ad5930c9\": rpc error: code = NotFound desc = could not find container \"8dc2567b03f7650153c22add0081cc0835cb631ea38dbc7aa49d37f1ad5930c9\": container with ID starting with 8dc2567b03f7650153c22add0081cc0835cb631ea38dbc7aa49d37f1ad5930c9 not found: ID does not exist"
Sep 16 16:55:58.698455 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.698430    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"]
Sep 16 16:55:58.704783 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.704764    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f604b4-openshift-default-6d747fd9dd-f7wv9"]
Sep 16 16:55:58.849805 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:55:58.849772    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d617c78-e49d-4cc3-b439-092189aaea66" path="/var/lib/kubelet/pods/7d617c78-e49d-4cc3-b439-092189aaea66/volumes"
Sep 16 16:56:00.191725 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.191693    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"]
Sep 16 16:56:00.192118 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.192085    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cef0a0d2-9293-47bb-a2e5-6d84ee858f6b" containerName="main"
Sep 16 16:56:00.192118 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.192098    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cef0a0d2-9293-47bb-a2e5-6d84ee858f6b" containerName="main"
Sep 16 16:56:00.192118 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.192107    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4385870d-6cac-4730-a6bd-f0a69ea8ee0f" containerName="maas-api"
Sep 16 16:56:00.192118 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.192113    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4385870d-6cac-4730-a6bd-f0a69ea8ee0f" containerName="maas-api"
Sep 16 16:56:00.192118 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.192120    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ff37461-4b79-454d-a0bb-ec6a55b8e302" containerName="maas-api"
Sep 16 16:56:00.192328 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.192127    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ff37461-4b79-454d-a0bb-ec6a55b8e302" containerName="maas-api"
Sep 16 16:56:00.192328 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.192134    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d617c78-e49d-4cc3-b439-092189aaea66" containerName="istio-proxy"
Sep 16 16:56:00.192328 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.192139    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d617c78-e49d-4cc3-b439-092189aaea66" containerName="istio-proxy"
Sep 16 16:56:00.192328 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.192153    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d617c78-e49d-4cc3-b439-092189aaea66" containerName="istio-proxy"
Sep 16 16:56:00.192328 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.192158    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d617c78-e49d-4cc3-b439-092189aaea66" containerName="istio-proxy"
Sep 16 16:56:00.192328 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.192166    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3a45e34-74c4-4028-b1bc-06d220aab39a" containerName="maas-api"
Sep 16 16:56:00.192328 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.192171    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3a45e34-74c4-4028-b1bc-06d220aab39a" containerName="maas-api"
Sep 16 16:56:00.192328 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.192279    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4385870d-6cac-4730-a6bd-f0a69ea8ee0f" containerName="maas-api"
Sep 16 16:56:00.192328 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.192295    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3a45e34-74c4-4028-b1bc-06d220aab39a" containerName="maas-api"
Sep 16 16:56:00.192328 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.192304    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cef0a0d2-9293-47bb-a2e5-6d84ee858f6b" containerName="main"
Sep 16 16:56:00.192328 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.192314    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d617c78-e49d-4cc3-b439-092189aaea66" containerName="istio-proxy"
Sep 16 16:56:00.192328 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.192321    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d617c78-e49d-4cc3-b439-092189aaea66" containerName="istio-proxy"
Sep 16 16:56:00.192328 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.192327    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ff37461-4b79-454d-a0bb-ec6a55b8e302" containerName="maas-api"
Sep 16 16:56:00.197463 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.197443    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:00.200750 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.200728    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ea8f09\"/\"openshift-service-ca.crt\""
Sep 16 16:56:00.200750 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.200738    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ea8f09\"/\"test-model-ea8f09-kserve-self-signed-certs\""
Sep 16 16:56:00.200930 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.200739    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-ea8f09\"/\"default-dockercfg-zdjwh\""
Sep 16 16:56:00.200930 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.200819    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-ea8f09\"/\"kube-root-ca.crt\""
Sep 16 16:56:00.205906 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.205884    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"]
Sep 16 16:56:00.255736 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.255707    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hllvl\" (UniqueName: \"kubernetes.io/projected/3c72c686-7770-4d1a-a17a-de1883fb52dd-kube-api-access-hllvl\") pod \"test-model-ea8f09-kserve-6569776f95-nl97x\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") " pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:00.255904 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.255748    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-tmp-dir\") pod \"test-model-ea8f09-kserve-6569776f95-nl97x\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") " pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:00.255904 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.255770    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-model-cache\") pod \"test-model-ea8f09-kserve-6569776f95-nl97x\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") " pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:00.255904 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.255815    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c72c686-7770-4d1a-a17a-de1883fb52dd-tls-certs\") pod \"test-model-ea8f09-kserve-6569776f95-nl97x\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") " pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:00.255904 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.255854    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-home\") pod \"test-model-ea8f09-kserve-6569776f95-nl97x\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") " pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:00.256145 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.255908    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-dshm\") pod \"test-model-ea8f09-kserve-6569776f95-nl97x\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") " pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:00.357271 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.357238    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hllvl\" (UniqueName: \"kubernetes.io/projected/3c72c686-7770-4d1a-a17a-de1883fb52dd-kube-api-access-hllvl\") pod \"test-model-ea8f09-kserve-6569776f95-nl97x\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") " pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:00.357438 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.357282    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-tmp-dir\") pod \"test-model-ea8f09-kserve-6569776f95-nl97x\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") " pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:00.357438 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.357346    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-model-cache\") pod \"test-model-ea8f09-kserve-6569776f95-nl97x\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") " pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:00.357438 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.357374    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c72c686-7770-4d1a-a17a-de1883fb52dd-tls-certs\") pod \"test-model-ea8f09-kserve-6569776f95-nl97x\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") " pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:00.357438 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.357398    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-home\") pod \"test-model-ea8f09-kserve-6569776f95-nl97x\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") " pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:00.357650 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.357461    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-dshm\") pod \"test-model-ea8f09-kserve-6569776f95-nl97x\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") " pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:00.357827 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.357799    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-tmp-dir\") pod \"test-model-ea8f09-kserve-6569776f95-nl97x\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") " pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:00.357827 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.357819    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-model-cache\") pod \"test-model-ea8f09-kserve-6569776f95-nl97x\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") " pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:00.358100 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.358075    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-home\") pod \"test-model-ea8f09-kserve-6569776f95-nl97x\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") " pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:00.359734 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.359709    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-dshm\") pod \"test-model-ea8f09-kserve-6569776f95-nl97x\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") " pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:00.360028 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.360010    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c72c686-7770-4d1a-a17a-de1883fb52dd-tls-certs\") pod \"test-model-ea8f09-kserve-6569776f95-nl97x\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") " pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:00.365736 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.365714    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hllvl\" (UniqueName: \"kubernetes.io/projected/3c72c686-7770-4d1a-a17a-de1883fb52dd-kube-api-access-hllvl\") pod \"test-model-ea8f09-kserve-6569776f95-nl97x\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") " pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:00.507749 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.507689    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:00.780045 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:00.780022    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"]
Sep 16 16:56:00.782358 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:56:00.782328    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c72c686_7770_4d1a_a17a_de1883fb52dd.slice/crio-e26ecf583d2e8c2279db0780a938a0c986fb07f2dce1bb4f0464f623f44eb54e WatchSource:0}: Error finding container e26ecf583d2e8c2279db0780a938a0c986fb07f2dce1bb4f0464f623f44eb54e: Status 404 returned error can't find the container with id e26ecf583d2e8c2279db0780a938a0c986fb07f2dce1bb4f0464f623f44eb54e
Sep 16 16:56:01.691955 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:01.691902    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x" event={"ID":"3c72c686-7770-4d1a-a17a-de1883fb52dd","Type":"ContainerStarted","Data":"126e553e68cdb2dfb8884b13df53fde611d010fd423bba9336d92b65c533ac91"}
Sep 16 16:56:01.691955 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:01.691962    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x" event={"ID":"3c72c686-7770-4d1a-a17a-de1883fb52dd","Type":"ContainerStarted","Data":"e26ecf583d2e8c2279db0780a938a0c986fb07f2dce1bb4f0464f623f44eb54e"}
Sep 16 16:56:01.709614 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:01.709568    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x" podStartSLOduration=1.709554829 podStartE2EDuration="1.709554829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:56:01.70787532 +0000 UTC m=+1161.469836769" watchObservedRunningTime="2026-09-16 16:56:01.709554829 +0000 UTC m=+1161.471516272"
Sep 16 16:56:10.508776 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:10.508736    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:10.509239 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:10.508797    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:10.521328 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:10.521301    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:10.746704 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:10.746676    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:56:34.376879 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:34.376846    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"]
Sep 16 16:56:34.380605 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:34.380582    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"
Sep 16 16:56:34.382645 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:34.382621    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f3a3973e-serving-cert\""
Sep 16 16:56:34.390957 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:34.390921    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"]
Sep 16 16:56:34.436474 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:34.436450    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78b9e893-5e8e-4aab-898b-d9145db247ec-maas-api-tls\") pod \"maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv\" (UID: \"78b9e893-5e8e-4aab-898b-d9145db247ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"
Sep 16 16:56:34.436588 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:34.436479    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b9e893-5e8e-4aab-898b-d9145db247ec-openshift-service-ca\") pod \"maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv\" (UID: \"78b9e893-5e8e-4aab-898b-d9145db247ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"
Sep 16 16:56:34.436588 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:34.436507    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b9e893-5e8e-4aab-898b-d9145db247ec-metrics-certs\") pod \"maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv\" (UID: \"78b9e893-5e8e-4aab-898b-d9145db247ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"
Sep 16 16:56:34.436588 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:34.436537    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s24sk\" (UniqueName: \"kubernetes.io/projected/78b9e893-5e8e-4aab-898b-d9145db247ec-kube-api-access-s24sk\") pod \"maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv\" (UID: \"78b9e893-5e8e-4aab-898b-d9145db247ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"
Sep 16 16:56:34.537347 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:34.537315    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78b9e893-5e8e-4aab-898b-d9145db247ec-maas-api-tls\") pod \"maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv\" (UID: \"78b9e893-5e8e-4aab-898b-d9145db247ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"
Sep 16 16:56:34.537347 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:34.537350    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b9e893-5e8e-4aab-898b-d9145db247ec-openshift-service-ca\") pod \"maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv\" (UID: \"78b9e893-5e8e-4aab-898b-d9145db247ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"
Sep 16 16:56:34.537507 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:34.537378    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b9e893-5e8e-4aab-898b-d9145db247ec-metrics-certs\") pod \"maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv\" (UID: \"78b9e893-5e8e-4aab-898b-d9145db247ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"
Sep 16 16:56:34.537507 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:34.537407    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s24sk\" (UniqueName: \"kubernetes.io/projected/78b9e893-5e8e-4aab-898b-d9145db247ec-kube-api-access-s24sk\") pod \"maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv\" (UID: \"78b9e893-5e8e-4aab-898b-d9145db247ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"
Sep 16 16:56:34.538015 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:34.537989    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b9e893-5e8e-4aab-898b-d9145db247ec-openshift-service-ca\") pod \"maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv\" (UID: \"78b9e893-5e8e-4aab-898b-d9145db247ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"
Sep 16 16:56:34.539769 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:34.539751    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b9e893-5e8e-4aab-898b-d9145db247ec-metrics-certs\") pod \"maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv\" (UID: \"78b9e893-5e8e-4aab-898b-d9145db247ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"
Sep 16 16:56:34.539835 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:34.539797    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78b9e893-5e8e-4aab-898b-d9145db247ec-maas-api-tls\") pod \"maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv\" (UID: \"78b9e893-5e8e-4aab-898b-d9145db247ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"
Sep 16 16:56:34.545654 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:34.545631    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s24sk\" (UniqueName: \"kubernetes.io/projected/78b9e893-5e8e-4aab-898b-d9145db247ec-kube-api-access-s24sk\") pod \"maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv\" (UID: \"78b9e893-5e8e-4aab-898b-d9145db247ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"
Sep 16 16:56:34.693095 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:34.693070    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"
Sep 16 16:56:34.964972 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:34.964933    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"]
Sep 16 16:56:34.966926 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:56:34.966901    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b9e893_5e8e_4aab_898b_d9145db247ec.slice/crio-ad9e3012dce7373b89b64514c780a37f13643bdce001920e15b53d737a1d8492 WatchSource:0}: Error finding container ad9e3012dce7373b89b64514c780a37f13643bdce001920e15b53d737a1d8492: Status 404 returned error can't find the container with id ad9e3012dce7373b89b64514c780a37f13643bdce001920e15b53d737a1d8492
Sep 16 16:56:35.860622 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:35.860534    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv" event={"ID":"78b9e893-5e8e-4aab-898b-d9145db247ec","Type":"ContainerStarted","Data":"c4f8beccc897a2f4cdc99a1dcc61370f6c3f95c5fe50fe0661fe42bd88393cbe"}
Sep 16 16:56:35.860622 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:35.860586    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv" event={"ID":"78b9e893-5e8e-4aab-898b-d9145db247ec","Type":"ContainerStarted","Data":"ad9e3012dce7373b89b64514c780a37f13643bdce001920e15b53d737a1d8492"}
Sep 16 16:56:35.860622 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:35.860616    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"
Sep 16 16:56:35.879750 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:35.879684    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv" podStartSLOduration=1.381195599 podStartE2EDuration="1.879664599s" totalImagesPullingTime="498.469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:56:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:56:35.876677254 +0000 UTC m=+1195.638638702" watchObservedRunningTime="2026-09-16 16:56:35.879664599 +0000 UTC m=+1195.641626047"
Sep 16 16:56:41.870870 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:56:41.870836    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"
Sep 16 16:57:10.131553 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:10.131516    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"]
Sep 16 16:57:15.014432 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:15.014395    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"]
Sep 16 16:57:15.014909 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:15.014606    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv" podUID="78b9e893-5e8e-4aab-898b-d9145db247ec" containerName="maas-api" containerID="cri-o://c4f8beccc897a2f4cdc99a1dcc61370f6c3f95c5fe50fe0661fe42bd88393cbe" gracePeriod=30
Sep 16 16:57:15.272014 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:15.271934    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"
Sep 16 16:57:15.296908 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:15.296863    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b9e893-5e8e-4aab-898b-d9145db247ec-metrics-certs\") pod \"78b9e893-5e8e-4aab-898b-d9145db247ec\" (UID: \"78b9e893-5e8e-4aab-898b-d9145db247ec\") "
Sep 16 16:57:15.297131 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:15.296917    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78b9e893-5e8e-4aab-898b-d9145db247ec-maas-api-tls\") pod \"78b9e893-5e8e-4aab-898b-d9145db247ec\" (UID: \"78b9e893-5e8e-4aab-898b-d9145db247ec\") "
Sep 16 16:57:15.297131 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:15.296989    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s24sk\" (UniqueName: \"kubernetes.io/projected/78b9e893-5e8e-4aab-898b-d9145db247ec-kube-api-access-s24sk\") pod \"78b9e893-5e8e-4aab-898b-d9145db247ec\" (UID: \"78b9e893-5e8e-4aab-898b-d9145db247ec\") "
Sep 16 16:57:15.297131 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:15.297031    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b9e893-5e8e-4aab-898b-d9145db247ec-openshift-service-ca\") pod \"78b9e893-5e8e-4aab-898b-d9145db247ec\" (UID: \"78b9e893-5e8e-4aab-898b-d9145db247ec\") "
Sep 16 16:57:15.298172 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:15.298140    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b9e893-5e8e-4aab-898b-d9145db247ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "78b9e893-5e8e-4aab-898b-d9145db247ec" (UID: "78b9e893-5e8e-4aab-898b-d9145db247ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:57:15.299986 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:15.299954    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78b9e893-5e8e-4aab-898b-d9145db247ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "78b9e893-5e8e-4aab-898b-d9145db247ec" (UID: "78b9e893-5e8e-4aab-898b-d9145db247ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:57:15.300412 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:15.300386    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78b9e893-5e8e-4aab-898b-d9145db247ec-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "78b9e893-5e8e-4aab-898b-d9145db247ec" (UID: "78b9e893-5e8e-4aab-898b-d9145db247ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:57:15.300503 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:15.300436    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b9e893-5e8e-4aab-898b-d9145db247ec-kube-api-access-s24sk" (OuterVolumeSpecName: "kube-api-access-s24sk") pod "78b9e893-5e8e-4aab-898b-d9145db247ec" (UID: "78b9e893-5e8e-4aab-898b-d9145db247ec"). InnerVolumeSpecName "kube-api-access-s24sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:57:15.397804 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:15.397765    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78b9e893-5e8e-4aab-898b-d9145db247ec-metrics-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:15.397804 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:15.397798    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78b9e893-5e8e-4aab-898b-d9145db247ec-maas-api-tls\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:15.397804 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:15.397809    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s24sk\" (UniqueName: \"kubernetes.io/projected/78b9e893-5e8e-4aab-898b-d9145db247ec-kube-api-access-s24sk\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:15.398045 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:15.397818    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78b9e893-5e8e-4aab-898b-d9145db247ec-openshift-service-ca\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:16.052190 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:16.052148    2556 generic.go:356] "Generic (PLEG): container finished" podID="78b9e893-5e8e-4aab-898b-d9145db247ec" containerID="c4f8beccc897a2f4cdc99a1dcc61370f6c3f95c5fe50fe0661fe42bd88393cbe" exitCode=0
Sep 16 16:57:16.052785 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:16.052205    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"
Sep 16 16:57:16.052785 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:16.052218    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv" event={"ID":"78b9e893-5e8e-4aab-898b-d9145db247ec","Type":"ContainerDied","Data":"c4f8beccc897a2f4cdc99a1dcc61370f6c3f95c5fe50fe0661fe42bd88393cbe"}
Sep 16 16:57:16.052785 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:16.052266    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv" event={"ID":"78b9e893-5e8e-4aab-898b-d9145db247ec","Type":"ContainerDied","Data":"ad9e3012dce7373b89b64514c780a37f13643bdce001920e15b53d737a1d8492"}
Sep 16 16:57:16.052785 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:16.052291    2556 scope.go:117] "RemoveContainer" containerID="c4f8beccc897a2f4cdc99a1dcc61370f6c3f95c5fe50fe0661fe42bd88393cbe"
Sep 16 16:57:16.061770 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:16.061753    2556 scope.go:117] "RemoveContainer" containerID="c4f8beccc897a2f4cdc99a1dcc61370f6c3f95c5fe50fe0661fe42bd88393cbe"
Sep 16 16:57:16.062067 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:57:16.062048    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f8beccc897a2f4cdc99a1dcc61370f6c3f95c5fe50fe0661fe42bd88393cbe\": container with ID starting with c4f8beccc897a2f4cdc99a1dcc61370f6c3f95c5fe50fe0661fe42bd88393cbe not found: ID does not exist" containerID="c4f8beccc897a2f4cdc99a1dcc61370f6c3f95c5fe50fe0661fe42bd88393cbe"
Sep 16 16:57:16.062127 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:16.062075    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f8beccc897a2f4cdc99a1dcc61370f6c3f95c5fe50fe0661fe42bd88393cbe"} err="failed to get container status \"c4f8beccc897a2f4cdc99a1dcc61370f6c3f95c5fe50fe0661fe42bd88393cbe\": rpc error: code = NotFound desc = could not find container \"c4f8beccc897a2f4cdc99a1dcc61370f6c3f95c5fe50fe0661fe42bd88393cbe\": container with ID starting with c4f8beccc897a2f4cdc99a1dcc61370f6c3f95c5fe50fe0661fe42bd88393cbe not found: ID does not exist"
Sep 16 16:57:16.074288 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:16.074267    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"]
Sep 16 16:57:16.081182 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:16.081158    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f3a3973e-75b6d948cc-54vfv"]
Sep 16 16:57:16.849771 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:16.849724    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78b9e893-5e8e-4aab-898b-d9145db247ec" path="/var/lib/kubelet/pods/78b9e893-5e8e-4aab-898b-d9145db247ec/volumes"
Sep 16 16:57:20.923402 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:20.923366    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"]
Sep 16 16:57:20.923801 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:20.923570    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b" podUID="cd879f74-608b-4626-a084-6af9bb94f68d" containerName="maas-api" containerID="cri-o://ad22e3cfd9777c7ef7dbcb9e49402b72b9cdb346c4ab81cdf937d0903093e495" gracePeriod=30
Sep 16 16:57:21.051629 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.051529    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz"]
Sep 16 16:57:21.051834 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.051780    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz" podUID="dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a" containerName="payload-pre-processing" containerID="cri-o://c021c05d067627186f90a66f959b39d3e1f89a612dc318ab40ff82e346693e13" gracePeriod=30
Sep 16 16:57:21.080756 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.080727    2556 generic.go:356] "Generic (PLEG): container finished" podID="cd879f74-608b-4626-a084-6af9bb94f68d" containerID="ad22e3cfd9777c7ef7dbcb9e49402b72b9cdb346c4ab81cdf937d0903093e495" exitCode=0
Sep 16 16:57:21.080868 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.080758    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b" event={"ID":"cd879f74-608b-4626-a084-6af9bb94f68d","Type":"ContainerDied","Data":"ad22e3cfd9777c7ef7dbcb9e49402b72b9cdb346c4ab81cdf937d0903093e495"}
Sep 16 16:57:21.197482 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.197458    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"
Sep 16 16:57:21.248782 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.248749    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjhf4\" (UniqueName: \"kubernetes.io/projected/cd879f74-608b-4626-a084-6af9bb94f68d-kube-api-access-sjhf4\") pod \"cd879f74-608b-4626-a084-6af9bb94f68d\" (UID: \"cd879f74-608b-4626-a084-6af9bb94f68d\") "
Sep 16 16:57:21.248949 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.248792    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd879f74-608b-4626-a084-6af9bb94f68d-maas-api-tls\") pod \"cd879f74-608b-4626-a084-6af9bb94f68d\" (UID: \"cd879f74-608b-4626-a084-6af9bb94f68d\") "
Sep 16 16:57:21.248949 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.248839    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd879f74-608b-4626-a084-6af9bb94f68d-metrics-certs\") pod \"cd879f74-608b-4626-a084-6af9bb94f68d\" (UID: \"cd879f74-608b-4626-a084-6af9bb94f68d\") "
Sep 16 16:57:21.248949 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.248895    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd879f74-608b-4626-a084-6af9bb94f68d-openshift-service-ca\") pod \"cd879f74-608b-4626-a084-6af9bb94f68d\" (UID: \"cd879f74-608b-4626-a084-6af9bb94f68d\") "
Sep 16 16:57:21.249543 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.249513    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd879f74-608b-4626-a084-6af9bb94f68d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cd879f74-608b-4626-a084-6af9bb94f68d" (UID: "cd879f74-608b-4626-a084-6af9bb94f68d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:57:21.251444 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.251324    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd879f74-608b-4626-a084-6af9bb94f68d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cd879f74-608b-4626-a084-6af9bb94f68d" (UID: "cd879f74-608b-4626-a084-6af9bb94f68d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:57:21.251613 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.251591    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd879f74-608b-4626-a084-6af9bb94f68d-kube-api-access-sjhf4" (OuterVolumeSpecName: "kube-api-access-sjhf4") pod "cd879f74-608b-4626-a084-6af9bb94f68d" (UID: "cd879f74-608b-4626-a084-6af9bb94f68d"). InnerVolumeSpecName "kube-api-access-sjhf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:57:21.252225 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.252205    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd879f74-608b-4626-a084-6af9bb94f68d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cd879f74-608b-4626-a084-6af9bb94f68d" (UID: "cd879f74-608b-4626-a084-6af9bb94f68d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:57:21.301693 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.301671    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz"
Sep 16 16:57:21.350451 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.350422    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a-plugins-config-volume\") pod \"dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a\" (UID: \"dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a\") "
Sep 16 16:57:21.350594 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.350531    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49jmq\" (UniqueName: \"kubernetes.io/projected/dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a-kube-api-access-49jmq\") pod \"dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a\" (UID: \"dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a\") "
Sep 16 16:57:21.350818 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.350798    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd879f74-608b-4626-a084-6af9bb94f68d-metrics-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:21.350897 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.350822    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd879f74-608b-4626-a084-6af9bb94f68d-openshift-service-ca\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:21.350897 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.350837    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjhf4\" (UniqueName: \"kubernetes.io/projected/cd879f74-608b-4626-a084-6af9bb94f68d-kube-api-access-sjhf4\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:21.350897 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.350837    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a" (UID: "dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:57:21.350897 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.350851    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd879f74-608b-4626-a084-6af9bb94f68d-maas-api-tls\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:21.352552 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.352529    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a-kube-api-access-49jmq" (OuterVolumeSpecName: "kube-api-access-49jmq") pod "dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a" (UID: "dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a"). InnerVolumeSpecName "kube-api-access-49jmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:57:21.451741 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.451709    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a-plugins-config-volume\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:21.451741 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:21.451736    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49jmq\" (UniqueName: \"kubernetes.io/projected/dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a-kube-api-access-49jmq\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:22.087170 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:22.087136    2556 generic.go:356] "Generic (PLEG): container finished" podID="dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a" containerID="c021c05d067627186f90a66f959b39d3e1f89a612dc318ab40ff82e346693e13" exitCode=0
Sep 16 16:57:22.087597 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:22.087187    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz"
Sep 16 16:57:22.087597 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:22.087222    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz" event={"ID":"dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a","Type":"ContainerDied","Data":"c021c05d067627186f90a66f959b39d3e1f89a612dc318ab40ff82e346693e13"}
Sep 16 16:57:22.087597 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:22.087302    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz" event={"ID":"dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a","Type":"ContainerDied","Data":"374fea5a7c007598c9ebdef0a6e272435d6a4165354bf23e96e7d9ddbdf3cff5"}
Sep 16 16:57:22.087597 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:22.087327    2556 scope.go:117] "RemoveContainer" containerID="c021c05d067627186f90a66f959b39d3e1f89a612dc318ab40ff82e346693e13"
Sep 16 16:57:22.088752 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:22.088586    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"
Sep 16 16:57:22.088752 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:22.088613    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b" event={"ID":"cd879f74-608b-4626-a084-6af9bb94f68d","Type":"ContainerDied","Data":"ba27bd5fca870487dc61c2bd5019cc8f482089f6664647edca082889ae77bc1e"}
Sep 16 16:57:22.096215 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:22.096141    2556 scope.go:117] "RemoveContainer" containerID="c021c05d067627186f90a66f959b39d3e1f89a612dc318ab40ff82e346693e13"
Sep 16 16:57:22.096448 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:57:22.096425    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c021c05d067627186f90a66f959b39d3e1f89a612dc318ab40ff82e346693e13\": container with ID starting with c021c05d067627186f90a66f959b39d3e1f89a612dc318ab40ff82e346693e13 not found: ID does not exist" containerID="c021c05d067627186f90a66f959b39d3e1f89a612dc318ab40ff82e346693e13"
Sep 16 16:57:22.096539 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:22.096458    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c021c05d067627186f90a66f959b39d3e1f89a612dc318ab40ff82e346693e13"} err="failed to get container status \"c021c05d067627186f90a66f959b39d3e1f89a612dc318ab40ff82e346693e13\": rpc error: code = NotFound desc = could not find container \"c021c05d067627186f90a66f959b39d3e1f89a612dc318ab40ff82e346693e13\": container with ID starting with c021c05d067627186f90a66f959b39d3e1f89a612dc318ab40ff82e346693e13 not found: ID does not exist"
Sep 16 16:57:22.096539 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:22.096481    2556 scope.go:117] "RemoveContainer" containerID="ad22e3cfd9777c7ef7dbcb9e49402b72b9cdb346c4ab81cdf937d0903093e495"
Sep 16 16:57:22.117580 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:22.117558    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"]
Sep 16 16:57:22.123104 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:22.123080    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e2ea3e-5cb9bbbdf5-zck8b"]
Sep 16 16:57:22.134522 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:22.134498    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz"]
Sep 16 16:57:22.141341 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:22.141321    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e2ea3e-6df5d54d54-qdqlz"]
Sep 16 16:57:22.851832 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:22.851799    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd879f74-608b-4626-a084-6af9bb94f68d" path="/var/lib/kubelet/pods/cd879f74-608b-4626-a084-6af9bb94f68d/volumes"
Sep 16 16:57:22.852193 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:22.852177    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a" path="/var/lib/kubelet/pods/dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a/volumes"
Sep 16 16:57:25.150361 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:25.150322    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x" podUID="3c72c686-7770-4d1a-a17a-de1883fb52dd" containerName="main" containerID="cri-o://126e553e68cdb2dfb8884b13df53fde611d010fd423bba9336d92b65c533ac91" gracePeriod=45
Sep 16 16:57:25.400773 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:25.400718    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:57:25.485182 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:25.485144    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hllvl\" (UniqueName: \"kubernetes.io/projected/3c72c686-7770-4d1a-a17a-de1883fb52dd-kube-api-access-hllvl\") pod \"3c72c686-7770-4d1a-a17a-de1883fb52dd\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") "
Sep 16 16:57:25.485343 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:25.485229    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-model-cache\") pod \"3c72c686-7770-4d1a-a17a-de1883fb52dd\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") "
Sep 16 16:57:25.485343 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:25.485250    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-home\") pod \"3c72c686-7770-4d1a-a17a-de1883fb52dd\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") "
Sep 16 16:57:25.485343 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:25.485268    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-tmp-dir\") pod \"3c72c686-7770-4d1a-a17a-de1883fb52dd\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") "
Sep 16 16:57:25.485343 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:25.485283    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-dshm\") pod \"3c72c686-7770-4d1a-a17a-de1883fb52dd\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") "
Sep 16 16:57:25.485343 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:25.485312    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c72c686-7770-4d1a-a17a-de1883fb52dd-tls-certs\") pod \"3c72c686-7770-4d1a-a17a-de1883fb52dd\" (UID: \"3c72c686-7770-4d1a-a17a-de1883fb52dd\") "
Sep 16 16:57:25.485625 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:25.485529    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c72c686-7770-4d1a-a17a-de1883fb52dd" (UID: "3c72c686-7770-4d1a-a17a-de1883fb52dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:57:25.485625 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:25.485537    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-home" (OuterVolumeSpecName: "home") pod "3c72c686-7770-4d1a-a17a-de1883fb52dd" (UID: "3c72c686-7770-4d1a-a17a-de1883fb52dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:57:25.485625 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:25.485555    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c72c686-7770-4d1a-a17a-de1883fb52dd" (UID: "3c72c686-7770-4d1a-a17a-de1883fb52dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:57:25.487399 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:25.487373    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c72c686-7770-4d1a-a17a-de1883fb52dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c72c686-7770-4d1a-a17a-de1883fb52dd" (UID: "3c72c686-7770-4d1a-a17a-de1883fb52dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:57:25.487399 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:25.487391    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-dshm" (OuterVolumeSpecName: "dshm") pod "3c72c686-7770-4d1a-a17a-de1883fb52dd" (UID: "3c72c686-7770-4d1a-a17a-de1883fb52dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 16:57:25.487570 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:25.487430    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c72c686-7770-4d1a-a17a-de1883fb52dd-kube-api-access-hllvl" (OuterVolumeSpecName: "kube-api-access-hllvl") pod "3c72c686-7770-4d1a-a17a-de1883fb52dd" (UID: "3c72c686-7770-4d1a-a17a-de1883fb52dd"). InnerVolumeSpecName "kube-api-access-hllvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:57:25.586097 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:25.586066    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-model-cache\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:25.586097 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:25.586093    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-home\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:25.586097 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:25.586102    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-tmp-dir\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:25.586325 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:25.586110    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c72c686-7770-4d1a-a17a-de1883fb52dd-dshm\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:25.586325 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:25.586117    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c72c686-7770-4d1a-a17a-de1883fb52dd-tls-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:25.586325 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:25.586126    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hllvl\" (UniqueName: \"kubernetes.io/projected/3c72c686-7770-4d1a-a17a-de1883fb52dd-kube-api-access-hllvl\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:26.110810 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:26.110770    2556 generic.go:356] "Generic (PLEG): container finished" podID="3c72c686-7770-4d1a-a17a-de1883fb52dd" containerID="126e553e68cdb2dfb8884b13df53fde611d010fd423bba9336d92b65c533ac91" exitCode=0
Sep 16 16:57:26.111020 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:26.110815    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"
Sep 16 16:57:26.111020 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:26.110850    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x" event={"ID":"3c72c686-7770-4d1a-a17a-de1883fb52dd","Type":"ContainerDied","Data":"126e553e68cdb2dfb8884b13df53fde611d010fd423bba9336d92b65c533ac91"}
Sep 16 16:57:26.111020 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:26.110901    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x" event={"ID":"3c72c686-7770-4d1a-a17a-de1883fb52dd","Type":"ContainerDied","Data":"e26ecf583d2e8c2279db0780a938a0c986fb07f2dce1bb4f0464f623f44eb54e"}
Sep 16 16:57:26.111020 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:26.110918    2556 scope.go:117] "RemoveContainer" containerID="126e553e68cdb2dfb8884b13df53fde611d010fd423bba9336d92b65c533ac91"
Sep 16 16:57:26.119726 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:26.119704    2556 scope.go:117] "RemoveContainer" containerID="126e553e68cdb2dfb8884b13df53fde611d010fd423bba9336d92b65c533ac91"
Sep 16 16:57:26.120069 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:57:26.120051    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"126e553e68cdb2dfb8884b13df53fde611d010fd423bba9336d92b65c533ac91\": container with ID starting with 126e553e68cdb2dfb8884b13df53fde611d010fd423bba9336d92b65c533ac91 not found: ID does not exist" containerID="126e553e68cdb2dfb8884b13df53fde611d010fd423bba9336d92b65c533ac91"
Sep 16 16:57:26.120146 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:26.120081    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126e553e68cdb2dfb8884b13df53fde611d010fd423bba9336d92b65c533ac91"} err="failed to get container status \"126e553e68cdb2dfb8884b13df53fde611d010fd423bba9336d92b65c533ac91\": rpc error: code = NotFound desc = could not find container \"126e553e68cdb2dfb8884b13df53fde611d010fd423bba9336d92b65c533ac91\": container with ID starting with 126e553e68cdb2dfb8884b13df53fde611d010fd423bba9336d92b65c533ac91 not found: ID does not exist"
Sep 16 16:57:26.135559 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:26.135533    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"]
Sep 16 16:57:26.140704 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:26.140684    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-ea8f09/test-model-ea8f09-kserve-6569776f95-nl97x"]
Sep 16 16:57:26.849078 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:26.849045    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c72c686-7770-4d1a-a17a-de1883fb52dd" path="/var/lib/kubelet/pods/3c72c686-7770-4d1a-a17a-de1883fb52dd/volumes"
Sep 16 16:57:38.391869 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.391795    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"]
Sep 16 16:57:38.393899 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.392332    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c72c686-7770-4d1a-a17a-de1883fb52dd" containerName="main"
Sep 16 16:57:38.393899 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.392352    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c72c686-7770-4d1a-a17a-de1883fb52dd" containerName="main"
Sep 16 16:57:38.393899 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.392370    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd879f74-608b-4626-a084-6af9bb94f68d" containerName="maas-api"
Sep 16 16:57:38.393899 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.392379    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd879f74-608b-4626-a084-6af9bb94f68d" containerName="maas-api"
Sep 16 16:57:38.393899 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.392404    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78b9e893-5e8e-4aab-898b-d9145db247ec" containerName="maas-api"
Sep 16 16:57:38.393899 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.392413    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="78b9e893-5e8e-4aab-898b-d9145db247ec" containerName="maas-api"
Sep 16 16:57:38.393899 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.392434    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a" containerName="payload-pre-processing"
Sep 16 16:57:38.393899 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.392443    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a" containerName="payload-pre-processing"
Sep 16 16:57:38.393899 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.392529    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="78b9e893-5e8e-4aab-898b-d9145db247ec" containerName="maas-api"
Sep 16 16:57:38.393899 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.392542    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd879f74-608b-4626-a084-6af9bb94f68d" containerName="maas-api"
Sep 16 16:57:38.393899 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.392554    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfc36aba-a11d-4ed2-a5c9-9d875af8cb4a" containerName="payload-pre-processing"
Sep 16 16:57:38.393899 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.392566    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c72c686-7770-4d1a-a17a-de1883fb52dd" containerName="main"
Sep 16 16:57:38.395640 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.395624    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.400334 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.400313    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-f315c3-openshift-default-dockercfg-mjzcq\""
Sep 16 16:57:38.413135 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.413107    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"]
Sep 16 16:57:38.491672 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.491640    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-istio-envoy\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.491848 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.491676    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-workload-socket\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.491848 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.491709    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-workload-certs\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.491848 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.491725    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f20909e3-9f17-4875-a2cd-2246087de774-istio-podinfo\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.491848 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.491782    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f20909e3-9f17-4875-a2cd-2246087de774-istiod-ca-cert\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.491848 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.491816    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-istio-data\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.491848 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.491839    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-credential-socket\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.492145 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.491867    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z996g\" (UniqueName: \"kubernetes.io/projected/f20909e3-9f17-4875-a2cd-2246087de774-kube-api-access-z996g\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.492145 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.491977    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f20909e3-9f17-4875-a2cd-2246087de774-istio-token\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.592719 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.592680    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-workload-socket\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.592889 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.592726    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-workload-certs\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.592889 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.592743    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f20909e3-9f17-4875-a2cd-2246087de774-istio-podinfo\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.592889 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.592764    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f20909e3-9f17-4875-a2cd-2246087de774-istiod-ca-cert\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.592889 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.592784    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-istio-data\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.592889 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.592812    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-credential-socket\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.592889 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.592836    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z996g\" (UniqueName: \"kubernetes.io/projected/f20909e3-9f17-4875-a2cd-2246087de774-kube-api-access-z996g\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.592889 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.592863    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f20909e3-9f17-4875-a2cd-2246087de774-istio-token\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.593337 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.592922    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-istio-envoy\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.593337 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.593205    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-workload-socket\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.593337 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.593258    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-istio-data\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.593494 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.593437    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-workload-certs\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.593494 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.593457    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-credential-socket\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.593494 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.593486    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f20909e3-9f17-4875-a2cd-2246087de774-istiod-ca-cert\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.595419 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.595392    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-istio-envoy\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.595547 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.595531    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f20909e3-9f17-4875-a2cd-2246087de774-istio-podinfo\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.601280 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.601260    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z996g\" (UniqueName: \"kubernetes.io/projected/f20909e3-9f17-4875-a2cd-2246087de774-kube-api-access-z996g\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.601457 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.601437    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f20909e3-9f17-4875-a2cd-2246087de774-istio-token\") pod \"e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") " pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.705351 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.705327    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:38.981868 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.981811    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"]
Sep 16 16:57:38.984506 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:57:38.984474    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20909e3_9f17_4875_a2cd_2246087de774.slice/crio-06fbf1e98cc2b46ad2dca7faecaae4e0fc653d704d26576f967da2cb800e3e6b WatchSource:0}: Error finding container 06fbf1e98cc2b46ad2dca7faecaae4e0fc653d704d26576f967da2cb800e3e6b: Status 404 returned error can't find the container with id 06fbf1e98cc2b46ad2dca7faecaae4e0fc653d704d26576f967da2cb800e3e6b
Sep 16 16:57:38.986702 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.986672    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 16:57:38.986811 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.986741    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 16:57:38.986811 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:38.986767    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 16:57:39.181212 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:39.181172    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn" event={"ID":"f20909e3-9f17-4875-a2cd-2246087de774","Type":"ContainerStarted","Data":"1b769886c4a5bada52d32c39dd4529115036fd7fa980818673a5c2e906efe282"}
Sep 16 16:57:39.181385 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:39.181234    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn" event={"ID":"f20909e3-9f17-4875-a2cd-2246087de774","Type":"ContainerStarted","Data":"06fbf1e98cc2b46ad2dca7faecaae4e0fc653d704d26576f967da2cb800e3e6b"}
Sep 16 16:57:39.207348 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:39.207288    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn" podStartSLOduration=1.207266956 podStartE2EDuration="1.207266956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:57:39.201849782 +0000 UTC m=+1258.963811243" watchObservedRunningTime="2026-09-16 16:57:39.207266956 +0000 UTC m=+1258.969228403"
Sep 16 16:57:39.705830 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:39.705801    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:40.711102 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:40.711067    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:41.190756 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.190717    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:41.191667 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.191646    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:41.462741 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.462640    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"]
Sep 16 16:57:41.462919 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.462890    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6" podUID="9b0eac9c-d48e-494b-b43a-839b9923dad4" containerName="maas-api" containerID="cri-o://87b17000cdd74104ae3e330a8e952b7cc387c62192b6dc001b2987970a1f7671" gracePeriod=30
Sep 16 16:57:41.519489 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.519459    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc"]
Sep 16 16:57:41.519755 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.519704    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc" podUID="2ec165c6-71a1-4b89-9fcf-2aa247a5f475" containerName="payload-processing" containerID="cri-o://d9f838b76222aa1353a09d4045ca1d6e3937fd6f850f198d16a8cc3adaa4f44e" gracePeriod=30
Sep 16 16:57:41.620359 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.620319    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc" podUID="2ec165c6-71a1-4b89-9fcf-2aa247a5f475" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.44:9004: connect: connection refused"
Sep 16 16:57:41.765796 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.765773    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"
Sep 16 16:57:41.796388 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.796363    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc"
Sep 16 16:57:41.821430 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.821408    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b0eac9c-d48e-494b-b43a-839b9923dad4-maas-api-tls\") pod \"9b0eac9c-d48e-494b-b43a-839b9923dad4\" (UID: \"9b0eac9c-d48e-494b-b43a-839b9923dad4\") "
Sep 16 16:57:41.821572 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.821491    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn5xh\" (UniqueName: \"kubernetes.io/projected/9b0eac9c-d48e-494b-b43a-839b9923dad4-kube-api-access-gn5xh\") pod \"9b0eac9c-d48e-494b-b43a-839b9923dad4\" (UID: \"9b0eac9c-d48e-494b-b43a-839b9923dad4\") "
Sep 16 16:57:41.821572 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.821513    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0eac9c-d48e-494b-b43a-839b9923dad4-openshift-service-ca\") pod \"9b0eac9c-d48e-494b-b43a-839b9923dad4\" (UID: \"9b0eac9c-d48e-494b-b43a-839b9923dad4\") "
Sep 16 16:57:41.821572 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.821540    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0eac9c-d48e-494b-b43a-839b9923dad4-metrics-certs\") pod \"9b0eac9c-d48e-494b-b43a-839b9923dad4\" (UID: \"9b0eac9c-d48e-494b-b43a-839b9923dad4\") "
Sep 16 16:57:41.822362 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.822204    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0eac9c-d48e-494b-b43a-839b9923dad4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9b0eac9c-d48e-494b-b43a-839b9923dad4" (UID: "9b0eac9c-d48e-494b-b43a-839b9923dad4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:57:41.824106 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.824078    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0eac9c-d48e-494b-b43a-839b9923dad4-kube-api-access-gn5xh" (OuterVolumeSpecName: "kube-api-access-gn5xh") pod "9b0eac9c-d48e-494b-b43a-839b9923dad4" (UID: "9b0eac9c-d48e-494b-b43a-839b9923dad4"). InnerVolumeSpecName "kube-api-access-gn5xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:57:41.824199 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.824089    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0eac9c-d48e-494b-b43a-839b9923dad4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9b0eac9c-d48e-494b-b43a-839b9923dad4" (UID: "9b0eac9c-d48e-494b-b43a-839b9923dad4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:57:41.824327 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.824311    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b0eac9c-d48e-494b-b43a-839b9923dad4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9b0eac9c-d48e-494b-b43a-839b9923dad4" (UID: "9b0eac9c-d48e-494b-b43a-839b9923dad4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 16:57:41.922386 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.922357    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vpqn\" (UniqueName: \"kubernetes.io/projected/2ec165c6-71a1-4b89-9fcf-2aa247a5f475-kube-api-access-5vpqn\") pod \"2ec165c6-71a1-4b89-9fcf-2aa247a5f475\" (UID: \"2ec165c6-71a1-4b89-9fcf-2aa247a5f475\") "
Sep 16 16:57:41.922386 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.922393    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec165c6-71a1-4b89-9fcf-2aa247a5f475-plugins-config-volume\") pod \"2ec165c6-71a1-4b89-9fcf-2aa247a5f475\" (UID: \"2ec165c6-71a1-4b89-9fcf-2aa247a5f475\") "
Sep 16 16:57:41.922816 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.922790    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b0eac9c-d48e-494b-b43a-839b9923dad4-maas-api-tls\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:41.922977 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.922825    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ec165c6-71a1-4b89-9fcf-2aa247a5f475-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2ec165c6-71a1-4b89-9fcf-2aa247a5f475" (UID: "2ec165c6-71a1-4b89-9fcf-2aa247a5f475"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 16:57:41.922977 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.922929    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn5xh\" (UniqueName: \"kubernetes.io/projected/9b0eac9c-d48e-494b-b43a-839b9923dad4-kube-api-access-gn5xh\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:41.922977 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.922970    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b0eac9c-d48e-494b-b43a-839b9923dad4-openshift-service-ca\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:41.923133 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.922985    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b0eac9c-d48e-494b-b43a-839b9923dad4-metrics-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:41.924609 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:41.924580    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec165c6-71a1-4b89-9fcf-2aa247a5f475-kube-api-access-5vpqn" (OuterVolumeSpecName: "kube-api-access-5vpqn") pod "2ec165c6-71a1-4b89-9fcf-2aa247a5f475" (UID: "2ec165c6-71a1-4b89-9fcf-2aa247a5f475"). InnerVolumeSpecName "kube-api-access-5vpqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 16:57:42.023689 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.023628    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vpqn\" (UniqueName: \"kubernetes.io/projected/2ec165c6-71a1-4b89-9fcf-2aa247a5f475-kube-api-access-5vpqn\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:42.023689 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.023652    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ec165c6-71a1-4b89-9fcf-2aa247a5f475-plugins-config-volume\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 16:57:42.196913 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.196877    2556 generic.go:356] "Generic (PLEG): container finished" podID="2ec165c6-71a1-4b89-9fcf-2aa247a5f475" containerID="d9f838b76222aa1353a09d4045ca1d6e3937fd6f850f198d16a8cc3adaa4f44e" exitCode=0
Sep 16 16:57:42.197095 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.196930    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc"
Sep 16 16:57:42.197095 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.196982    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc" event={"ID":"2ec165c6-71a1-4b89-9fcf-2aa247a5f475","Type":"ContainerDied","Data":"d9f838b76222aa1353a09d4045ca1d6e3937fd6f850f198d16a8cc3adaa4f44e"}
Sep 16 16:57:42.197095 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.197055    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc" event={"ID":"2ec165c6-71a1-4b89-9fcf-2aa247a5f475","Type":"ContainerDied","Data":"b1235f05bd1cf92411c35496b27cc9bc5d05f0091e018baf6ced004872e1f167"}
Sep 16 16:57:42.197095 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.197077    2556 scope.go:117] "RemoveContainer" containerID="d9f838b76222aa1353a09d4045ca1d6e3937fd6f850f198d16a8cc3adaa4f44e"
Sep 16 16:57:42.198362 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.198340    2556 generic.go:356] "Generic (PLEG): container finished" podID="9b0eac9c-d48e-494b-b43a-839b9923dad4" containerID="87b17000cdd74104ae3e330a8e952b7cc387c62192b6dc001b2987970a1f7671" exitCode=0
Sep 16 16:57:42.198448 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.198396    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"
Sep 16 16:57:42.198448 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.198407    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6" event={"ID":"9b0eac9c-d48e-494b-b43a-839b9923dad4","Type":"ContainerDied","Data":"87b17000cdd74104ae3e330a8e952b7cc387c62192b6dc001b2987970a1f7671"}
Sep 16 16:57:42.198448 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.198447    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6" event={"ID":"9b0eac9c-d48e-494b-b43a-839b9923dad4","Type":"ContainerDied","Data":"40c37c359c10737a1be63437a1632d3b1660ebc9a8decb71da146e47b3e678f5"}
Sep 16 16:57:42.205573 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.205558    2556 scope.go:117] "RemoveContainer" containerID="d9f838b76222aa1353a09d4045ca1d6e3937fd6f850f198d16a8cc3adaa4f44e"
Sep 16 16:57:42.205822 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:57:42.205801    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9f838b76222aa1353a09d4045ca1d6e3937fd6f850f198d16a8cc3adaa4f44e\": container with ID starting with d9f838b76222aa1353a09d4045ca1d6e3937fd6f850f198d16a8cc3adaa4f44e not found: ID does not exist" containerID="d9f838b76222aa1353a09d4045ca1d6e3937fd6f850f198d16a8cc3adaa4f44e"
Sep 16 16:57:42.205917 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.205827    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9f838b76222aa1353a09d4045ca1d6e3937fd6f850f198d16a8cc3adaa4f44e"} err="failed to get container status \"d9f838b76222aa1353a09d4045ca1d6e3937fd6f850f198d16a8cc3adaa4f44e\": rpc error: code = NotFound desc = could not find container \"d9f838b76222aa1353a09d4045ca1d6e3937fd6f850f198d16a8cc3adaa4f44e\": container with ID starting with d9f838b76222aa1353a09d4045ca1d6e3937fd6f850f198d16a8cc3adaa4f44e not found: ID does not exist"
Sep 16 16:57:42.205917 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.205847    2556 scope.go:117] "RemoveContainer" containerID="87b17000cdd74104ae3e330a8e952b7cc387c62192b6dc001b2987970a1f7671"
Sep 16 16:57:42.213764 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.213749    2556 scope.go:117] "RemoveContainer" containerID="87b17000cdd74104ae3e330a8e952b7cc387c62192b6dc001b2987970a1f7671"
Sep 16 16:57:42.213993 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:57:42.213972    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b17000cdd74104ae3e330a8e952b7cc387c62192b6dc001b2987970a1f7671\": container with ID starting with 87b17000cdd74104ae3e330a8e952b7cc387c62192b6dc001b2987970a1f7671 not found: ID does not exist" containerID="87b17000cdd74104ae3e330a8e952b7cc387c62192b6dc001b2987970a1f7671"
Sep 16 16:57:42.214070 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.213998    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b17000cdd74104ae3e330a8e952b7cc387c62192b6dc001b2987970a1f7671"} err="failed to get container status \"87b17000cdd74104ae3e330a8e952b7cc387c62192b6dc001b2987970a1f7671\": rpc error: code = NotFound desc = could not find container \"87b17000cdd74104ae3e330a8e952b7cc387c62192b6dc001b2987970a1f7671\": container with ID starting with 87b17000cdd74104ae3e330a8e952b7cc387c62192b6dc001b2987970a1f7671 not found: ID does not exist"
Sep 16 16:57:42.225431 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.225410    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"]
Sep 16 16:57:42.228086 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.228065    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ea8f09-7558578944-8s7z6"]
Sep 16 16:57:42.239547 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.239524    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc"]
Sep 16 16:57:42.241541 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.241522    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ea8f09-85854849bf-vm7gc"]
Sep 16 16:57:42.469777 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.469746    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"]
Sep 16 16:57:42.470177 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.470159    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ec165c6-71a1-4b89-9fcf-2aa247a5f475" containerName="payload-processing"
Sep 16 16:57:42.470281 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.470180    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ec165c6-71a1-4b89-9fcf-2aa247a5f475" containerName="payload-processing"
Sep 16 16:57:42.470376 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.470283    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b0eac9c-d48e-494b-b43a-839b9923dad4" containerName="maas-api"
Sep 16 16:57:42.470376 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.470295    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b0eac9c-d48e-494b-b43a-839b9923dad4" containerName="maas-api"
Sep 16 16:57:42.470991 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.470964    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ec165c6-71a1-4b89-9fcf-2aa247a5f475" containerName="payload-processing"
Sep 16 16:57:42.471099 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.471010    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b0eac9c-d48e-494b-b43a-839b9923dad4" containerName="maas-api"
Sep 16 16:57:42.477980 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.477957    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"
Sep 16 16:57:42.481616 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.481394    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-f315c3-serving-cert\""
Sep 16 16:57:42.488220 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.488193    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"]
Sep 16 16:57:42.526767 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.526740    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7543b8b9-2f41-484f-9e31-5f695467893c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2\" (UID: \"7543b8b9-2f41-484f-9e31-5f695467893c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"
Sep 16 16:57:42.526898 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.526792    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7543b8b9-2f41-484f-9e31-5f695467893c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2\" (UID: \"7543b8b9-2f41-484f-9e31-5f695467893c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"
Sep 16 16:57:42.526898 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.526854    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7543b8b9-2f41-484f-9e31-5f695467893c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2\" (UID: \"7543b8b9-2f41-484f-9e31-5f695467893c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"
Sep 16 16:57:42.526898 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.526892    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8kwf\" (UniqueName: \"kubernetes.io/projected/7543b8b9-2f41-484f-9e31-5f695467893c-kube-api-access-n8kwf\") pod \"maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2\" (UID: \"7543b8b9-2f41-484f-9e31-5f695467893c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"
Sep 16 16:57:42.627777 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.627741    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7543b8b9-2f41-484f-9e31-5f695467893c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2\" (UID: \"7543b8b9-2f41-484f-9e31-5f695467893c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"
Sep 16 16:57:42.627968 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.627800    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7543b8b9-2f41-484f-9e31-5f695467893c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2\" (UID: \"7543b8b9-2f41-484f-9e31-5f695467893c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"
Sep 16 16:57:42.627968 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.627842    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7543b8b9-2f41-484f-9e31-5f695467893c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2\" (UID: \"7543b8b9-2f41-484f-9e31-5f695467893c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"
Sep 16 16:57:42.627968 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.627870    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8kwf\" (UniqueName: \"kubernetes.io/projected/7543b8b9-2f41-484f-9e31-5f695467893c-kube-api-access-n8kwf\") pod \"maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2\" (UID: \"7543b8b9-2f41-484f-9e31-5f695467893c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"
Sep 16 16:57:42.628109 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:57:42.627989    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-serving-cert: secret "maas-api-e2e-shared-a-w2-f315c3-serving-cert" not found
Sep 16 16:57:42.628109 ip-10-0-140-120 kubenswrapper[2556]: E0916 16:57:42.628060    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7543b8b9-2f41-484f-9e31-5f695467893c-maas-api-tls podName:7543b8b9-2f41-484f-9e31-5f695467893c nodeName:}" failed. No retries permitted until 2026-09-16 16:57:43.128039335 +0000 UTC m=+1262.890000774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7543b8b9-2f41-484f-9e31-5f695467893c-maas-api-tls") pod "maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2" (UID: "7543b8b9-2f41-484f-9e31-5f695467893c") : secret "maas-api-e2e-shared-a-w2-f315c3-serving-cert" not found
Sep 16 16:57:42.628545 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.628519    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7543b8b9-2f41-484f-9e31-5f695467893c-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2\" (UID: \"7543b8b9-2f41-484f-9e31-5f695467893c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"
Sep 16 16:57:42.630219 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.630199    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7543b8b9-2f41-484f-9e31-5f695467893c-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2\" (UID: \"7543b8b9-2f41-484f-9e31-5f695467893c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"
Sep 16 16:57:42.645440 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.645414    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8kwf\" (UniqueName: \"kubernetes.io/projected/7543b8b9-2f41-484f-9e31-5f695467893c-kube-api-access-n8kwf\") pod \"maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2\" (UID: \"7543b8b9-2f41-484f-9e31-5f695467893c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"
Sep 16 16:57:42.848668 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.848588    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ec165c6-71a1-4b89-9fcf-2aa247a5f475" path="/var/lib/kubelet/pods/2ec165c6-71a1-4b89-9fcf-2aa247a5f475/volumes"
Sep 16 16:57:42.849045 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:42.848985    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b0eac9c-d48e-494b-b43a-839b9923dad4" path="/var/lib/kubelet/pods/9b0eac9c-d48e-494b-b43a-839b9923dad4/volumes"
Sep 16 16:57:43.131638 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:43.131557    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7543b8b9-2f41-484f-9e31-5f695467893c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2\" (UID: \"7543b8b9-2f41-484f-9e31-5f695467893c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"
Sep 16 16:57:43.133855 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:43.133832    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7543b8b9-2f41-484f-9e31-5f695467893c-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2\" (UID: \"7543b8b9-2f41-484f-9e31-5f695467893c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"
Sep 16 16:57:43.395313 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:43.395223    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"
Sep 16 16:57:43.665880 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:43.665775    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"]
Sep 16 16:57:43.669772 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:57:43.669683    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7543b8b9_2f41_484f_9e31_5f695467893c.slice/crio-9be0dd372b0996b7f7cacb620b50d20822d34bd349a10b12a4104f07472dfd77 WatchSource:0}: Error finding container 9be0dd372b0996b7f7cacb620b50d20822d34bd349a10b12a4104f07472dfd77: Status 404 returned error can't find the container with id 9be0dd372b0996b7f7cacb620b50d20822d34bd349a10b12a4104f07472dfd77
Sep 16 16:57:43.671750 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:43.671711    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 16:57:44.212265 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:44.212228    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2" event={"ID":"7543b8b9-2f41-484f-9e31-5f695467893c","Type":"ContainerStarted","Data":"248e465c292b99fbb52c51a6b76894e458a174d1c5591e6b425e0e6992237e31"}
Sep 16 16:57:44.212623 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:44.212296    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2" event={"ID":"7543b8b9-2f41-484f-9e31-5f695467893c","Type":"ContainerStarted","Data":"9be0dd372b0996b7f7cacb620b50d20822d34bd349a10b12a4104f07472dfd77"}
Sep 16 16:57:44.212623 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:44.212338    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"
Sep 16 16:57:44.232661 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:44.232615    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2" podStartSLOduration=1.7673806239999998 podStartE2EDuration="2.232598339s" totalImagesPullingTime="465.217715ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:57:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:57:44.230986946 +0000 UTC m=+1263.992948393" watchObservedRunningTime="2026-09-16 16:57:44.232598339 +0000 UTC m=+1263.994559835"
Sep 16 16:57:49.599780 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.599742    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"]
Sep 16 16:57:49.603696 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.603674    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.608913 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.608883    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-52fd8c-openshift-default-dockercfg-mmdkg\""
Sep 16 16:57:49.623338 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.623314    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"]
Sep 16 16:57:49.684926 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.684892    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-workload-certs\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.684926 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.684927    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576f4074-7fe9-42cb-810b-bb18acdbfa69-istiod-ca-cert\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.685172 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.685002    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-credential-socket\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.685172 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.685029    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-token\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.685172 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.685129    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-data\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.685172 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.685160    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j92xj\" (UniqueName: \"kubernetes.io/projected/576f4074-7fe9-42cb-810b-bb18acdbfa69-kube-api-access-j92xj\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.685334 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.685182    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-workload-socket\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.685334 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.685197    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-podinfo\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.685334 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.685219    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-envoy\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.786400 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.786361    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j92xj\" (UniqueName: \"kubernetes.io/projected/576f4074-7fe9-42cb-810b-bb18acdbfa69-kube-api-access-j92xj\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.786400 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.786402    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-workload-socket\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.786659 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.786421    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-podinfo\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.786659 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.786441    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-envoy\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.786659 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.786558    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-workload-certs\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.786659 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.786578    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576f4074-7fe9-42cb-810b-bb18acdbfa69-istiod-ca-cert\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.786659 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.786605    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-credential-socket\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.786659 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.786622    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-token\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.787004 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.786704    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-data\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.787004 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.786867    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-workload-socket\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.787004 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.786891    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-workload-certs\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.787180 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.787008    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-credential-socket\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.787180 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.787144    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-data\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.787300 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.787282    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576f4074-7fe9-42cb-810b-bb18acdbfa69-istiod-ca-cert\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.789084 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.789061    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-envoy\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.789195 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.789170    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-podinfo\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.794999 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.794974    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j92xj\" (UniqueName: \"kubernetes.io/projected/576f4074-7fe9-42cb-810b-bb18acdbfa69-kube-api-access-j92xj\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.797292 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.797272    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-token\") pod \"e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") " pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:49.914273 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:49.914169    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:50.213747 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:50.213716    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"]
Sep 16 16:57:50.216170 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:57:50.216146    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576f4074_7fe9_42cb_810b_bb18acdbfa69.slice/crio-81ada1ef4924b6d3d9672cfd146bfb7537f07ca2b3703a3a5e20096c708fae73 WatchSource:0}: Error finding container 81ada1ef4924b6d3d9672cfd146bfb7537f07ca2b3703a3a5e20096c708fae73: Status 404 returned error can't find the container with id 81ada1ef4924b6d3d9672cfd146bfb7537f07ca2b3703a3a5e20096c708fae73
Sep 16 16:57:50.218412 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:50.218380    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 16:57:50.218524 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:50.218461    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 16:57:50.218589 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:50.218528    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 16:57:50.222741 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:50.222720    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"
Sep 16 16:57:50.242539 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:50.242513    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r" event={"ID":"576f4074-7fe9-42cb-810b-bb18acdbfa69","Type":"ContainerStarted","Data":"81ada1ef4924b6d3d9672cfd146bfb7537f07ca2b3703a3a5e20096c708fae73"}
Sep 16 16:57:51.251112 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:51.251070    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r" event={"ID":"576f4074-7fe9-42cb-810b-bb18acdbfa69","Type":"ContainerStarted","Data":"bb92001a47da9b6413d668a98f3cd3db175359678daabc5f6458f0987412bb6b"}
Sep 16 16:57:51.275507 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:51.275317    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r" podStartSLOduration=2.275297752 podStartE2EDuration="2.275297752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:57:51.273856239 +0000 UTC m=+1271.035817686" watchObservedRunningTime="2026-09-16 16:57:51.275297752 +0000 UTC m=+1271.037259242"
Sep 16 16:57:51.914751 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:51.914717    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:51.919496 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:51.919471    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:52.259578 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:52.259545    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:52.266577 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:52.266553    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:57:56.411006 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:56.410389    2556 patch_prober.go:28] interesting pod/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": read tcp 10.134.0.2:55112->10.134.0.48:15021: read: connection reset by peer" start-of-body=
Sep 16 16:57:56.411006 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:56.410465    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn" podUID="f20909e3-9f17-4875-a2cd-2246087de774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": read tcp 10.134.0.2:55112->10.134.0.48:15021: read: connection reset by peer"
Sep 16 16:57:57.289832 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:57.289792    2556 generic.go:356] "Generic (PLEG): container finished" podID="f20909e3-9f17-4875-a2cd-2246087de774" containerID="1b769886c4a5bada52d32c39dd4529115036fd7fa980818673a5c2e906efe282" exitCode=0
Sep 16 16:57:57.290059 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:57.289838    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn" event={"ID":"f20909e3-9f17-4875-a2cd-2246087de774","Type":"ContainerDied","Data":"1b769886c4a5bada52d32c39dd4529115036fd7fa980818673a5c2e906efe282"}
Sep 16 16:57:57.290323 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:57.290304    2556 scope.go:117] "RemoveContainer" containerID="1b769886c4a5bada52d32c39dd4529115036fd7fa980818673a5c2e906efe282"
Sep 16 16:57:57.291353 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:57.291323    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 16:57:57.291432 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:57.291392    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 16:57:57.291432 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:57.291424    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 16:57:58.296519 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:58.296486    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn" event={"ID":"f20909e3-9f17-4875-a2cd-2246087de774","Type":"ContainerStarted","Data":"8425da95efb86ca8bea235e68e4007d30d090009ab56adc4ff23dcda56f15407"}
Sep 16 16:57:58.705706 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:58.705669    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:57:58.705913 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:58.705889    2556 patch_prober.go:28] interesting pod/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 16:57:58.705981 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:58.705963    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn" podUID="f20909e3-9f17-4875-a2cd-2246087de774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 16:57:59.706142 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:59.706101    2556 patch_prober.go:28] interesting pod/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 16:57:59.706507 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:57:59.706160    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn" podUID="f20909e3-9f17-4875-a2cd-2246087de774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 16:58:00.706178 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:00.706145    2556 patch_prober.go:28] interesting pod/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 16:58:00.706578 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:00.706202    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn" podUID="f20909e3-9f17-4875-a2cd-2246087de774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 16:58:01.706038 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:01.706004    2556 patch_prober.go:28] interesting pod/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 16:58:01.706203 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:01.706064    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn" podUID="f20909e3-9f17-4875-a2cd-2246087de774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 16:58:02.271877 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:02.271837    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"]
Sep 16 16:58:02.276206 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:02.276123    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"
Sep 16 16:58:02.278614 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:02.278591    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-2ce8bc-serving-cert\""
Sep 16 16:58:02.287549 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:02.287520    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"]
Sep 16 16:58:02.399322 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:02.399288    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81ac47a1-7620-4932-bc86-045295332f13-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw\" (UID: \"81ac47a1-7620-4932-bc86-045295332f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"
Sep 16 16:58:02.399322 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:02.399321    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81ac47a1-7620-4932-bc86-045295332f13-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw\" (UID: \"81ac47a1-7620-4932-bc86-045295332f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"
Sep 16 16:58:02.399526 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:02.399364    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trvcr\" (UniqueName: \"kubernetes.io/projected/81ac47a1-7620-4932-bc86-045295332f13-kube-api-access-trvcr\") pod \"maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw\" (UID: \"81ac47a1-7620-4932-bc86-045295332f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"
Sep 16 16:58:02.399526 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:02.399387    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81ac47a1-7620-4932-bc86-045295332f13-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw\" (UID: \"81ac47a1-7620-4932-bc86-045295332f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"
Sep 16 16:58:02.500278 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:02.500240    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81ac47a1-7620-4932-bc86-045295332f13-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw\" (UID: \"81ac47a1-7620-4932-bc86-045295332f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"
Sep 16 16:58:02.500278 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:02.500281    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81ac47a1-7620-4932-bc86-045295332f13-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw\" (UID: \"81ac47a1-7620-4932-bc86-045295332f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"
Sep 16 16:58:02.500517 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:02.500343    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trvcr\" (UniqueName: \"kubernetes.io/projected/81ac47a1-7620-4932-bc86-045295332f13-kube-api-access-trvcr\") pod \"maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw\" (UID: \"81ac47a1-7620-4932-bc86-045295332f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"
Sep 16 16:58:02.500517 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:02.500374    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81ac47a1-7620-4932-bc86-045295332f13-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw\" (UID: \"81ac47a1-7620-4932-bc86-045295332f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"
Sep 16 16:58:02.501108 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:02.501078    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81ac47a1-7620-4932-bc86-045295332f13-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw\" (UID: \"81ac47a1-7620-4932-bc86-045295332f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"
Sep 16 16:58:02.502739 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:02.502708    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81ac47a1-7620-4932-bc86-045295332f13-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw\" (UID: \"81ac47a1-7620-4932-bc86-045295332f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"
Sep 16 16:58:02.502839 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:02.502755    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81ac47a1-7620-4932-bc86-045295332f13-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw\" (UID: \"81ac47a1-7620-4932-bc86-045295332f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"
Sep 16 16:58:02.510294 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:02.510272    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trvcr\" (UniqueName: \"kubernetes.io/projected/81ac47a1-7620-4932-bc86-045295332f13-kube-api-access-trvcr\") pod \"maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw\" (UID: \"81ac47a1-7620-4932-bc86-045295332f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"
Sep 16 16:58:02.588634 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:02.588543    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"
Sep 16 16:58:02.706069 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:02.705933    2556 patch_prober.go:28] interesting pod/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 16 16:58:02.706069 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:02.706021    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn" podUID="f20909e3-9f17-4875-a2cd-2246087de774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 16 16:58:02.906737 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:02.906644    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"]
Sep 16 16:58:02.916833 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:58:02.916793    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ac47a1_7620_4932_bc86_045295332f13.slice/crio-7df09024ffa96f4539ac33cdb626ea96c582b22e8eb4189b9bca4a3a87dcbc12 WatchSource:0}: Error finding container 7df09024ffa96f4539ac33cdb626ea96c582b22e8eb4189b9bca4a3a87dcbc12: Status 404 returned error can't find the container with id 7df09024ffa96f4539ac33cdb626ea96c582b22e8eb4189b9bca4a3a87dcbc12
Sep 16 16:58:03.025880 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:03.025843    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh"]
Sep 16 16:58:03.033831 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:03.033789    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh"
Sep 16 16:58:03.036346 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:03.036315    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-2ce8bc-dockercfg-j82zk\""
Sep 16 16:58:03.036557 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:03.036315    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-2ce8bc\""
Sep 16 16:58:03.039877 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:03.039852    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh"]
Sep 16 16:58:03.209707 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:03.209658    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwv95\" (UniqueName: \"kubernetes.io/projected/1c06f327-248b-48cb-a75e-3a3e57f43162-kube-api-access-nwv95\") pod \"payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh\" (UID: \"1c06f327-248b-48cb-a75e-3a3e57f43162\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh"
Sep 16 16:58:03.209893 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:03.209748    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c06f327-248b-48cb-a75e-3a3e57f43162-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh\" (UID: \"1c06f327-248b-48cb-a75e-3a3e57f43162\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh"
Sep 16 16:58:03.310849 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:03.310809    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwv95\" (UniqueName: \"kubernetes.io/projected/1c06f327-248b-48cb-a75e-3a3e57f43162-kube-api-access-nwv95\") pod \"payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh\" (UID: \"1c06f327-248b-48cb-a75e-3a3e57f43162\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh"
Sep 16 16:58:03.311051 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:03.310862    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c06f327-248b-48cb-a75e-3a3e57f43162-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh\" (UID: \"1c06f327-248b-48cb-a75e-3a3e57f43162\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh"
Sep 16 16:58:03.311859 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:03.311835    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c06f327-248b-48cb-a75e-3a3e57f43162-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh\" (UID: \"1c06f327-248b-48cb-a75e-3a3e57f43162\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh"
Sep 16 16:58:03.320126 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:03.320097    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwv95\" (UniqueName: \"kubernetes.io/projected/1c06f327-248b-48cb-a75e-3a3e57f43162-kube-api-access-nwv95\") pod \"payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh\" (UID: \"1c06f327-248b-48cb-a75e-3a3e57f43162\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh"
Sep 16 16:58:03.321880 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:03.321849    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw" event={"ID":"81ac47a1-7620-4932-bc86-045295332f13","Type":"ContainerStarted","Data":"7df09024ffa96f4539ac33cdb626ea96c582b22e8eb4189b9bca4a3a87dcbc12"}
Sep 16 16:58:03.349219 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:03.349170    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh"
Sep 16 16:58:03.644713 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:03.644653    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh"]
Sep 16 16:58:03.710089 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:03.710059    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:58:03.710477 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:03.710457    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:58:03.711379 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:03.711360    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 16:58:04.330403 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:04.330364    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw" event={"ID":"81ac47a1-7620-4932-bc86-045295332f13","Type":"ContainerStarted","Data":"2ea5d3347646872b603e80919b8912adbf0e10625799511e6cca5fd077cf5fd9"}
Sep 16 16:58:04.330580 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:04.330429    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"
Sep 16 16:58:04.332058 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:04.332028    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh" event={"ID":"1c06f327-248b-48cb-a75e-3a3e57f43162","Type":"ContainerStarted","Data":"0242662710bc041b8ba8c4e1701bb3a9aed0b1753927f2f5d9d88eb53f560661"}
Sep 16 16:58:04.332160 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:04.332074    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh" event={"ID":"1c06f327-248b-48cb-a75e-3a3e57f43162","Type":"ContainerStarted","Data":"870cd38307e9fec5434973f89b53163501d04d228e5bc35835104a480d0806c3"}
Sep 16 16:58:04.332267 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:04.332248    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh"
Sep 16 16:58:04.347689 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:04.347629    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw" podStartSLOduration=1.866996817 podStartE2EDuration="2.347611123s" totalImagesPullingTime="480.614306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:58:04.346476214 +0000 UTC m=+1284.108437661" watchObservedRunningTime="2026-09-16 16:58:04.347611123 +0000 UTC m=+1284.109572572"
Sep 16 16:58:04.363411 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:04.363355    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh" podStartSLOduration=1.363337746 podStartE2EDuration="1.363337746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:58:04.36224048 +0000 UTC m=+1284.124201929" watchObservedRunningTime="2026-09-16 16:58:04.363337746 +0000 UTC m=+1284.125299194"
Sep 16 16:58:10.343250 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:10.343215    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"
Sep 16 16:58:11.369308 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:11.369275    2556 generic.go:356] "Generic (PLEG): container finished" podID="576f4074-7fe9-42cb-810b-bb18acdbfa69" containerID="bb92001a47da9b6413d668a98f3cd3db175359678daabc5f6458f0987412bb6b" exitCode=0
Sep 16 16:58:11.369741 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:11.369360    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r" event={"ID":"576f4074-7fe9-42cb-810b-bb18acdbfa69","Type":"ContainerDied","Data":"bb92001a47da9b6413d668a98f3cd3db175359678daabc5f6458f0987412bb6b"}
Sep 16 16:58:11.369896 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:11.369880    2556 scope.go:117] "RemoveContainer" containerID="bb92001a47da9b6413d668a98f3cd3db175359678daabc5f6458f0987412bb6b"
Sep 16 16:58:11.370995 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:11.370969    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 16:58:11.371063 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:11.371042    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 16:58:11.371101 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:11.371077    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 16:58:12.376759 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:12.376719    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r" event={"ID":"576f4074-7fe9-42cb-810b-bb18acdbfa69","Type":"ContainerStarted","Data":"73d45c78c4fc180761f3aeee8bee01c20090f56ef42df521a84fe38d357c919c"}
Sep 16 16:58:12.914667 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:12.914636    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:58:12.914962 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:12.914910    2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 16 16:58:12.915089 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:12.914994    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r" podUID="576f4074-7fe9-42cb-810b-bb18acdbfa69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 16 16:58:13.914750 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:13.914711    2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 16 16:58:13.915150 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:13.914769    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r" podUID="576f4074-7fe9-42cb-810b-bb18acdbfa69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 16 16:58:14.915450 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:14.915416    2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 16 16:58:14.915821 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:14.915475    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r" podUID="576f4074-7fe9-42cb-810b-bb18acdbfa69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 16 16:58:15.340018 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:15.339987    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh"
Sep 16 16:58:15.890626 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:15.890589    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml"]
Sep 16 16:58:15.893129 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:15.893109    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml"
Sep 16 16:58:15.896022 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:15.895992    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-34d301\""
Sep 16 16:58:15.896118 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:15.896097    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-34d301-dockercfg-r6xvk\""
Sep 16 16:58:15.912365 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:15.912341    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml"]
Sep 16 16:58:15.914526 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:15.914505    2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 16 16:58:15.914601 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:15.914547    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r" podUID="576f4074-7fe9-42cb-810b-bb18acdbfa69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 16 16:58:15.928791 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:15.928762    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml\" (UID: \"e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml"
Sep 16 16:58:15.929214 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:15.928839    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2jb8\" (UniqueName: \"kubernetes.io/projected/e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a-kube-api-access-f2jb8\") pod \"payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml\" (UID: \"e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml"
Sep 16 16:58:16.029540 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:16.029502    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml\" (UID: \"e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml"
Sep 16 16:58:16.029731 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:16.029561    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2jb8\" (UniqueName: \"kubernetes.io/projected/e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a-kube-api-access-f2jb8\") pod \"payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml\" (UID: \"e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml"
Sep 16 16:58:16.030292 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:16.030266    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml\" (UID: \"e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml"
Sep 16 16:58:16.038272 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:16.038249    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2jb8\" (UniqueName: \"kubernetes.io/projected/e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a-kube-api-access-f2jb8\") pod \"payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml\" (UID: \"e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml"
Sep 16 16:58:16.204156 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:16.204127    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml"
Sep 16 16:58:16.515631 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:16.515605    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml"]
Sep 16 16:58:16.516714 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:58:16.516687    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a62cc7_1785_4a1d_92e4_4d64e5d8e27a.slice/crio-534b433bce13d626071cf0ba0bd5ed59fd30ed62c4dc2de4c86d145f3b0d5350 WatchSource:0}: Error finding container 534b433bce13d626071cf0ba0bd5ed59fd30ed62c4dc2de4c86d145f3b0d5350: Status 404 returned error can't find the container with id 534b433bce13d626071cf0ba0bd5ed59fd30ed62c4dc2de4c86d145f3b0d5350
Sep 16 16:58:16.914833 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:16.914748    2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Sep 16 16:58:16.914833 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:16.914809    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r" podUID="576f4074-7fe9-42cb-810b-bb18acdbfa69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Sep 16 16:58:17.404652 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:17.404615    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml" event={"ID":"e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a","Type":"ContainerStarted","Data":"e5a2c7307d9e816e6bf0acb11216479692b941d4bd050ac57dd15264be241987"}
Sep 16 16:58:17.405161 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:17.404676    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml" event={"ID":"e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a","Type":"ContainerStarted","Data":"534b433bce13d626071cf0ba0bd5ed59fd30ed62c4dc2de4c86d145f3b0d5350"}
Sep 16 16:58:17.405161 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:17.404756    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml"
Sep 16 16:58:17.450905 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:17.450824    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml" podStartSLOduration=2.450802648 podStartE2EDuration="2.450802648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:58:17.446175204 +0000 UTC m=+1297.208136653" watchObservedRunningTime="2026-09-16 16:58:17.450802648 +0000 UTC m=+1297.212764097"
Sep 16 16:58:17.918788 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:17.918759    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:58:17.919068 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:17.919047    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:58:17.919768 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:17.919747    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 16:58:28.411358 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:28.411327    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml"
Sep 16 16:58:36.963314 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:36.963281    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2"]
Sep 16 16:58:36.966192 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:36.966172    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2"
Sep 16 16:58:36.968635 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:36.968605    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6148d487\""
Sep 16 16:58:36.968971 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:36.968935    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6148d487-dockercfg-q9mt6\""
Sep 16 16:58:36.980506 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:36.980483    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2"]
Sep 16 16:58:37.019683 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:37.019647    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74b024f7-f1bd-4be6-8e99-354dcd0cc4f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2\" (UID: \"74b024f7-f1bd-4be6-8e99-354dcd0cc4f1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2"
Sep 16 16:58:37.019889 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:37.019790    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz49s\" (UniqueName: \"kubernetes.io/projected/74b024f7-f1bd-4be6-8e99-354dcd0cc4f1-kube-api-access-lz49s\") pod \"payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2\" (UID: \"74b024f7-f1bd-4be6-8e99-354dcd0cc4f1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2"
Sep 16 16:58:37.121240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:37.121208    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74b024f7-f1bd-4be6-8e99-354dcd0cc4f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2\" (UID: \"74b024f7-f1bd-4be6-8e99-354dcd0cc4f1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2"
Sep 16 16:58:37.121240 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:37.121245    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz49s\" (UniqueName: \"kubernetes.io/projected/74b024f7-f1bd-4be6-8e99-354dcd0cc4f1-kube-api-access-lz49s\") pod \"payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2\" (UID: \"74b024f7-f1bd-4be6-8e99-354dcd0cc4f1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2"
Sep 16 16:58:37.121990 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:37.121972    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74b024f7-f1bd-4be6-8e99-354dcd0cc4f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2\" (UID: \"74b024f7-f1bd-4be6-8e99-354dcd0cc4f1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2"
Sep 16 16:58:37.130899 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:37.130875    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz49s\" (UniqueName: \"kubernetes.io/projected/74b024f7-f1bd-4be6-8e99-354dcd0cc4f1-kube-api-access-lz49s\") pod \"payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2\" (UID: \"74b024f7-f1bd-4be6-8e99-354dcd0cc4f1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2"
Sep 16 16:58:37.276673 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:37.276585    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2"
Sep 16 16:58:37.551837 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:37.551810    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2"]
Sep 16 16:58:37.553962 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:58:37.553912    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b024f7_f1bd_4be6_8e99_354dcd0cc4f1.slice/crio-792498a6abf5ff15c6226a5ec5d14cef95a714ee0fb3155285cecfd82bb90401 WatchSource:0}: Error finding container 792498a6abf5ff15c6226a5ec5d14cef95a714ee0fb3155285cecfd82bb90401: Status 404 returned error can't find the container with id 792498a6abf5ff15c6226a5ec5d14cef95a714ee0fb3155285cecfd82bb90401
Sep 16 16:58:38.514450 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:38.514403    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2" event={"ID":"74b024f7-f1bd-4be6-8e99-354dcd0cc4f1","Type":"ContainerStarted","Data":"41de3e07c7e7ec4d263927a572c5c070effdbdf7fa6a536410f5653bdf9fac8e"}
Sep 16 16:58:38.514870 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:38.514561    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2"
Sep 16 16:58:38.514870 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:38.514602    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2" event={"ID":"74b024f7-f1bd-4be6-8e99-354dcd0cc4f1","Type":"ContainerStarted","Data":"792498a6abf5ff15c6226a5ec5d14cef95a714ee0fb3155285cecfd82bb90401"}
Sep 16 16:58:38.537157 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:38.537110    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2" podStartSLOduration=2.537095764 podStartE2EDuration="2.537095764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:58:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:58:38.536535443 +0000 UTC m=+1318.298496891" watchObservedRunningTime="2026-09-16 16:58:38.537095764 +0000 UTC m=+1318.299057211"
Sep 16 16:58:49.521713 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:58:49.521678    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2"
Sep 16 16:59:42.388664 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.388628    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"]
Sep 16 16:59:42.391934 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.391904    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:42.395279 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.395259    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6148d487\"/\"default-dockercfg-chjf8\""
Sep 16 16:59:42.395381 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.395259    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6148d487\"/\"kube-root-ca.crt\""
Sep 16 16:59:42.395381 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.395263    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6148d487\"/\"openshift-service-ca.crt\""
Sep 16 16:59:42.395499 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.395385    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6148d487\"/\"e2e-lifecycle-model-6148d487-kserve-self-signed-certs\""
Sep 16 16:59:42.402043 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.402024    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"]
Sep 16 16:59:42.492775 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.492741    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-dshm\") pod \"e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") " pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:42.492775 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.492776    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-tmp-dir\") pod \"e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") " pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:42.492997 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.492909    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f72afe6c-e7ac-4286-aab7-edc1347bd537-tls-certs\") pod \"e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") " pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:42.492997 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.492966    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-model-cache\") pod \"e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") " pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:42.492997 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.492987    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nc5t\" (UniqueName: \"kubernetes.io/projected/f72afe6c-e7ac-4286-aab7-edc1347bd537-kube-api-access-7nc5t\") pod \"e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") " pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:42.493105 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.493067    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-home\") pod \"e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") " pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:42.593727 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.593694    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-model-cache\") pod \"e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") " pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:42.593727 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.593729    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nc5t\" (UniqueName: \"kubernetes.io/projected/f72afe6c-e7ac-4286-aab7-edc1347bd537-kube-api-access-7nc5t\") pod \"e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") " pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:42.593972 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.593769    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-home\") pod \"e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") " pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:42.593972 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.593799    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-dshm\") pod \"e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") " pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:42.593972 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.593815    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-tmp-dir\") pod \"e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") " pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:42.593972 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.593873    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f72afe6c-e7ac-4286-aab7-edc1347bd537-tls-certs\") pod \"e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") " pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:42.594190 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.594116    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-model-cache\") pod \"e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") " pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:42.594190 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.594164    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-home\") pod \"e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") " pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:42.594300 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.594279    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-tmp-dir\") pod \"e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") " pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:42.596100 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.596070    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-dshm\") pod \"e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") " pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:42.596374 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.596355    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f72afe6c-e7ac-4286-aab7-edc1347bd537-tls-certs\") pod \"e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") " pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:42.602211 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.602190    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nc5t\" (UniqueName: \"kubernetes.io/projected/f72afe6c-e7ac-4286-aab7-edc1347bd537-kube-api-access-7nc5t\") pod \"e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") " pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:42.701134 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.701105    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:42.980205 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:42.980137    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"]
Sep 16 16:59:42.982712 ip-10-0-140-120 kubenswrapper[2556]: W0916 16:59:42.982688    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf72afe6c_e7ac_4286_aab7_edc1347bd537.slice/crio-77ee76ccc7be9369e99d71130fbf1e941625f0d7a7ee17dca0b11952f349b737 WatchSource:0}: Error finding container 77ee76ccc7be9369e99d71130fbf1e941625f0d7a7ee17dca0b11952f349b737: Status 404 returned error can't find the container with id 77ee76ccc7be9369e99d71130fbf1e941625f0d7a7ee17dca0b11952f349b737
Sep 16 16:59:43.842118 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:43.842086    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l" event={"ID":"f72afe6c-e7ac-4286-aab7-edc1347bd537","Type":"ContainerStarted","Data":"fdd2ea4d6e6d026152b40f22d21d9f071a46e20753bbe4093ffc4e469e73f82a"}
Sep 16 16:59:43.842513 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:43.842131    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l" event={"ID":"f72afe6c-e7ac-4286-aab7-edc1347bd537","Type":"ContainerStarted","Data":"77ee76ccc7be9369e99d71130fbf1e941625f0d7a7ee17dca0b11952f349b737"}
Sep 16 16:59:43.877473 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:43.877422    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l" podStartSLOduration=1.877391728 podStartE2EDuration="1.877391728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 16:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 16:59:43.875310308 +0000 UTC m=+1383.637271757" watchObservedRunningTime="2026-09-16 16:59:43.877391728 +0000 UTC m=+1383.639353175"
Sep 16 16:59:52.701918 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:52.701880    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:52.702409 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:52.701932    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:52.714587 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:52.714557    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 16:59:52.899350 ip-10-0-140-120 kubenswrapper[2556]: I0916 16:59:52.899322    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 17:00:00.176731 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:00.176700    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-52fd8c-29826300-f5jl8"]
Sep 16 17:00:00.180230 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:00.180214    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-52fd8c-29826300-f5jl8"
Sep 16 17:00:00.182560 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:00.182536    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-69rc9\""
Sep 16 17:00:00.186067 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:00.186043    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-52fd8c-29826300-f5jl8"]
Sep 16 17:00:00.193136 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:00.193116    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-52fd8c-29826300-f5jl8"
Sep 16 17:00:00.466882 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:00.466854    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-52fd8c-29826300-f5jl8"]
Sep 16 17:00:00.468558 ip-10-0-140-120 kubenswrapper[2556]: W0916 17:00:00.468532    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1223020_cfe7_427e_a1da_0c51dc1f493b.slice/crio-0be4efeba27a27d5b3ccbb047d51f1c092d5ac5a005e7d596c434910c77fe199 WatchSource:0}: Error finding container 0be4efeba27a27d5b3ccbb047d51f1c092d5ac5a005e7d596c434910c77fe199: Status 404 returned error can't find the container with id 0be4efeba27a27d5b3ccbb047d51f1c092d5ac5a005e7d596c434910c77fe199
Sep 16 17:00:00.935164 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:00.935126    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-52fd8c-29826300-f5jl8" event={"ID":"a1223020-cfe7-427e-a1da-0c51dc1f493b","Type":"ContainerStarted","Data":"0be4efeba27a27d5b3ccbb047d51f1c092d5ac5a005e7d596c434910c77fe199"}
Sep 16 17:00:03.952828 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:03.952793    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-52fd8c-29826300-f5jl8" event={"ID":"a1223020-cfe7-427e-a1da-0c51dc1f493b","Type":"ContainerStarted","Data":"9128183d82e45afd426a5d693586341be550e2786d6d329c7dace962fa547b2f"}
Sep 16 17:00:03.967180 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:03.967133    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-52fd8c-29826300-f5jl8" podStartSLOduration=1.051035249 podStartE2EDuration="3.967118763s" totalImagesPullingTime="2.916083514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:00:03.965890832 +0000 UTC m=+1403.727852278" watchObservedRunningTime="2026-09-16 17:00:03.967118763 +0000 UTC m=+1403.729080210"
Sep 16 17:00:04.959800 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:04.959765    2556 generic.go:356] "Generic (PLEG): container finished" podID="a1223020-cfe7-427e-a1da-0c51dc1f493b" containerID="9128183d82e45afd426a5d693586341be550e2786d6d329c7dace962fa547b2f" exitCode=0
Sep 16 17:00:04.960192 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:04.959850    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-52fd8c-29826300-f5jl8" event={"ID":"a1223020-cfe7-427e-a1da-0c51dc1f493b","Type":"ContainerDied","Data":"9128183d82e45afd426a5d693586341be550e2786d6d329c7dace962fa547b2f"}
Sep 16 17:00:06.099331 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:06.099304    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-52fd8c-29826300-f5jl8"
Sep 16 17:00:06.971753 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:06.971710    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-52fd8c-29826300-f5jl8"
Sep 16 17:00:06.971753 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:06.971748    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-52fd8c-29826300-f5jl8" event={"ID":"a1223020-cfe7-427e-a1da-0c51dc1f493b","Type":"ContainerDied","Data":"0be4efeba27a27d5b3ccbb047d51f1c092d5ac5a005e7d596c434910c77fe199"}
Sep 16 17:00:06.971991 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:06.971792    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0be4efeba27a27d5b3ccbb047d51f1c092d5ac5a005e7d596c434910c77fe199"
Sep 16 17:00:38.351458 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:38.351236    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2"]
Sep 16 17:00:38.352339 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:38.352015    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2" podUID="74b024f7-f1bd-4be6-8e99-354dcd0cc4f1" containerName="payload-pre-processing" containerID="cri-o://41de3e07c7e7ec4d263927a572c5c070effdbdf7fa6a536410f5653bdf9fac8e" gracePeriod=30
Sep 16 17:00:38.613257 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:38.613200    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2"
Sep 16 17:00:38.721352 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:38.721317    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74b024f7-f1bd-4be6-8e99-354dcd0cc4f1-plugins-config-volume\") pod \"74b024f7-f1bd-4be6-8e99-354dcd0cc4f1\" (UID: \"74b024f7-f1bd-4be6-8e99-354dcd0cc4f1\") "
Sep 16 17:00:38.721542 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:38.721416    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz49s\" (UniqueName: \"kubernetes.io/projected/74b024f7-f1bd-4be6-8e99-354dcd0cc4f1-kube-api-access-lz49s\") pod \"74b024f7-f1bd-4be6-8e99-354dcd0cc4f1\" (UID: \"74b024f7-f1bd-4be6-8e99-354dcd0cc4f1\") "
Sep 16 17:00:38.721748 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:38.721721    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b024f7-f1bd-4be6-8e99-354dcd0cc4f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74b024f7-f1bd-4be6-8e99-354dcd0cc4f1" (UID: "74b024f7-f1bd-4be6-8e99-354dcd0cc4f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:00:38.723450 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:38.723426    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b024f7-f1bd-4be6-8e99-354dcd0cc4f1-kube-api-access-lz49s" (OuterVolumeSpecName: "kube-api-access-lz49s") pod "74b024f7-f1bd-4be6-8e99-354dcd0cc4f1" (UID: "74b024f7-f1bd-4be6-8e99-354dcd0cc4f1"). InnerVolumeSpecName "kube-api-access-lz49s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:00:38.822506 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:38.822476    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74b024f7-f1bd-4be6-8e99-354dcd0cc4f1-plugins-config-volume\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:00:38.822506 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:38.822503    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz49s\" (UniqueName: \"kubernetes.io/projected/74b024f7-f1bd-4be6-8e99-354dcd0cc4f1-kube-api-access-lz49s\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:00:39.132698 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:39.132661    2556 generic.go:356] "Generic (PLEG): container finished" podID="74b024f7-f1bd-4be6-8e99-354dcd0cc4f1" containerID="41de3e07c7e7ec4d263927a572c5c070effdbdf7fa6a536410f5653bdf9fac8e" exitCode=0
Sep 16 17:00:39.132889 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:39.132706    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2"
Sep 16 17:00:39.132889 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:39.132709    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2" event={"ID":"74b024f7-f1bd-4be6-8e99-354dcd0cc4f1","Type":"ContainerDied","Data":"41de3e07c7e7ec4d263927a572c5c070effdbdf7fa6a536410f5653bdf9fac8e"}
Sep 16 17:00:39.132889 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:39.132826    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2" event={"ID":"74b024f7-f1bd-4be6-8e99-354dcd0cc4f1","Type":"ContainerDied","Data":"792498a6abf5ff15c6226a5ec5d14cef95a714ee0fb3155285cecfd82bb90401"}
Sep 16 17:00:39.132889 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:39.132857    2556 scope.go:117] "RemoveContainer" containerID="41de3e07c7e7ec4d263927a572c5c070effdbdf7fa6a536410f5653bdf9fac8e"
Sep 16 17:00:39.144084 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:39.144041    2556 scope.go:117] "RemoveContainer" containerID="41de3e07c7e7ec4d263927a572c5c070effdbdf7fa6a536410f5653bdf9fac8e"
Sep 16 17:00:39.144404 ip-10-0-140-120 kubenswrapper[2556]: E0916 17:00:39.144374    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41de3e07c7e7ec4d263927a572c5c070effdbdf7fa6a536410f5653bdf9fac8e\": container with ID starting with 41de3e07c7e7ec4d263927a572c5c070effdbdf7fa6a536410f5653bdf9fac8e not found: ID does not exist" containerID="41de3e07c7e7ec4d263927a572c5c070effdbdf7fa6a536410f5653bdf9fac8e"
Sep 16 17:00:39.144493 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:39.144413    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41de3e07c7e7ec4d263927a572c5c070effdbdf7fa6a536410f5653bdf9fac8e"} err="failed to get container status \"41de3e07c7e7ec4d263927a572c5c070effdbdf7fa6a536410f5653bdf9fac8e\": rpc error: code = NotFound desc = could not find container \"41de3e07c7e7ec4d263927a572c5c070effdbdf7fa6a536410f5653bdf9fac8e\": container with ID starting with 41de3e07c7e7ec4d263927a572c5c070effdbdf7fa6a536410f5653bdf9fac8e not found: ID does not exist"
Sep 16 17:00:39.177864 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:39.177834    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2"]
Sep 16 17:00:39.185423 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:39.185399    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6148d487-6f9db67646-ldzb2"]
Sep 16 17:00:40.849926 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:40.849889    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74b024f7-f1bd-4be6-8e99-354dcd0cc4f1" path="/var/lib/kubelet/pods/74b024f7-f1bd-4be6-8e99-354dcd0cc4f1/volumes"
Sep 16 17:00:46.692097 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:00:46.692062    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"]
Sep 16 17:01:01.711318 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:01.711255    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l" podUID="f72afe6c-e7ac-4286-aab7-edc1347bd537" containerName="main" containerID="cri-o://fdd2ea4d6e6d026152b40f22d21d9f071a46e20753bbe4093ffc4e469e73f82a" gracePeriod=45
Sep 16 17:01:01.962222 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:01.962156    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 17:01:02.034662 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.034627    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nc5t\" (UniqueName: \"kubernetes.io/projected/f72afe6c-e7ac-4286-aab7-edc1347bd537-kube-api-access-7nc5t\") pod \"f72afe6c-e7ac-4286-aab7-edc1347bd537\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") "
Sep 16 17:01:02.034855 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.034725    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-model-cache\") pod \"f72afe6c-e7ac-4286-aab7-edc1347bd537\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") "
Sep 16 17:01:02.034855 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.034767    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-dshm\") pod \"f72afe6c-e7ac-4286-aab7-edc1347bd537\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") "
Sep 16 17:01:02.034855 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.034792    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-home\") pod \"f72afe6c-e7ac-4286-aab7-edc1347bd537\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") "
Sep 16 17:01:02.034855 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.034820    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-tmp-dir\") pod \"f72afe6c-e7ac-4286-aab7-edc1347bd537\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") "
Sep 16 17:01:02.034855 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.034845    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f72afe6c-e7ac-4286-aab7-edc1347bd537-tls-certs\") pod \"f72afe6c-e7ac-4286-aab7-edc1347bd537\" (UID: \"f72afe6c-e7ac-4286-aab7-edc1347bd537\") "
Sep 16 17:01:02.035153 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.035000    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-model-cache" (OuterVolumeSpecName: "model-cache") pod "f72afe6c-e7ac-4286-aab7-edc1347bd537" (UID: "f72afe6c-e7ac-4286-aab7-edc1347bd537"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:01:02.035153 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.035061    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-home" (OuterVolumeSpecName: "home") pod "f72afe6c-e7ac-4286-aab7-edc1347bd537" (UID: "f72afe6c-e7ac-4286-aab7-edc1347bd537"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:01:02.035278 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.035145    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f72afe6c-e7ac-4286-aab7-edc1347bd537" (UID: "f72afe6c-e7ac-4286-aab7-edc1347bd537"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:01:02.035339 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.035293    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-model-cache\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:01:02.035339 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.035312    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-home\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:01:02.035430 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.035352    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-tmp-dir\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:01:02.036997 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.036965    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-dshm" (OuterVolumeSpecName: "dshm") pod "f72afe6c-e7ac-4286-aab7-edc1347bd537" (UID: "f72afe6c-e7ac-4286-aab7-edc1347bd537"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:01:02.037153 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.037051    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72afe6c-e7ac-4286-aab7-edc1347bd537-kube-api-access-7nc5t" (OuterVolumeSpecName: "kube-api-access-7nc5t") pod "f72afe6c-e7ac-4286-aab7-edc1347bd537" (UID: "f72afe6c-e7ac-4286-aab7-edc1347bd537"). InnerVolumeSpecName "kube-api-access-7nc5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:01:02.037373 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.037348    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72afe6c-e7ac-4286-aab7-edc1347bd537-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f72afe6c-e7ac-4286-aab7-edc1347bd537" (UID: "f72afe6c-e7ac-4286-aab7-edc1347bd537"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:01:02.136578 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.136539    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f72afe6c-e7ac-4286-aab7-edc1347bd537-dshm\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:01:02.136578 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.136573    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f72afe6c-e7ac-4286-aab7-edc1347bd537-tls-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:01:02.136578 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.136584    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nc5t\" (UniqueName: \"kubernetes.io/projected/f72afe6c-e7ac-4286-aab7-edc1347bd537-kube-api-access-7nc5t\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:01:02.249582 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.249490    2556 generic.go:356] "Generic (PLEG): container finished" podID="f72afe6c-e7ac-4286-aab7-edc1347bd537" containerID="fdd2ea4d6e6d026152b40f22d21d9f071a46e20753bbe4093ffc4e469e73f82a" exitCode=0
Sep 16 17:01:02.249582 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.249536    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"
Sep 16 17:01:02.249780 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.249579    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l" event={"ID":"f72afe6c-e7ac-4286-aab7-edc1347bd537","Type":"ContainerDied","Data":"fdd2ea4d6e6d026152b40f22d21d9f071a46e20753bbe4093ffc4e469e73f82a"}
Sep 16 17:01:02.249780 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.249621    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l" event={"ID":"f72afe6c-e7ac-4286-aab7-edc1347bd537","Type":"ContainerDied","Data":"77ee76ccc7be9369e99d71130fbf1e941625f0d7a7ee17dca0b11952f349b737"}
Sep 16 17:01:02.249780 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.249638    2556 scope.go:117] "RemoveContainer" containerID="fdd2ea4d6e6d026152b40f22d21d9f071a46e20753bbe4093ffc4e469e73f82a"
Sep 16 17:01:02.258778 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.258758    2556 scope.go:117] "RemoveContainer" containerID="fdd2ea4d6e6d026152b40f22d21d9f071a46e20753bbe4093ffc4e469e73f82a"
Sep 16 17:01:02.259070 ip-10-0-140-120 kubenswrapper[2556]: E0916 17:01:02.259050    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd2ea4d6e6d026152b40f22d21d9f071a46e20753bbe4093ffc4e469e73f82a\": container with ID starting with fdd2ea4d6e6d026152b40f22d21d9f071a46e20753bbe4093ffc4e469e73f82a not found: ID does not exist" containerID="fdd2ea4d6e6d026152b40f22d21d9f071a46e20753bbe4093ffc4e469e73f82a"
Sep 16 17:01:02.259148 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.259081    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd2ea4d6e6d026152b40f22d21d9f071a46e20753bbe4093ffc4e469e73f82a"} err="failed to get container status \"fdd2ea4d6e6d026152b40f22d21d9f071a46e20753bbe4093ffc4e469e73f82a\": rpc error: code = NotFound desc = could not find container \"fdd2ea4d6e6d026152b40f22d21d9f071a46e20753bbe4093ffc4e469e73f82a\": container with ID starting with fdd2ea4d6e6d026152b40f22d21d9f071a46e20753bbe4093ffc4e469e73f82a not found: ID does not exist"
Sep 16 17:01:02.272791 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.272761    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"]
Sep 16 17:01:02.275055 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.275034    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-6148d487/e2e-lifecycle-model-6148d487-kserve-d75b6c979-tw54l"]
Sep 16 17:01:02.849649 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:01:02.849620    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f72afe6c-e7ac-4286-aab7-edc1347bd537" path="/var/lib/kubelet/pods/f72afe6c-e7ac-4286-aab7-edc1347bd537/volumes"
Sep 16 17:02:31.191414 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.191376    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"]
Sep 16 17:02:31.191912 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.191857    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f72afe6c-e7ac-4286-aab7-edc1347bd537" containerName="main"
Sep 16 17:02:31.191912 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.191877    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f72afe6c-e7ac-4286-aab7-edc1347bd537" containerName="main"
Sep 16 17:02:31.191912 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.191895    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74b024f7-f1bd-4be6-8e99-354dcd0cc4f1" containerName="payload-pre-processing"
Sep 16 17:02:31.191912 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.191902    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="74b024f7-f1bd-4be6-8e99-354dcd0cc4f1" containerName="payload-pre-processing"
Sep 16 17:02:31.192080 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.191915    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1223020-cfe7-427e-a1da-0c51dc1f493b" containerName="cleanup"
Sep 16 17:02:31.192080 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.191920    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1223020-cfe7-427e-a1da-0c51dc1f493b" containerName="cleanup"
Sep 16 17:02:31.192080 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.191997    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f72afe6c-e7ac-4286-aab7-edc1347bd537" containerName="main"
Sep 16 17:02:31.192080 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.192005    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1223020-cfe7-427e-a1da-0c51dc1f493b" containerName="cleanup"
Sep 16 17:02:31.192080 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.192015    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="74b024f7-f1bd-4be6-8e99-354dcd0cc4f1" containerName="payload-pre-processing"
Sep 16 17:02:31.194183 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.194167    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"
Sep 16 17:02:31.202730 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.202708    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9359a3-serving-cert\""
Sep 16 17:02:31.222879 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.222856    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"]
Sep 16 17:02:31.320172 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.320142    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59fabb07-97fc-4515-93c4-1bac3f908a7c-openshift-service-ca\") pod \"maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8\" (UID: \"59fabb07-97fc-4515-93c4-1bac3f908a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"
Sep 16 17:02:31.320172 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.320175    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59fabb07-97fc-4515-93c4-1bac3f908a7c-metrics-certs\") pod \"maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8\" (UID: \"59fabb07-97fc-4515-93c4-1bac3f908a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"
Sep 16 17:02:31.320364 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.320198    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59fabb07-97fc-4515-93c4-1bac3f908a7c-maas-api-tls\") pod \"maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8\" (UID: \"59fabb07-97fc-4515-93c4-1bac3f908a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"
Sep 16 17:02:31.320364 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.320304    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-984mt\" (UniqueName: \"kubernetes.io/projected/59fabb07-97fc-4515-93c4-1bac3f908a7c-kube-api-access-984mt\") pod \"maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8\" (UID: \"59fabb07-97fc-4515-93c4-1bac3f908a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"
Sep 16 17:02:31.421140 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.421099    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59fabb07-97fc-4515-93c4-1bac3f908a7c-maas-api-tls\") pod \"maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8\" (UID: \"59fabb07-97fc-4515-93c4-1bac3f908a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"
Sep 16 17:02:31.421344 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.421171    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-984mt\" (UniqueName: \"kubernetes.io/projected/59fabb07-97fc-4515-93c4-1bac3f908a7c-kube-api-access-984mt\") pod \"maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8\" (UID: \"59fabb07-97fc-4515-93c4-1bac3f908a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"
Sep 16 17:02:31.421344 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.421232    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59fabb07-97fc-4515-93c4-1bac3f908a7c-openshift-service-ca\") pod \"maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8\" (UID: \"59fabb07-97fc-4515-93c4-1bac3f908a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"
Sep 16 17:02:31.421344 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.421250    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59fabb07-97fc-4515-93c4-1bac3f908a7c-metrics-certs\") pod \"maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8\" (UID: \"59fabb07-97fc-4515-93c4-1bac3f908a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"
Sep 16 17:02:31.421344 ip-10-0-140-120 kubenswrapper[2556]: E0916 17:02:31.421263    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-serving-cert: secret "maas-api-e2e-api-a-9359a3-serving-cert" not found
Sep 16 17:02:31.421567 ip-10-0-140-120 kubenswrapper[2556]: E0916 17:02:31.421349    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59fabb07-97fc-4515-93c4-1bac3f908a7c-maas-api-tls podName:59fabb07-97fc-4515-93c4-1bac3f908a7c nodeName:}" failed. No retries permitted until 2026-09-16 17:02:31.921327657 +0000 UTC m=+1551.683289096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/59fabb07-97fc-4515-93c4-1bac3f908a7c-maas-api-tls") pod "maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8" (UID: "59fabb07-97fc-4515-93c4-1bac3f908a7c") : secret "maas-api-e2e-api-a-9359a3-serving-cert" not found
Sep 16 17:02:31.421924 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.421905    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59fabb07-97fc-4515-93c4-1bac3f908a7c-openshift-service-ca\") pod \"maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8\" (UID: \"59fabb07-97fc-4515-93c4-1bac3f908a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"
Sep 16 17:02:31.423575 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.423559    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59fabb07-97fc-4515-93c4-1bac3f908a7c-metrics-certs\") pod \"maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8\" (UID: \"59fabb07-97fc-4515-93c4-1bac3f908a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"
Sep 16 17:02:31.431777 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.431751    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-984mt\" (UniqueName: \"kubernetes.io/projected/59fabb07-97fc-4515-93c4-1bac3f908a7c-kube-api-access-984mt\") pod \"maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8\" (UID: \"59fabb07-97fc-4515-93c4-1bac3f908a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"
Sep 16 17:02:31.925568 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.925531    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59fabb07-97fc-4515-93c4-1bac3f908a7c-maas-api-tls\") pod \"maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8\" (UID: \"59fabb07-97fc-4515-93c4-1bac3f908a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"
Sep 16 17:02:31.927842 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:31.927819    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59fabb07-97fc-4515-93c4-1bac3f908a7c-maas-api-tls\") pod \"maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8\" (UID: \"59fabb07-97fc-4515-93c4-1bac3f908a7c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"
Sep 16 17:02:32.104896 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:32.104863    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"
Sep 16 17:02:32.388787 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:32.388760    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"]
Sep 16 17:02:32.390524 ip-10-0-140-120 kubenswrapper[2556]: W0916 17:02:32.390495    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59fabb07_97fc_4515_93c4_1bac3f908a7c.slice/crio-b6b4a768ed0c771c91ad67ae79f0b9c6590bbb4fb207283c0e363f0ed1f60478 WatchSource:0}: Error finding container b6b4a768ed0c771c91ad67ae79f0b9c6590bbb4fb207283c0e363f0ed1f60478: Status 404 returned error can't find the container with id b6b4a768ed0c771c91ad67ae79f0b9c6590bbb4fb207283c0e363f0ed1f60478
Sep 16 17:02:32.693934 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:32.693898    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8" event={"ID":"59fabb07-97fc-4515-93c4-1bac3f908a7c","Type":"ContainerStarted","Data":"b6b4a768ed0c771c91ad67ae79f0b9c6590bbb4fb207283c0e363f0ed1f60478"}
Sep 16 17:02:33.700545 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:33.700512    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8" event={"ID":"59fabb07-97fc-4515-93c4-1bac3f908a7c","Type":"ContainerStarted","Data":"ad6d251f68eb7dc7c92e494eaa2d799f59e58115acdb581e075f83a6ca19e012"}
Sep 16 17:02:33.700990 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:33.700622    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"
Sep 16 17:02:33.717876 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:33.717828    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8" podStartSLOduration=2.219157144 podStartE2EDuration="2.71781292s" totalImagesPullingTime="498.655776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:02:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:02:33.716305299 +0000 UTC m=+1553.478266747" watchObservedRunningTime="2026-09-16 17:02:33.71781292 +0000 UTC m=+1553.479774367"
Sep 16 17:02:39.710911 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:39.710880    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"
Sep 16 17:02:58.422507 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.422471    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"]
Sep 16 17:02:58.425315 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.425292    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.428078 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.428057    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4acea1-openshift-default-dockercfg-pj4wn\""
Sep 16 17:02:58.444578 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.444552    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"]
Sep 16 17:02:58.557811 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.557744    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-credential-socket\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.558134 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.557845    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-workload-socket\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.558134 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.557982    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-workload-certs\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.558134 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.558018    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-data\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.558134 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.558077    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-envoy\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.558134 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.558104    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-podinfo\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.558500 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.558153    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxxrt\" (UniqueName: \"kubernetes.io/projected/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-kube-api-access-nxxrt\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.558500 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.558259    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-token\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.558500 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.558300    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istiod-ca-cert\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.659338 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.659300    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-credential-socket\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.659497 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.659346    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-workload-socket\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.659497 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.659383    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-workload-certs\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.659497 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.659399    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-data\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.659497 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.659427    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-envoy\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.659706 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.659660    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-podinfo\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.659706 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.659681    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-credential-socket\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.659921 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.659715    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxxrt\" (UniqueName: \"kubernetes.io/projected/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-kube-api-access-nxxrt\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.659921 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.659774    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-token\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.659921 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.659802    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-data\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.659921 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.659861    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-workload-socket\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.659921 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.659889    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-workload-certs\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.660158 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.659927    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istiod-ca-cert\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.660535 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.660513    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istiod-ca-cert\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.661648 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.661629    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-envoy\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.662029 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.662009    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-podinfo\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.674018 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.673972    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxxrt\" (UniqueName: \"kubernetes.io/projected/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-kube-api-access-nxxrt\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.685249 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.685217    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-token\") pod \"e2e-api-b-4acea1-openshift-default-645987b79b-6kwts\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") " pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:58.735347 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:58.735315    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:02:59.030286 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:59.030261    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"]
Sep 16 17:02:59.030409 ip-10-0-140-120 kubenswrapper[2556]: W0916 17:02:59.030289    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88c5c24_2795_4143_b5ef_dadfbb0e4c0b.slice/crio-57a3cf0c3a454c4e33cacd11e38ba0d61cdce708547644d2dad4ee9c2d2936c5 WatchSource:0}: Error finding container 57a3cf0c3a454c4e33cacd11e38ba0d61cdce708547644d2dad4ee9c2d2936c5: Status 404 returned error can't find the container with id 57a3cf0c3a454c4e33cacd11e38ba0d61cdce708547644d2dad4ee9c2d2936c5
Sep 16 17:02:59.032390 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:59.032359    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 17:02:59.032484 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:59.032420    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 17:02:59.032484 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:59.032451    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 17:02:59.834877 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:59.834844    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts" event={"ID":"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b","Type":"ContainerStarted","Data":"4c8791902a57e409e1b03a2655390dec925042f45195361a06b591e2db5589ca"}
Sep 16 17:02:59.835342 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:59.834897    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts" event={"ID":"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b","Type":"ContainerStarted","Data":"57a3cf0c3a454c4e33cacd11e38ba0d61cdce708547644d2dad4ee9c2d2936c5"}
Sep 16 17:02:59.867209 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:02:59.867149    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts" podStartSLOduration=1.867128014 podStartE2EDuration="1.867128014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:02:59.863585583 +0000 UTC m=+1579.625547030" watchObservedRunningTime="2026-09-16 17:02:59.867128014 +0000 UTC m=+1579.629089477"
Sep 16 17:03:00.735609 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:00.735570    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:03:00.740773 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:00.740744    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:03:00.840543 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:00.840514    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:03:00.841436 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:00.841415    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:03:28.996462 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:28.996424    2556 generic.go:356] "Generic (PLEG): container finished" podID="d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" containerID="4c8791902a57e409e1b03a2655390dec925042f45195361a06b591e2db5589ca" exitCode=0
Sep 16 17:03:28.997008 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:28.996488    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts" event={"ID":"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b","Type":"ContainerDied","Data":"4c8791902a57e409e1b03a2655390dec925042f45195361a06b591e2db5589ca"}
Sep 16 17:03:28.997103 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:28.997016    2556 scope.go:117] "RemoveContainer" containerID="4c8791902a57e409e1b03a2655390dec925042f45195361a06b591e2db5589ca"
Sep 16 17:03:28.998376 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:28.998346    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 17:03:28.998462 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:28.998415    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 17:03:28.998462 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:28.998447    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 17:03:30.003549 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.003502    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts" event={"ID":"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b","Type":"ContainerStarted","Data":"5dab7370e20c4ddbc43de2d7b3eb9fe51825c36b368be3f54d2bbb547da91af7"}
Sep 16 17:03:30.402422 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.402344    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"]
Sep 16 17:03:30.406043 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.406021    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:30.409099 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.409070    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-52fd8c\"/\"default-dockercfg-hsq9j\""
Sep 16 17:03:30.409262 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.409133    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-52fd8c\"/\"openshift-service-ca.crt\""
Sep 16 17:03:30.409262 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.409133    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-52fd8c\"/\"kube-root-ca.crt\""
Sep 16 17:03:30.409399 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.409162    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-52fd8c\"/\"rate-test-model-52fd8c-kserve-self-signed-certs\""
Sep 16 17:03:30.420560 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.420537    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"]
Sep 16 17:03:30.553586 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.553549    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4134341c-cc5e-4a57-99ef-901eafc3b78a-tls-certs\") pod \"rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") " pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:30.554067 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.554045    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-home\") pod \"rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") " pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:30.554265 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.554247    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jbng\" (UniqueName: \"kubernetes.io/projected/4134341c-cc5e-4a57-99ef-901eafc3b78a-kube-api-access-6jbng\") pod \"rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") " pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:30.554444 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.554430    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-model-cache\") pod \"rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") " pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:30.554595 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.554581    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-tmp-dir\") pod \"rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") " pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:30.554751 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.554738    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-dshm\") pod \"rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") " pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:30.655720 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.655495    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-tmp-dir\") pod \"rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") " pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:30.656827 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.656804    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-dshm\") pod \"rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") " pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:30.657508 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.657489    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4134341c-cc5e-4a57-99ef-901eafc3b78a-tls-certs\") pod \"rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") " pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:30.657732 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.657655    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-home\") pod \"rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") " pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:30.657952 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.657916    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jbng\" (UniqueName: \"kubernetes.io/projected/4134341c-cc5e-4a57-99ef-901eafc3b78a-kube-api-access-6jbng\") pod \"rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") " pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:30.658140 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.658122    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-model-cache\") pod \"rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") " pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:30.658275 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.658176    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-home\") pod \"rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") " pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:30.658488 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.656708    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-tmp-dir\") pod \"rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") " pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:30.658740 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.658723    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-model-cache\") pod \"rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") " pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:30.661345 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.661325    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-dshm\") pod \"rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") " pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:30.661469 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.661427    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4134341c-cc5e-4a57-99ef-901eafc3b78a-tls-certs\") pod \"rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") " pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:30.674851 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.674815    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jbng\" (UniqueName: \"kubernetes.io/projected/4134341c-cc5e-4a57-99ef-901eafc3b78a-kube-api-access-6jbng\") pod \"rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") " pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:30.715572 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.715544    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:30.736343 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.736319    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:03:30.737399 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.737375    2556 patch_prober.go:28] interesting pod/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 16 17:03:30.737487 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.737423    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts" podUID="d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 16 17:03:30.841323 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:30.841293    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:03:31.017783 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:31.017757    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"]
Sep 16 17:03:31.019227 ip-10-0-140-120 kubenswrapper[2556]: W0916 17:03:31.019206    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4134341c_cc5e_4a57_99ef_901eafc3b78a.slice/crio-0601562dab3f6d862a09d800e5bd77f2c17db4b8b1023baa817b904abf0b8a36 WatchSource:0}: Error finding container 0601562dab3f6d862a09d800e5bd77f2c17db4b8b1023baa817b904abf0b8a36: Status 404 returned error can't find the container with id 0601562dab3f6d862a09d800e5bd77f2c17db4b8b1023baa817b904abf0b8a36
Sep 16 17:03:31.736623 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:31.736587    2556 patch_prober.go:28] interesting pod/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 16 17:03:31.736791 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:31.736647    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts" podUID="d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 16 17:03:32.017923 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:32.017832    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk" event={"ID":"4134341c-cc5e-4a57-99ef-901eafc3b78a","Type":"ContainerStarted","Data":"02fbb12e474dbdff239ce2d338de4e8d4dcaad1da140cc01f8e34b854023249d"}
Sep 16 17:03:32.017923 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:32.017881    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk" event={"ID":"4134341c-cc5e-4a57-99ef-901eafc3b78a","Type":"ContainerStarted","Data":"0601562dab3f6d862a09d800e5bd77f2c17db4b8b1023baa817b904abf0b8a36"}
Sep 16 17:03:32.039948 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:32.039901    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk" podStartSLOduration=2.039887525 podStartE2EDuration="2.039887525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:03:32.038740796 +0000 UTC m=+1611.800702243" watchObservedRunningTime="2026-09-16 17:03:32.039887525 +0000 UTC m=+1611.801848989"
Sep 16 17:03:32.736552 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:32.736514    2556 patch_prober.go:28] interesting pod/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 16 17:03:32.736734 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:32.736569    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts" podUID="d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 16 17:03:33.735590 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:33.735556    2556 patch_prober.go:28] interesting pod/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 16 17:03:33.735990 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:33.735612    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts" podUID="d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 16 17:03:34.736011 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:34.735973    2556 patch_prober.go:28] interesting pod/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 16 17:03:34.736414 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:34.736034    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts" podUID="d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 16 17:03:35.740391 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:35.740357    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:03:35.741224 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:35.741206    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:03:40.716581 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:40.716547    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:40.717129 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:40.716593    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:40.729910 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:40.729884    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:41.073538 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:41.073449    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:03:55.707187 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:55.707147    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"]
Sep 16 17:03:55.707589 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:55.707386    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8" podUID="59fabb07-97fc-4515-93c4-1bac3f908a7c" containerName="maas-api" containerID="cri-o://ad6d251f68eb7dc7c92e494eaa2d799f59e58115acdb581e075f83a6ca19e012" gracePeriod=30
Sep 16 17:03:55.961736 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:55.961668    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"
Sep 16 17:03:56.104561 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.101292    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59fabb07-97fc-4515-93c4-1bac3f908a7c-openshift-service-ca\") pod \"59fabb07-97fc-4515-93c4-1bac3f908a7c\" (UID: \"59fabb07-97fc-4515-93c4-1bac3f908a7c\") "
Sep 16 17:03:56.104561 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.101406    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59fabb07-97fc-4515-93c4-1bac3f908a7c-maas-api-tls\") pod \"59fabb07-97fc-4515-93c4-1bac3f908a7c\" (UID: \"59fabb07-97fc-4515-93c4-1bac3f908a7c\") "
Sep 16 17:03:56.104561 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.101464    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-984mt\" (UniqueName: \"kubernetes.io/projected/59fabb07-97fc-4515-93c4-1bac3f908a7c-kube-api-access-984mt\") pod \"59fabb07-97fc-4515-93c4-1bac3f908a7c\" (UID: \"59fabb07-97fc-4515-93c4-1bac3f908a7c\") "
Sep 16 17:03:56.104561 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.101502    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59fabb07-97fc-4515-93c4-1bac3f908a7c-metrics-certs\") pod \"59fabb07-97fc-4515-93c4-1bac3f908a7c\" (UID: \"59fabb07-97fc-4515-93c4-1bac3f908a7c\") "
Sep 16 17:03:56.104561 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.101868    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59fabb07-97fc-4515-93c4-1bac3f908a7c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "59fabb07-97fc-4515-93c4-1bac3f908a7c" (UID: "59fabb07-97fc-4515-93c4-1bac3f908a7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:03:56.104561 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.104431    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59fabb07-97fc-4515-93c4-1bac3f908a7c-kube-api-access-984mt" (OuterVolumeSpecName: "kube-api-access-984mt") pod "59fabb07-97fc-4515-93c4-1bac3f908a7c" (UID: "59fabb07-97fc-4515-93c4-1bac3f908a7c"). InnerVolumeSpecName "kube-api-access-984mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:03:56.104561 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.104489    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59fabb07-97fc-4515-93c4-1bac3f908a7c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "59fabb07-97fc-4515-93c4-1bac3f908a7c" (UID: "59fabb07-97fc-4515-93c4-1bac3f908a7c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:03:56.105553 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.105527    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59fabb07-97fc-4515-93c4-1bac3f908a7c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "59fabb07-97fc-4515-93c4-1bac3f908a7c" (UID: "59fabb07-97fc-4515-93c4-1bac3f908a7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:03:56.138390 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.138356    2556 generic.go:356] "Generic (PLEG): container finished" podID="59fabb07-97fc-4515-93c4-1bac3f908a7c" containerID="ad6d251f68eb7dc7c92e494eaa2d799f59e58115acdb581e075f83a6ca19e012" exitCode=0
Sep 16 17:03:56.138589 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.138398    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"
Sep 16 17:03:56.138589 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.138443    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8" event={"ID":"59fabb07-97fc-4515-93c4-1bac3f908a7c","Type":"ContainerDied","Data":"ad6d251f68eb7dc7c92e494eaa2d799f59e58115acdb581e075f83a6ca19e012"}
Sep 16 17:03:56.138589 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.138501    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8" event={"ID":"59fabb07-97fc-4515-93c4-1bac3f908a7c","Type":"ContainerDied","Data":"b6b4a768ed0c771c91ad67ae79f0b9c6590bbb4fb207283c0e363f0ed1f60478"}
Sep 16 17:03:56.138589 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.138530    2556 scope.go:117] "RemoveContainer" containerID="ad6d251f68eb7dc7c92e494eaa2d799f59e58115acdb581e075f83a6ca19e012"
Sep 16 17:03:56.156628 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.156607    2556 scope.go:117] "RemoveContainer" containerID="ad6d251f68eb7dc7c92e494eaa2d799f59e58115acdb581e075f83a6ca19e012"
Sep 16 17:03:56.156897 ip-10-0-140-120 kubenswrapper[2556]: E0916 17:03:56.156876    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad6d251f68eb7dc7c92e494eaa2d799f59e58115acdb581e075f83a6ca19e012\": container with ID starting with ad6d251f68eb7dc7c92e494eaa2d799f59e58115acdb581e075f83a6ca19e012 not found: ID does not exist" containerID="ad6d251f68eb7dc7c92e494eaa2d799f59e58115acdb581e075f83a6ca19e012"
Sep 16 17:03:56.157008 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.156909    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad6d251f68eb7dc7c92e494eaa2d799f59e58115acdb581e075f83a6ca19e012"} err="failed to get container status \"ad6d251f68eb7dc7c92e494eaa2d799f59e58115acdb581e075f83a6ca19e012\": rpc error: code = NotFound desc = could not find container \"ad6d251f68eb7dc7c92e494eaa2d799f59e58115acdb581e075f83a6ca19e012\": container with ID starting with ad6d251f68eb7dc7c92e494eaa2d799f59e58115acdb581e075f83a6ca19e012 not found: ID does not exist"
Sep 16 17:03:56.169604 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.169569    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"]
Sep 16 17:03:56.173240 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.173215    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9359a3-57d6655b9c-kffj8"]
Sep 16 17:03:56.202616 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.202574    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59fabb07-97fc-4515-93c4-1bac3f908a7c-openshift-service-ca\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:03:56.202616 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.202614    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59fabb07-97fc-4515-93c4-1bac3f908a7c-maas-api-tls\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:03:56.202798 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.202631    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-984mt\" (UniqueName: \"kubernetes.io/projected/59fabb07-97fc-4515-93c4-1bac3f908a7c-kube-api-access-984mt\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:03:56.202798 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.202646    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59fabb07-97fc-4515-93c4-1bac3f908a7c-metrics-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:03:56.848869 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:03:56.848831    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59fabb07-97fc-4515-93c4-1bac3f908a7c" path="/var/lib/kubelet/pods/59fabb07-97fc-4515-93c4-1bac3f908a7c/volumes"
Sep 16 17:04:25.057398 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:25.057364    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"]
Sep 16 17:04:25.057910 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:25.057580    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts" podUID="d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" containerName="istio-proxy" containerID="cri-o://5dab7370e20c4ddbc43de2d7b3eb9fe51825c36b368be3f54d2bbb547da91af7" gracePeriod=30
Sep 16 17:04:30.309690 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.309665    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:04:30.320879 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.320815    2556 generic.go:356] "Generic (PLEG): container finished" podID="d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" containerID="5dab7370e20c4ddbc43de2d7b3eb9fe51825c36b368be3f54d2bbb547da91af7" exitCode=0
Sep 16 17:04:30.320879 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.320859    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"
Sep 16 17:04:30.321079 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.320900    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts" event={"ID":"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b","Type":"ContainerDied","Data":"5dab7370e20c4ddbc43de2d7b3eb9fe51825c36b368be3f54d2bbb547da91af7"}
Sep 16 17:04:30.321079 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.320980    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts" event={"ID":"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b","Type":"ContainerDied","Data":"57a3cf0c3a454c4e33cacd11e38ba0d61cdce708547644d2dad4ee9c2d2936c5"}
Sep 16 17:04:30.321079 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.321010    2556 scope.go:117] "RemoveContainer" containerID="5dab7370e20c4ddbc43de2d7b3eb9fe51825c36b368be3f54d2bbb547da91af7"
Sep 16 17:04:30.329335 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.329319    2556 scope.go:117] "RemoveContainer" containerID="4c8791902a57e409e1b03a2655390dec925042f45195361a06b591e2db5589ca"
Sep 16 17:04:30.339957 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.339898    2556 scope.go:117] "RemoveContainer" containerID="5dab7370e20c4ddbc43de2d7b3eb9fe51825c36b368be3f54d2bbb547da91af7"
Sep 16 17:04:30.340250 ip-10-0-140-120 kubenswrapper[2556]: E0916 17:04:30.340230    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dab7370e20c4ddbc43de2d7b3eb9fe51825c36b368be3f54d2bbb547da91af7\": container with ID starting with 5dab7370e20c4ddbc43de2d7b3eb9fe51825c36b368be3f54d2bbb547da91af7 not found: ID does not exist" containerID="5dab7370e20c4ddbc43de2d7b3eb9fe51825c36b368be3f54d2bbb547da91af7"
Sep 16 17:04:30.340327 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.340260    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dab7370e20c4ddbc43de2d7b3eb9fe51825c36b368be3f54d2bbb547da91af7"} err="failed to get container status \"5dab7370e20c4ddbc43de2d7b3eb9fe51825c36b368be3f54d2bbb547da91af7\": rpc error: code = NotFound desc = could not find container \"5dab7370e20c4ddbc43de2d7b3eb9fe51825c36b368be3f54d2bbb547da91af7\": container with ID starting with 5dab7370e20c4ddbc43de2d7b3eb9fe51825c36b368be3f54d2bbb547da91af7 not found: ID does not exist"
Sep 16 17:04:30.340327 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.340279    2556 scope.go:117] "RemoveContainer" containerID="4c8791902a57e409e1b03a2655390dec925042f45195361a06b591e2db5589ca"
Sep 16 17:04:30.340593 ip-10-0-140-120 kubenswrapper[2556]: E0916 17:04:30.340565    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c8791902a57e409e1b03a2655390dec925042f45195361a06b591e2db5589ca\": container with ID starting with 4c8791902a57e409e1b03a2655390dec925042f45195361a06b591e2db5589ca not found: ID does not exist" containerID="4c8791902a57e409e1b03a2655390dec925042f45195361a06b591e2db5589ca"
Sep 16 17:04:30.340695 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.340602    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c8791902a57e409e1b03a2655390dec925042f45195361a06b591e2db5589ca"} err="failed to get container status \"4c8791902a57e409e1b03a2655390dec925042f45195361a06b591e2db5589ca\": rpc error: code = NotFound desc = could not find container \"4c8791902a57e409e1b03a2655390dec925042f45195361a06b591e2db5589ca\": container with ID starting with 4c8791902a57e409e1b03a2655390dec925042f45195361a06b591e2db5589ca not found: ID does not exist"
Sep 16 17:04:30.427843 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.427817    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-workload-socket\") pod \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") "
Sep 16 17:04:30.428001 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.427857    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istiod-ca-cert\") pod \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") "
Sep 16 17:04:30.428001 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.427885    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-credential-socket\") pod \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") "
Sep 16 17:04:30.428001 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.427973    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-workload-certs\") pod \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") "
Sep 16 17:04:30.428001 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.427999    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-envoy\") pod \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") "
Sep 16 17:04:30.428232 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.428025    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-data\") pod \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") "
Sep 16 17:04:30.428232 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.428060    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-token\") pod \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") "
Sep 16 17:04:30.428232 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.428086    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxxrt\" (UniqueName: \"kubernetes.io/projected/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-kube-api-access-nxxrt\") pod \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") "
Sep 16 17:04:30.428232 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.428092    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" (UID: "d88c5c24-2795-4143-b5ef-dadfbb0e4c0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:04:30.428232 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.428115    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-podinfo\") pod \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\" (UID: \"d88c5c24-2795-4143-b5ef-dadfbb0e4c0b\") "
Sep 16 17:04:30.428487 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.428260    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" (UID: "d88c5c24-2795-4143-b5ef-dadfbb0e4c0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:04:30.428487 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.428302    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" (UID: "d88c5c24-2795-4143-b5ef-dadfbb0e4c0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:04:30.428487 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.428353    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-data" (OuterVolumeSpecName: "istio-data") pod "d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" (UID: "d88c5c24-2795-4143-b5ef-dadfbb0e4c0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:04:30.428487 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.428409    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" (UID: "d88c5c24-2795-4143-b5ef-dadfbb0e4c0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:04:30.428487 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.428426    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-workload-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:30.428487 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.428446    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-workload-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:30.428487 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.428463    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istiod-ca-cert\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:30.430179 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.430146    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" (UID: "d88c5c24-2795-4143-b5ef-dadfbb0e4c0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:04:30.430322 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.430299    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-kube-api-access-nxxrt" (OuterVolumeSpecName: "kube-api-access-nxxrt") pod "d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" (UID: "d88c5c24-2795-4143-b5ef-dadfbb0e4c0b"). InnerVolumeSpecName "kube-api-access-nxxrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:04:30.430482 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.430462    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" (UID: "d88c5c24-2795-4143-b5ef-dadfbb0e4c0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 17:04:30.430537 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.430520    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-token" (OuterVolumeSpecName: "istio-token") pod "d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" (UID: "d88c5c24-2795-4143-b5ef-dadfbb0e4c0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:04:30.529121 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.529095    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-data\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:30.529121 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.529117    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-token\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:30.529260 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.529127    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxxrt\" (UniqueName: \"kubernetes.io/projected/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-kube-api-access-nxxrt\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:30.529260 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.529136    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-podinfo\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:30.529260 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.529145    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-credential-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:30.529260 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.529153    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b-istio-envoy\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:30.644813 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.644778    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"]
Sep 16 17:04:30.649617 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.649591    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4acea1-openshift-default-645987b79b-6kwts"]
Sep 16 17:04:30.850101 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:30.850059    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" path="/var/lib/kubelet/pods/d88c5c24-2795-4143-b5ef-dadfbb0e4c0b/volumes"
Sep 16 17:04:36.035425 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.035357    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"]
Sep 16 17:04:36.036210 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.036148    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2" podUID="7543b8b9-2f41-484f-9e31-5f695467893c" containerName="maas-api" containerID="cri-o://248e465c292b99fbb52c51a6b76894e458a174d1c5591e6b425e0e6992237e31" gracePeriod=30
Sep 16 17:04:36.289081 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.289023    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"
Sep 16 17:04:36.351159 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.351125    2556 generic.go:356] "Generic (PLEG): container finished" podID="7543b8b9-2f41-484f-9e31-5f695467893c" containerID="248e465c292b99fbb52c51a6b76894e458a174d1c5591e6b425e0e6992237e31" exitCode=0
Sep 16 17:04:36.351323 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.351167    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"
Sep 16 17:04:36.351323 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.351208    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2" event={"ID":"7543b8b9-2f41-484f-9e31-5f695467893c","Type":"ContainerDied","Data":"248e465c292b99fbb52c51a6b76894e458a174d1c5591e6b425e0e6992237e31"}
Sep 16 17:04:36.351323 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.351265    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2" event={"ID":"7543b8b9-2f41-484f-9e31-5f695467893c","Type":"ContainerDied","Data":"9be0dd372b0996b7f7cacb620b50d20822d34bd349a10b12a4104f07472dfd77"}
Sep 16 17:04:36.351323 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.351293    2556 scope.go:117] "RemoveContainer" containerID="248e465c292b99fbb52c51a6b76894e458a174d1c5591e6b425e0e6992237e31"
Sep 16 17:04:36.359778 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.359767    2556 scope.go:117] "RemoveContainer" containerID="248e465c292b99fbb52c51a6b76894e458a174d1c5591e6b425e0e6992237e31"
Sep 16 17:04:36.360081 ip-10-0-140-120 kubenswrapper[2556]: E0916 17:04:36.360061    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"248e465c292b99fbb52c51a6b76894e458a174d1c5591e6b425e0e6992237e31\": container with ID starting with 248e465c292b99fbb52c51a6b76894e458a174d1c5591e6b425e0e6992237e31 not found: ID does not exist" containerID="248e465c292b99fbb52c51a6b76894e458a174d1c5591e6b425e0e6992237e31"
Sep 16 17:04:36.360160 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.360090    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"248e465c292b99fbb52c51a6b76894e458a174d1c5591e6b425e0e6992237e31"} err="failed to get container status \"248e465c292b99fbb52c51a6b76894e458a174d1c5591e6b425e0e6992237e31\": rpc error: code = NotFound desc = could not find container \"248e465c292b99fbb52c51a6b76894e458a174d1c5591e6b425e0e6992237e31\": container with ID starting with 248e465c292b99fbb52c51a6b76894e458a174d1c5591e6b425e0e6992237e31 not found: ID does not exist"
Sep 16 17:04:36.383005 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.382984    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8kwf\" (UniqueName: \"kubernetes.io/projected/7543b8b9-2f41-484f-9e31-5f695467893c-kube-api-access-n8kwf\") pod \"7543b8b9-2f41-484f-9e31-5f695467893c\" (UID: \"7543b8b9-2f41-484f-9e31-5f695467893c\") "
Sep 16 17:04:36.383085 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.383016    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7543b8b9-2f41-484f-9e31-5f695467893c-metrics-certs\") pod \"7543b8b9-2f41-484f-9e31-5f695467893c\" (UID: \"7543b8b9-2f41-484f-9e31-5f695467893c\") "
Sep 16 17:04:36.383085 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.383068    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7543b8b9-2f41-484f-9e31-5f695467893c-openshift-service-ca\") pod \"7543b8b9-2f41-484f-9e31-5f695467893c\" (UID: \"7543b8b9-2f41-484f-9e31-5f695467893c\") "
Sep 16 17:04:36.383153 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.383104    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7543b8b9-2f41-484f-9e31-5f695467893c-maas-api-tls\") pod \"7543b8b9-2f41-484f-9e31-5f695467893c\" (UID: \"7543b8b9-2f41-484f-9e31-5f695467893c\") "
Sep 16 17:04:36.383458 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.383429    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7543b8b9-2f41-484f-9e31-5f695467893c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7543b8b9-2f41-484f-9e31-5f695467893c" (UID: "7543b8b9-2f41-484f-9e31-5f695467893c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:04:36.385150 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.385125    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7543b8b9-2f41-484f-9e31-5f695467893c-kube-api-access-n8kwf" (OuterVolumeSpecName: "kube-api-access-n8kwf") pod "7543b8b9-2f41-484f-9e31-5f695467893c" (UID: "7543b8b9-2f41-484f-9e31-5f695467893c"). InnerVolumeSpecName "kube-api-access-n8kwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:04:36.385235 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.385153    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7543b8b9-2f41-484f-9e31-5f695467893c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7543b8b9-2f41-484f-9e31-5f695467893c" (UID: "7543b8b9-2f41-484f-9e31-5f695467893c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:04:36.385235 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.385164    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7543b8b9-2f41-484f-9e31-5f695467893c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7543b8b9-2f41-484f-9e31-5f695467893c" (UID: "7543b8b9-2f41-484f-9e31-5f695467893c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:04:36.484748 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.484708    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8kwf\" (UniqueName: \"kubernetes.io/projected/7543b8b9-2f41-484f-9e31-5f695467893c-kube-api-access-n8kwf\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:36.484748 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.484739    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7543b8b9-2f41-484f-9e31-5f695467893c-metrics-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:36.484748 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.484748    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7543b8b9-2f41-484f-9e31-5f695467893c-openshift-service-ca\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:36.484748 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.484756    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7543b8b9-2f41-484f-9e31-5f695467893c-maas-api-tls\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:36.672395 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.672363    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"]
Sep 16 17:04:36.676597 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.676570    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f315c3-fbcdb659-zt6z2"]
Sep 16 17:04:36.850166 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:36.850134    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7543b8b9-2f41-484f-9e31-5f695467893c" path="/var/lib/kubelet/pods/7543b8b9-2f41-484f-9e31-5f695467893c/volumes"
Sep 16 17:04:43.274869 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:43.274826    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"]
Sep 16 17:04:43.275301 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:43.275093    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn" podUID="f20909e3-9f17-4875-a2cd-2246087de774" containerName="istio-proxy" containerID="cri-o://8425da95efb86ca8bea235e68e4007d30d090009ab56adc4ff23dcda56f15407" gracePeriod=30
Sep 16 17:04:48.415554 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.415523    2556 generic.go:356] "Generic (PLEG): container finished" podID="f20909e3-9f17-4875-a2cd-2246087de774" containerID="8425da95efb86ca8bea235e68e4007d30d090009ab56adc4ff23dcda56f15407" exitCode=0
Sep 16 17:04:48.415930 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.415600    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn" event={"ID":"f20909e3-9f17-4875-a2cd-2246087de774","Type":"ContainerDied","Data":"8425da95efb86ca8bea235e68e4007d30d090009ab56adc4ff23dcda56f15407"}
Sep 16 17:04:48.415930 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.415664    2556 scope.go:117] "RemoveContainer" containerID="1b769886c4a5bada52d32c39dd4529115036fd7fa980818673a5c2e906efe282"
Sep 16 17:04:48.526467 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.526445    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 17:04:48.594599 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.594525    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-workload-certs\") pod \"f20909e3-9f17-4875-a2cd-2246087de774\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") "
Sep 16 17:04:48.594599 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.594563    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-workload-socket\") pod \"f20909e3-9f17-4875-a2cd-2246087de774\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") "
Sep 16 17:04:48.594599 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.594591    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f20909e3-9f17-4875-a2cd-2246087de774-istiod-ca-cert\") pod \"f20909e3-9f17-4875-a2cd-2246087de774\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") "
Sep 16 17:04:48.594863 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.594607    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f20909e3-9f17-4875-a2cd-2246087de774-istio-token\") pod \"f20909e3-9f17-4875-a2cd-2246087de774\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") "
Sep 16 17:04:48.594863 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.594624    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-credential-socket\") pod \"f20909e3-9f17-4875-a2cd-2246087de774\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") "
Sep 16 17:04:48.594863 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.594643    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f20909e3-9f17-4875-a2cd-2246087de774-istio-podinfo\") pod \"f20909e3-9f17-4875-a2cd-2246087de774\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") "
Sep 16 17:04:48.594863 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.594661    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-istio-data\") pod \"f20909e3-9f17-4875-a2cd-2246087de774\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") "
Sep 16 17:04:48.594863 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.594691    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z996g\" (UniqueName: \"kubernetes.io/projected/f20909e3-9f17-4875-a2cd-2246087de774-kube-api-access-z996g\") pod \"f20909e3-9f17-4875-a2cd-2246087de774\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") "
Sep 16 17:04:48.594863 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.594720    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-istio-envoy\") pod \"f20909e3-9f17-4875-a2cd-2246087de774\" (UID: \"f20909e3-9f17-4875-a2cd-2246087de774\") "
Sep 16 17:04:48.594863 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.594800    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f20909e3-9f17-4875-a2cd-2246087de774" (UID: "f20909e3-9f17-4875-a2cd-2246087de774"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:04:48.594863 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.594839    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f20909e3-9f17-4875-a2cd-2246087de774" (UID: "f20909e3-9f17-4875-a2cd-2246087de774"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:04:48.595419 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.594998    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-istio-data" (OuterVolumeSpecName: "istio-data") pod "f20909e3-9f17-4875-a2cd-2246087de774" (UID: "f20909e3-9f17-4875-a2cd-2246087de774"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:04:48.595419 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.595207    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-workload-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:48.595419 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.595234    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-workload-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:48.595419 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.595249    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-istio-data\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:48.595419 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.595364    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20909e3-9f17-4875-a2cd-2246087de774-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f20909e3-9f17-4875-a2cd-2246087de774" (UID: "f20909e3-9f17-4875-a2cd-2246087de774"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:04:48.595419 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.595186    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f20909e3-9f17-4875-a2cd-2246087de774" (UID: "f20909e3-9f17-4875-a2cd-2246087de774"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:04:48.597092 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.597068    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f20909e3-9f17-4875-a2cd-2246087de774" (UID: "f20909e3-9f17-4875-a2cd-2246087de774"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:04:48.597209 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.597141    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f20909e3-9f17-4875-a2cd-2246087de774-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f20909e3-9f17-4875-a2cd-2246087de774" (UID: "f20909e3-9f17-4875-a2cd-2246087de774"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 17:04:48.597294 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.597278    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20909e3-9f17-4875-a2cd-2246087de774-istio-token" (OuterVolumeSpecName: "istio-token") pod "f20909e3-9f17-4875-a2cd-2246087de774" (UID: "f20909e3-9f17-4875-a2cd-2246087de774"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:04:48.597388 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.597368    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20909e3-9f17-4875-a2cd-2246087de774-kube-api-access-z996g" (OuterVolumeSpecName: "kube-api-access-z996g") pod "f20909e3-9f17-4875-a2cd-2246087de774" (UID: "f20909e3-9f17-4875-a2cd-2246087de774"). InnerVolumeSpecName "kube-api-access-z996g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:04:48.695675 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.695650    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f20909e3-9f17-4875-a2cd-2246087de774-istiod-ca-cert\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:48.695675 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.695673    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f20909e3-9f17-4875-a2cd-2246087de774-istio-token\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:48.695822 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.695681    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-credential-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:48.695822 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.695690    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f20909e3-9f17-4875-a2cd-2246087de774-istio-podinfo\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:48.695822 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.695698    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z996g\" (UniqueName: \"kubernetes.io/projected/f20909e3-9f17-4875-a2cd-2246087de774-kube-api-access-z996g\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:48.695822 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:48.695707    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f20909e3-9f17-4875-a2cd-2246087de774-istio-envoy\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:49.422486 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:49.422457    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"
Sep 16 17:04:49.422978 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:49.422490    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn" event={"ID":"f20909e3-9f17-4875-a2cd-2246087de774","Type":"ContainerDied","Data":"06fbf1e98cc2b46ad2dca7faecaae4e0fc653d704d26576f967da2cb800e3e6b"}
Sep 16 17:04:49.422978 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:49.422554    2556 scope.go:117] "RemoveContainer" containerID="8425da95efb86ca8bea235e68e4007d30d090009ab56adc4ff23dcda56f15407"
Sep 16 17:04:49.440964 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:49.440923    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"]
Sep 16 17:04:49.446346 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:49.446324    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f315c3-openshift-default-7565b74f48-w7rnn"]
Sep 16 17:04:50.849487 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:50.849453    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f20909e3-9f17-4875-a2cd-2246087de774" path="/var/lib/kubelet/pods/f20909e3-9f17-4875-a2cd-2246087de774/volumes"
Sep 16 17:04:54.210427 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.210384    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"]
Sep 16 17:04:54.210837 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.210628    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw" podUID="81ac47a1-7620-4932-bc86-045295332f13" containerName="maas-api" containerID="cri-o://2ea5d3347646872b603e80919b8912adbf0e10625799511e6cca5fd077cf5fd9" gracePeriod=30
Sep 16 17:04:54.274756 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.274724    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh"]
Sep 16 17:04:54.274989 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.274923    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh" podUID="1c06f327-248b-48cb-a75e-3a3e57f43162" containerName="payload-pre-processing" containerID="cri-o://0242662710bc041b8ba8c4e1701bb3a9aed0b1753927f2f5d9d88eb53f560661" gracePeriod=30
Sep 16 17:04:54.452641 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.452494    2556 generic.go:356] "Generic (PLEG): container finished" podID="81ac47a1-7620-4932-bc86-045295332f13" containerID="2ea5d3347646872b603e80919b8912adbf0e10625799511e6cca5fd077cf5fd9" exitCode=0
Sep 16 17:04:54.452641 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.452572    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw" event={"ID":"81ac47a1-7620-4932-bc86-045295332f13","Type":"ContainerDied","Data":"2ea5d3347646872b603e80919b8912adbf0e10625799511e6cca5fd077cf5fd9"}
Sep 16 17:04:54.454843 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.454816    2556 generic.go:356] "Generic (PLEG): container finished" podID="1c06f327-248b-48cb-a75e-3a3e57f43162" containerID="0242662710bc041b8ba8c4e1701bb3a9aed0b1753927f2f5d9d88eb53f560661" exitCode=0
Sep 16 17:04:54.455014 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.454901    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh" event={"ID":"1c06f327-248b-48cb-a75e-3a3e57f43162","Type":"ContainerDied","Data":"0242662710bc041b8ba8c4e1701bb3a9aed0b1753927f2f5d9d88eb53f560661"}
Sep 16 17:04:54.521789 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.521764    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"
Sep 16 17:04:54.538657 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.538635    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81ac47a1-7620-4932-bc86-045295332f13-metrics-certs\") pod \"81ac47a1-7620-4932-bc86-045295332f13\" (UID: \"81ac47a1-7620-4932-bc86-045295332f13\") "
Sep 16 17:04:54.538787 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.538682    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trvcr\" (UniqueName: \"kubernetes.io/projected/81ac47a1-7620-4932-bc86-045295332f13-kube-api-access-trvcr\") pod \"81ac47a1-7620-4932-bc86-045295332f13\" (UID: \"81ac47a1-7620-4932-bc86-045295332f13\") "
Sep 16 17:04:54.538787 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.538702    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81ac47a1-7620-4932-bc86-045295332f13-maas-api-tls\") pod \"81ac47a1-7620-4932-bc86-045295332f13\" (UID: \"81ac47a1-7620-4932-bc86-045295332f13\") "
Sep 16 17:04:54.538787 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.538723    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81ac47a1-7620-4932-bc86-045295332f13-openshift-service-ca\") pod \"81ac47a1-7620-4932-bc86-045295332f13\" (UID: \"81ac47a1-7620-4932-bc86-045295332f13\") "
Sep 16 17:04:54.539154 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.539120    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ac47a1-7620-4932-bc86-045295332f13-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "81ac47a1-7620-4932-bc86-045295332f13" (UID: "81ac47a1-7620-4932-bc86-045295332f13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:04:54.540960 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.540916    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81ac47a1-7620-4932-bc86-045295332f13-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "81ac47a1-7620-4932-bc86-045295332f13" (UID: "81ac47a1-7620-4932-bc86-045295332f13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:04:54.541481 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.541325    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81ac47a1-7620-4932-bc86-045295332f13-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "81ac47a1-7620-4932-bc86-045295332f13" (UID: "81ac47a1-7620-4932-bc86-045295332f13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:04:54.541481 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.541337    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ac47a1-7620-4932-bc86-045295332f13-kube-api-access-trvcr" (OuterVolumeSpecName: "kube-api-access-trvcr") pod "81ac47a1-7620-4932-bc86-045295332f13" (UID: "81ac47a1-7620-4932-bc86-045295332f13"). InnerVolumeSpecName "kube-api-access-trvcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:04:54.552559 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.552534    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh"
Sep 16 17:04:54.640040 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.640009    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwv95\" (UniqueName: \"kubernetes.io/projected/1c06f327-248b-48cb-a75e-3a3e57f43162-kube-api-access-nwv95\") pod \"1c06f327-248b-48cb-a75e-3a3e57f43162\" (UID: \"1c06f327-248b-48cb-a75e-3a3e57f43162\") "
Sep 16 17:04:54.640205 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.640062    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c06f327-248b-48cb-a75e-3a3e57f43162-plugins-config-volume\") pod \"1c06f327-248b-48cb-a75e-3a3e57f43162\" (UID: \"1c06f327-248b-48cb-a75e-3a3e57f43162\") "
Sep 16 17:04:54.640264 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.640247    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trvcr\" (UniqueName: \"kubernetes.io/projected/81ac47a1-7620-4932-bc86-045295332f13-kube-api-access-trvcr\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:54.640302 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.640271    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/81ac47a1-7620-4932-bc86-045295332f13-maas-api-tls\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:54.640302 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.640286    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/81ac47a1-7620-4932-bc86-045295332f13-openshift-service-ca\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:54.640370 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.640299    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81ac47a1-7620-4932-bc86-045295332f13-metrics-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:54.640478 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.640457    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c06f327-248b-48cb-a75e-3a3e57f43162-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c06f327-248b-48cb-a75e-3a3e57f43162" (UID: "1c06f327-248b-48cb-a75e-3a3e57f43162"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:04:54.642000 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.641979    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c06f327-248b-48cb-a75e-3a3e57f43162-kube-api-access-nwv95" (OuterVolumeSpecName: "kube-api-access-nwv95") pod "1c06f327-248b-48cb-a75e-3a3e57f43162" (UID: "1c06f327-248b-48cb-a75e-3a3e57f43162"). InnerVolumeSpecName "kube-api-access-nwv95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:04:54.741744 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.741669    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwv95\" (UniqueName: \"kubernetes.io/projected/1c06f327-248b-48cb-a75e-3a3e57f43162-kube-api-access-nwv95\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:54.741744 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:54.741694    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c06f327-248b-48cb-a75e-3a3e57f43162-plugins-config-volume\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:04:55.461598 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:55.461569    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh"
Sep 16 17:04:55.462058 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:55.461607    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh" event={"ID":"1c06f327-248b-48cb-a75e-3a3e57f43162","Type":"ContainerDied","Data":"870cd38307e9fec5434973f89b53163501d04d228e5bc35835104a480d0806c3"}
Sep 16 17:04:55.462058 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:55.461672    2556 scope.go:117] "RemoveContainer" containerID="0242662710bc041b8ba8c4e1701bb3a9aed0b1753927f2f5d9d88eb53f560661"
Sep 16 17:04:55.463242 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:55.463201    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw" event={"ID":"81ac47a1-7620-4932-bc86-045295332f13","Type":"ContainerDied","Data":"7df09024ffa96f4539ac33cdb626ea96c582b22e8eb4189b9bca4a3a87dcbc12"}
Sep 16 17:04:55.463242 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:55.463232    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"
Sep 16 17:04:55.470503 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:55.470490    2556 scope.go:117] "RemoveContainer" containerID="2ea5d3347646872b603e80919b8912adbf0e10625799511e6cca5fd077cf5fd9"
Sep 16 17:04:55.483056 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:55.482953    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"]
Sep 16 17:04:55.484989 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:55.484968    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-2ce8bc-6b8879f76b-vs6nw"]
Sep 16 17:04:55.498729 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:55.498705    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh"]
Sep 16 17:04:55.508615 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:55.508590    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-2ce8bc-858cccd9b6-mv4kh"]
Sep 16 17:04:56.848436 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:56.848399    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c06f327-248b-48cb-a75e-3a3e57f43162" path="/var/lib/kubelet/pods/1c06f327-248b-48cb-a75e-3a3e57f43162/volumes"
Sep 16 17:04:56.848874 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:56.848786    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81ac47a1-7620-4932-bc86-045295332f13" path="/var/lib/kubelet/pods/81ac47a1-7620-4932-bc86-045295332f13/volumes"
Sep 16 17:04:57.118460 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:57.118385    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"]
Sep 16 17:04:58.928796 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.928763    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"]
Sep 16 17:04:58.929190 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929178    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20909e3-9f17-4875-a2cd-2246087de774" containerName="istio-proxy"
Sep 16 17:04:58.929233 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929193    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20909e3-9f17-4875-a2cd-2246087de774" containerName="istio-proxy"
Sep 16 17:04:58.929233 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929202    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81ac47a1-7620-4932-bc86-045295332f13" containerName="maas-api"
Sep 16 17:04:58.929233 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929208    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="81ac47a1-7620-4932-bc86-045295332f13" containerName="maas-api"
Sep 16 17:04:58.929233 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929215    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" containerName="istio-proxy"
Sep 16 17:04:58.929233 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929220    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" containerName="istio-proxy"
Sep 16 17:04:58.929233 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929233    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59fabb07-97fc-4515-93c4-1bac3f908a7c" containerName="maas-api"
Sep 16 17:04:58.929416 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929238    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="59fabb07-97fc-4515-93c4-1bac3f908a7c" containerName="maas-api"
Sep 16 17:04:58.929416 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929249    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f20909e3-9f17-4875-a2cd-2246087de774" containerName="istio-proxy"
Sep 16 17:04:58.929416 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929253    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f20909e3-9f17-4875-a2cd-2246087de774" containerName="istio-proxy"
Sep 16 17:04:58.929416 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929260    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7543b8b9-2f41-484f-9e31-5f695467893c" containerName="maas-api"
Sep 16 17:04:58.929416 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929265    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="7543b8b9-2f41-484f-9e31-5f695467893c" containerName="maas-api"
Sep 16 17:04:58.929416 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929271    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c06f327-248b-48cb-a75e-3a3e57f43162" containerName="payload-pre-processing"
Sep 16 17:04:58.929416 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929276    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c06f327-248b-48cb-a75e-3a3e57f43162" containerName="payload-pre-processing"
Sep 16 17:04:58.929416 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929282    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" containerName="istio-proxy"
Sep 16 17:04:58.929416 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929287    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" containerName="istio-proxy"
Sep 16 17:04:58.929416 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929345    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f20909e3-9f17-4875-a2cd-2246087de774" containerName="istio-proxy"
Sep 16 17:04:58.929416 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929355    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="59fabb07-97fc-4515-93c4-1bac3f908a7c" containerName="maas-api"
Sep 16 17:04:58.929416 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929362    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" containerName="istio-proxy"
Sep 16 17:04:58.929416 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929369    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="81ac47a1-7620-4932-bc86-045295332f13" containerName="maas-api"
Sep 16 17:04:58.929416 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929375    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="7543b8b9-2f41-484f-9e31-5f695467893c" containerName="maas-api"
Sep 16 17:04:58.929416 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929380    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d88c5c24-2795-4143-b5ef-dadfbb0e4c0b" containerName="istio-proxy"
Sep 16 17:04:58.929416 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.929385    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c06f327-248b-48cb-a75e-3a3e57f43162" containerName="payload-pre-processing"
Sep 16 17:04:58.933933 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.933916    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:58.936490 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.936468    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-9ac315-openshift-default-dockercfg-6jwkl\""
Sep 16 17:04:58.945284 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.945260    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"]
Sep 16 17:04:58.978769 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.978744    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62plr\" (UniqueName: \"kubernetes.io/projected/e00bc8c1-aef0-45c2-a034-0ad3fe638892-kube-api-access-62plr\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:58.978911 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.978776    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-workload-socket\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:58.978911 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.978796    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istiod-ca-cert\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:58.978911 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.978825    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-workload-certs\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:58.978911 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.978876    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-envoy\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:58.979106 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.978911    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-token\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:58.979106 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.978983    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-credential-socket\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:58.979106 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.979033    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-podinfo\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:58.979106 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:58.979053    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-data\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:59.080240 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.080197    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-envoy\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:59.080412 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.080250    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-token\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:59.080412 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.080316    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-credential-socket\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:59.080412 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.080393    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-podinfo\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:59.080585 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.080427    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-data\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:59.080585 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.080472    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62plr\" (UniqueName: \"kubernetes.io/projected/e00bc8c1-aef0-45c2-a034-0ad3fe638892-kube-api-access-62plr\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:59.080585 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.080498    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-workload-socket\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:59.080585 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.080536    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istiod-ca-cert\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:59.080585 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.080581    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-workload-certs\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:59.080846 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.080786    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-credential-socket\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:59.081061 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.081038    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-workload-socket\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:59.081241 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.081097    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-data\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:59.081241 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.081151    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-workload-certs\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:59.081410 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.081346    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istiod-ca-cert\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:59.083180 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.083154    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-envoy\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:59.084125 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.084098    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-podinfo\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:59.089342 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.089317    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-token\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:59.089456 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.089439    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62plr\" (UniqueName: \"kubernetes.io/projected/e00bc8c1-aef0-45c2-a034-0ad3fe638892-kube-api-access-62plr\") pod \"e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") " pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:59.244884 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.244850    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:04:59.525338 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.525312    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"]
Sep 16 17:04:59.527288 ip-10-0-140-120 kubenswrapper[2556]: W0916 17:04:59.527261    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode00bc8c1_aef0_45c2_a034_0ad3fe638892.slice/crio-eae41b04d0ae8f976e973335a11a6ff86f55fcca30d6b83ff9a2d8f0cfc467ed WatchSource:0}: Error finding container eae41b04d0ae8f976e973335a11a6ff86f55fcca30d6b83ff9a2d8f0cfc467ed: Status 404 returned error can't find the container with id eae41b04d0ae8f976e973335a11a6ff86f55fcca30d6b83ff9a2d8f0cfc467ed
Sep 16 17:04:59.529796 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.529768    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 17:04:59.529843 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.529829    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 17:04:59.529885 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:04:59.529861    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 17:05:00.490847 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:00.490812    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9" event={"ID":"e00bc8c1-aef0-45c2-a034-0ad3fe638892","Type":"ContainerStarted","Data":"74eb4ea0a169b69df10638998eba744eb284266b22289e22fb4dddc7c2308aff"}
Sep 16 17:05:00.491234 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:00.490864    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9" event={"ID":"e00bc8c1-aef0-45c2-a034-0ad3fe638892","Type":"ContainerStarted","Data":"eae41b04d0ae8f976e973335a11a6ff86f55fcca30d6b83ff9a2d8f0cfc467ed"}
Sep 16 17:05:00.516820 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:00.516776    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9" podStartSLOduration=2.516761061 podStartE2EDuration="2.516761061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:05:00.513846741 +0000 UTC m=+1700.275808188" watchObservedRunningTime="2026-09-16 17:05:00.516761061 +0000 UTC m=+1700.278722571"
Sep 16 17:05:01.246070 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:01.246037    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:05:01.271993 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:01.271681    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:05:01.499733 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:01.499643    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:05:01.503128 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:01.503102    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:05:04.277628 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:04.277593    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"]
Sep 16 17:05:04.278276 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:04.278255    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="f20909e3-9f17-4875-a2cd-2246087de774" containerName="istio-proxy"
Sep 16 17:05:04.281125 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:04.281105    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"
Sep 16 17:05:04.284708 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:04.284690    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-9ac315-serving-cert\""
Sep 16 17:05:04.295500 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:04.295478    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"]
Sep 16 17:05:04.332482 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:04.332448    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-978bx\" (UniqueName: \"kubernetes.io/projected/37794aad-c1f0-4d9b-bd68-786dab64dcd4-kube-api-access-978bx\") pod \"maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh\" (UID: \"37794aad-c1f0-4d9b-bd68-786dab64dcd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"
Sep 16 17:05:04.332602 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:04.332488    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37794aad-c1f0-4d9b-bd68-786dab64dcd4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh\" (UID: \"37794aad-c1f0-4d9b-bd68-786dab64dcd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"
Sep 16 17:05:04.332602 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:04.332581    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37794aad-c1f0-4d9b-bd68-786dab64dcd4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh\" (UID: \"37794aad-c1f0-4d9b-bd68-786dab64dcd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"
Sep 16 17:05:04.332692 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:04.332672    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37794aad-c1f0-4d9b-bd68-786dab64dcd4-metrics-certs\") pod \"maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh\" (UID: \"37794aad-c1f0-4d9b-bd68-786dab64dcd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"
Sep 16 17:05:04.433407 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:04.433372    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37794aad-c1f0-4d9b-bd68-786dab64dcd4-metrics-certs\") pod \"maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh\" (UID: \"37794aad-c1f0-4d9b-bd68-786dab64dcd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"
Sep 16 17:05:04.433407 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:04.433411    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-978bx\" (UniqueName: \"kubernetes.io/projected/37794aad-c1f0-4d9b-bd68-786dab64dcd4-kube-api-access-978bx\") pod \"maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh\" (UID: \"37794aad-c1f0-4d9b-bd68-786dab64dcd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"
Sep 16 17:05:04.433643 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:04.433436    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37794aad-c1f0-4d9b-bd68-786dab64dcd4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh\" (UID: \"37794aad-c1f0-4d9b-bd68-786dab64dcd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"
Sep 16 17:05:04.433643 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:04.433481    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37794aad-c1f0-4d9b-bd68-786dab64dcd4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh\" (UID: \"37794aad-c1f0-4d9b-bd68-786dab64dcd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"
Sep 16 17:05:04.434197 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:04.434170    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37794aad-c1f0-4d9b-bd68-786dab64dcd4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh\" (UID: \"37794aad-c1f0-4d9b-bd68-786dab64dcd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"
Sep 16 17:05:04.435840 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:04.435819    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37794aad-c1f0-4d9b-bd68-786dab64dcd4-metrics-certs\") pod \"maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh\" (UID: \"37794aad-c1f0-4d9b-bd68-786dab64dcd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"
Sep 16 17:05:04.435994 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:04.435972    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37794aad-c1f0-4d9b-bd68-786dab64dcd4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh\" (UID: \"37794aad-c1f0-4d9b-bd68-786dab64dcd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"
Sep 16 17:05:04.441727 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:04.441702    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-978bx\" (UniqueName: \"kubernetes.io/projected/37794aad-c1f0-4d9b-bd68-786dab64dcd4-kube-api-access-978bx\") pod \"maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh\" (UID: \"37794aad-c1f0-4d9b-bd68-786dab64dcd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"
Sep 16 17:05:04.592354 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:04.592264    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"
Sep 16 17:05:04.868482 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:04.868455    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"]
Sep 16 17:05:04.870453 ip-10-0-140-120 kubenswrapper[2556]: W0916 17:05:04.870424    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37794aad_c1f0_4d9b_bd68_786dab64dcd4.slice/crio-ab0de8f942efb909c18f70dffa5954a8188f5d312febf4b6207ab979df0d1b29 WatchSource:0}: Error finding container ab0de8f942efb909c18f70dffa5954a8188f5d312febf4b6207ab979df0d1b29: Status 404 returned error can't find the container with id ab0de8f942efb909c18f70dffa5954a8188f5d312febf4b6207ab979df0d1b29
Sep 16 17:05:04.872000 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:04.871977    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:05:05.520915 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:05.520890    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh" event={"ID":"37794aad-c1f0-4d9b-bd68-786dab64dcd4","Type":"ContainerStarted","Data":"ab0de8f942efb909c18f70dffa5954a8188f5d312febf4b6207ab979df0d1b29"}
Sep 16 17:05:06.528259 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:06.528213    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh" event={"ID":"37794aad-c1f0-4d9b-bd68-786dab64dcd4","Type":"ContainerStarted","Data":"59acc7b58c3877466ccecf6afcee497a49a5a6e751d3a330547d417a1cb63769"}
Sep 16 17:05:06.528730 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:06.528347    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"
Sep 16 17:05:06.546950 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:06.546886    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh" podStartSLOduration=1.954914585 podStartE2EDuration="2.546868869s" totalImagesPullingTime="591.954284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:05:06.545872528 +0000 UTC m=+1706.307833975" watchObservedRunningTime="2026-09-16 17:05:06.546868869 +0000 UTC m=+1706.308830316"
Sep 16 17:05:12.136725 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.136675    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk" podUID="4134341c-cc5e-4a57-99ef-901eafc3b78a" containerName="main" containerID="cri-o://02fbb12e474dbdff239ce2d338de4e8d4dcaad1da140cc01f8e34b854023249d" gracePeriod=45
Sep 16 17:05:12.417835 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.417812    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:05:12.513396 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.513353    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-model-cache\") pod \"4134341c-cc5e-4a57-99ef-901eafc3b78a\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") "
Sep 16 17:05:12.513562 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.513439    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jbng\" (UniqueName: \"kubernetes.io/projected/4134341c-cc5e-4a57-99ef-901eafc3b78a-kube-api-access-6jbng\") pod \"4134341c-cc5e-4a57-99ef-901eafc3b78a\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") "
Sep 16 17:05:12.513562 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.513470    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-home\") pod \"4134341c-cc5e-4a57-99ef-901eafc3b78a\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") "
Sep 16 17:05:12.513640 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.513589    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-model-cache" (OuterVolumeSpecName: "model-cache") pod "4134341c-cc5e-4a57-99ef-901eafc3b78a" (UID: "4134341c-cc5e-4a57-99ef-901eafc3b78a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:05:12.513735 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.513714    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-home" (OuterVolumeSpecName: "home") pod "4134341c-cc5e-4a57-99ef-901eafc3b78a" (UID: "4134341c-cc5e-4a57-99ef-901eafc3b78a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:05:12.513844 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.513830    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-dshm\") pod \"4134341c-cc5e-4a57-99ef-901eafc3b78a\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") "
Sep 16 17:05:12.513882 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.513862    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4134341c-cc5e-4a57-99ef-901eafc3b78a-tls-certs\") pod \"4134341c-cc5e-4a57-99ef-901eafc3b78a\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") "
Sep 16 17:05:12.513882 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.513878    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-tmp-dir\") pod \"4134341c-cc5e-4a57-99ef-901eafc3b78a\" (UID: \"4134341c-cc5e-4a57-99ef-901eafc3b78a\") "
Sep 16 17:05:12.514240 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.514136    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4134341c-cc5e-4a57-99ef-901eafc3b78a" (UID: "4134341c-cc5e-4a57-99ef-901eafc3b78a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:05:12.514240 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.514219    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-model-cache\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:05:12.514240 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.514236    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-home\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:05:12.514467 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.514247    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-tmp-dir\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:05:12.515881 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.515861    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4134341c-cc5e-4a57-99ef-901eafc3b78a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4134341c-cc5e-4a57-99ef-901eafc3b78a" (UID: "4134341c-cc5e-4a57-99ef-901eafc3b78a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:05:12.516016 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.515860    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-dshm" (OuterVolumeSpecName: "dshm") pod "4134341c-cc5e-4a57-99ef-901eafc3b78a" (UID: "4134341c-cc5e-4a57-99ef-901eafc3b78a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:05:12.516100 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.516078    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4134341c-cc5e-4a57-99ef-901eafc3b78a-kube-api-access-6jbng" (OuterVolumeSpecName: "kube-api-access-6jbng") pod "4134341c-cc5e-4a57-99ef-901eafc3b78a" (UID: "4134341c-cc5e-4a57-99ef-901eafc3b78a"). InnerVolumeSpecName "kube-api-access-6jbng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:05:12.538798 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.538778    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"
Sep 16 17:05:12.568781 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.568724    2556 generic.go:356] "Generic (PLEG): container finished" podID="4134341c-cc5e-4a57-99ef-901eafc3b78a" containerID="02fbb12e474dbdff239ce2d338de4e8d4dcaad1da140cc01f8e34b854023249d" exitCode=0
Sep 16 17:05:12.568980 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.568805    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk" event={"ID":"4134341c-cc5e-4a57-99ef-901eafc3b78a","Type":"ContainerDied","Data":"02fbb12e474dbdff239ce2d338de4e8d4dcaad1da140cc01f8e34b854023249d"}
Sep 16 17:05:12.568980 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.568842    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk" event={"ID":"4134341c-cc5e-4a57-99ef-901eafc3b78a","Type":"ContainerDied","Data":"0601562dab3f6d862a09d800e5bd77f2c17db4b8b1023baa817b904abf0b8a36"}
Sep 16 17:05:12.568980 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.568864    2556 scope.go:117] "RemoveContainer" containerID="02fbb12e474dbdff239ce2d338de4e8d4dcaad1da140cc01f8e34b854023249d"
Sep 16 17:05:12.569169 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.569057    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"
Sep 16 17:05:12.591476 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.591450    2556 scope.go:117] "RemoveContainer" containerID="02fbb12e474dbdff239ce2d338de4e8d4dcaad1da140cc01f8e34b854023249d"
Sep 16 17:05:12.591857 ip-10-0-140-120 kubenswrapper[2556]: E0916 17:05:12.591812    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02fbb12e474dbdff239ce2d338de4e8d4dcaad1da140cc01f8e34b854023249d\": container with ID starting with 02fbb12e474dbdff239ce2d338de4e8d4dcaad1da140cc01f8e34b854023249d not found: ID does not exist" containerID="02fbb12e474dbdff239ce2d338de4e8d4dcaad1da140cc01f8e34b854023249d"
Sep 16 17:05:12.591965 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.591867    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02fbb12e474dbdff239ce2d338de4e8d4dcaad1da140cc01f8e34b854023249d"} err="failed to get container status \"02fbb12e474dbdff239ce2d338de4e8d4dcaad1da140cc01f8e34b854023249d\": rpc error: code = NotFound desc = could not find container \"02fbb12e474dbdff239ce2d338de4e8d4dcaad1da140cc01f8e34b854023249d\": container with ID starting with 02fbb12e474dbdff239ce2d338de4e8d4dcaad1da140cc01f8e34b854023249d not found: ID does not exist"
Sep 16 17:05:12.603760 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.603736    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"]
Sep 16 17:05:12.609973 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.609927    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-52fd8c/rate-test-model-52fd8c-kserve-67f6cf57c9-jd4rk"]
Sep 16 17:05:12.615319 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.615066    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jbng\" (UniqueName: \"kubernetes.io/projected/4134341c-cc5e-4a57-99ef-901eafc3b78a-kube-api-access-6jbng\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:05:12.615319 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.615099    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4134341c-cc5e-4a57-99ef-901eafc3b78a-dshm\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:05:12.615319 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.615114    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4134341c-cc5e-4a57-99ef-901eafc3b78a-tls-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:05:12.852741 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:12.852692    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4134341c-cc5e-4a57-99ef-901eafc3b78a" path="/var/lib/kubelet/pods/4134341c-cc5e-4a57-99ef-901eafc3b78a/volumes"
Sep 16 17:05:17.543061 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:17.543023    2556 patch_prober.go:28] interesting pod/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 17:05:17.543517 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:17.543101    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9" podUID="e00bc8c1-aef0-45c2-a034-0ad3fe638892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 17:05:18.605207 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:18.605173    2556 generic.go:356] "Generic (PLEG): container finished" podID="e00bc8c1-aef0-45c2-a034-0ad3fe638892" containerID="74eb4ea0a169b69df10638998eba744eb284266b22289e22fb4dddc7c2308aff" exitCode=0
Sep 16 17:05:18.605207 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:18.605211    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9" event={"ID":"e00bc8c1-aef0-45c2-a034-0ad3fe638892","Type":"ContainerDied","Data":"74eb4ea0a169b69df10638998eba744eb284266b22289e22fb4dddc7c2308aff"}
Sep 16 17:05:18.605668 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:18.605570    2556 scope.go:117] "RemoveContainer" containerID="74eb4ea0a169b69df10638998eba744eb284266b22289e22fb4dddc7c2308aff"
Sep 16 17:05:18.606710 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:18.606679    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 17:05:18.606771 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:18.606748    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 17:05:18.606815 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:18.606780    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 17:05:19.612119 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:19.612081    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9" event={"ID":"e00bc8c1-aef0-45c2-a034-0ad3fe638892","Type":"ContainerStarted","Data":"beeac25fdf5fa8ab945d8487a9f104aa0ec2cef5d95c793157dfb838e72288a1"}
Sep 16 17:05:20.245683 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:20.245598    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:05:20.246239 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:20.246206    2556 patch_prober.go:28] interesting pod/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 16 17:05:20.246459 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:20.246430    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9" podUID="e00bc8c1-aef0-45c2-a034-0ad3fe638892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 16 17:05:21.246047 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:21.246003    2556 patch_prober.go:28] interesting pod/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 16 17:05:21.246533 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:21.246079    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9" podUID="e00bc8c1-aef0-45c2-a034-0ad3fe638892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 16 17:05:22.246262 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:22.246215    2556 patch_prober.go:28] interesting pod/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 16 17:05:22.246749 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:22.246320    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9" podUID="e00bc8c1-aef0-45c2-a034-0ad3fe638892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 16 17:05:23.255476 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:23.255432    2556 patch_prober.go:28] interesting pod/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 16 17:05:23.256115 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:23.256081    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9" podUID="e00bc8c1-aef0-45c2-a034-0ad3fe638892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 16 17:05:24.246223 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:24.246179    2556 patch_prober.go:28] interesting pod/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 16 17:05:24.246420 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:24.246248    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9" podUID="e00bc8c1-aef0-45c2-a034-0ad3fe638892" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 16 17:05:25.249648 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:25.249616    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:05:25.250122 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:25.249878    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:05:25.250448 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:25.250429    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:05:35.102696 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:35.102661    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql"]
Sep 16 17:05:35.103200 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:35.103069    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4134341c-cc5e-4a57-99ef-901eafc3b78a" containerName="main"
Sep 16 17:05:35.103200 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:35.103082    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4134341c-cc5e-4a57-99ef-901eafc3b78a" containerName="main"
Sep 16 17:05:35.103200 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:35.103139    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="4134341c-cc5e-4a57-99ef-901eafc3b78a" containerName="main"
Sep 16 17:05:35.105141 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:35.105121    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql"
Sep 16 17:05:35.107254 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:35.107233    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6586c3-dockercfg-hznn7\""
Sep 16 17:05:35.107339 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:35.107278    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6586c3\""
Sep 16 17:05:35.114678 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:35.114654    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql"]
Sep 16 17:05:35.121920 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:35.121898    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67aff6eb-30af-4e42-ae7f-02ac193f1b19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql\" (UID: \"67aff6eb-30af-4e42-ae7f-02ac193f1b19\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql"
Sep 16 17:05:35.122049 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:35.121928    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt9jx\" (UniqueName: \"kubernetes.io/projected/67aff6eb-30af-4e42-ae7f-02ac193f1b19-kube-api-access-rt9jx\") pod \"payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql\" (UID: \"67aff6eb-30af-4e42-ae7f-02ac193f1b19\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql"
Sep 16 17:05:35.222398 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:35.222368    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67aff6eb-30af-4e42-ae7f-02ac193f1b19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql\" (UID: \"67aff6eb-30af-4e42-ae7f-02ac193f1b19\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql"
Sep 16 17:05:35.222398 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:35.222399    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt9jx\" (UniqueName: \"kubernetes.io/projected/67aff6eb-30af-4e42-ae7f-02ac193f1b19-kube-api-access-rt9jx\") pod \"payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql\" (UID: \"67aff6eb-30af-4e42-ae7f-02ac193f1b19\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql"
Sep 16 17:05:35.223129 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:35.223103    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67aff6eb-30af-4e42-ae7f-02ac193f1b19-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql\" (UID: \"67aff6eb-30af-4e42-ae7f-02ac193f1b19\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql"
Sep 16 17:05:35.230740 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:35.230716    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt9jx\" (UniqueName: \"kubernetes.io/projected/67aff6eb-30af-4e42-ae7f-02ac193f1b19-kube-api-access-rt9jx\") pod \"payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql\" (UID: \"67aff6eb-30af-4e42-ae7f-02ac193f1b19\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql"
Sep 16 17:05:35.416158 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:35.416067    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql"
Sep 16 17:05:35.686875 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:35.686842    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql"]
Sep 16 17:05:35.688863 ip-10-0-140-120 kubenswrapper[2556]: W0916 17:05:35.688833    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67aff6eb_30af_4e42_ae7f_02ac193f1b19.slice/crio-1d4f2a06d2d20759c2b63785cda3d7ac153669b8904c1303a6a64fbbf627ae21 WatchSource:0}: Error finding container 1d4f2a06d2d20759c2b63785cda3d7ac153669b8904c1303a6a64fbbf627ae21: Status 404 returned error can't find the container with id 1d4f2a06d2d20759c2b63785cda3d7ac153669b8904c1303a6a64fbbf627ae21
Sep 16 17:05:36.698383 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:36.698345    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql" event={"ID":"67aff6eb-30af-4e42-ae7f-02ac193f1b19","Type":"ContainerStarted","Data":"6e646cdcf64254b1cfa68044c6bcbaabe541675c8f25192dc2d368a3148b7a64"}
Sep 16 17:05:36.698764 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:36.698403    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql" event={"ID":"67aff6eb-30af-4e42-ae7f-02ac193f1b19","Type":"ContainerStarted","Data":"1d4f2a06d2d20759c2b63785cda3d7ac153669b8904c1303a6a64fbbf627ae21"}
Sep 16 17:05:36.698764 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:36.698423    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql"
Sep 16 17:05:36.715633 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:36.715584    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql" podStartSLOduration=1.7155677539999998 podStartE2EDuration="1.715567754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:05:36.713521742 +0000 UTC m=+1736.475483191" watchObservedRunningTime="2026-09-16 17:05:36.715567754 +0000 UTC m=+1736.477529211"
Sep 16 17:05:47.705448 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:05:47.705415    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql"
Sep 16 17:06:06.591547 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:06.591508    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mpcpf_bb67f300-0327-4e84-9998-045e6b646f34/payload-processing/0.log"
Sep 16 17:06:06.701185 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:06.701147    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql_67aff6eb-30af-4e42-ae7f-02ac193f1b19/payload-processing/0.log"
Sep 16 17:06:52.747423 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:52.747378    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mpcpf_bb67f300-0327-4e84-9998-045e6b646f34/payload-processing/0.log"
Sep 16 17:06:52.867341 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:52.867314    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql_67aff6eb-30af-4e42-ae7f-02ac193f1b19/payload-processing/0.log"
Sep 16 17:06:53.398633 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.398596    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"]
Sep 16 17:06:53.402205 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.402173    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.404583 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.404564    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-48fbf0-openshift-default-dockercfg-8g2mf\""
Sep 16 17:06:53.413879 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.413855    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"]
Sep 16 17:06:53.499798 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.499761    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-workload-socket\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.499966 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.499813    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-workload-certs\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.499966 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.499889    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-data\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.500059 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.499997    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxd4w\" (UniqueName: \"kubernetes.io/projected/d114ea8f-b67e-44f7-a263-29d40b8cf812-kube-api-access-hxd4w\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.500059 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.500033    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-token\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.500137 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.500075    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d114ea8f-b67e-44f7-a263-29d40b8cf812-istiod-ca-cert\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.500180 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.500138    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-envoy\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.500180 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.500162    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-podinfo\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.500264 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.500237    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-credential-socket\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.601421 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.601373    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-envoy\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.601421 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.601423    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-podinfo\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.601657 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.601447    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-credential-socket\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.601657 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.601464    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-workload-socket\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.601657 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.601482    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-workload-certs\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.601657 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.601511    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-data\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.601657 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.601558    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxd4w\" (UniqueName: \"kubernetes.io/projected/d114ea8f-b67e-44f7-a263-29d40b8cf812-kube-api-access-hxd4w\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.601657 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.601584    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-token\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.601657 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.601632    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d114ea8f-b67e-44f7-a263-29d40b8cf812-istiod-ca-cert\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.602008 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.601912    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-credential-socket\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.602008 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.601933    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-workload-socket\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.602120 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.602056    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-data\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.602120 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.602093    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-workload-certs\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.602325 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.602307    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d114ea8f-b67e-44f7-a263-29d40b8cf812-istiod-ca-cert\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.604279 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.604251    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-envoy\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.604419 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.604396    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-podinfo\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.611238 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.611216    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxd4w\" (UniqueName: \"kubernetes.io/projected/d114ea8f-b67e-44f7-a263-29d40b8cf812-kube-api-access-hxd4w\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.611334 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.611284    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-token\") pod \"e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") " pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.712979 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.712926    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:53.987632 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.987606    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"]
Sep 16 17:06:53.989189 ip-10-0-140-120 kubenswrapper[2556]: W0916 17:06:53.989160    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd114ea8f_b67e_44f7_a263_29d40b8cf812.slice/crio-42e687e716bd3a0d4a395f6c7738fcdb3b83523b93f8e6a4e50e7dabae0d7684 WatchSource:0}: Error finding container 42e687e716bd3a0d4a395f6c7738fcdb3b83523b93f8e6a4e50e7dabae0d7684: Status 404 returned error can't find the container with id 42e687e716bd3a0d4a395f6c7738fcdb3b83523b93f8e6a4e50e7dabae0d7684
Sep 16 17:06:53.991395 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.991362    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 17:06:53.991481 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.991427    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 17:06:53.991481 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:53.991457    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 17:06:54.071399 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:54.071362    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4" event={"ID":"d114ea8f-b67e-44f7-a263-29d40b8cf812","Type":"ContainerStarted","Data":"6907613684a191f1e40f3e253fa7951b648cd7859ba0ce0f84af03b7bf105107"}
Sep 16 17:06:54.071539 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:54.071431    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4" event={"ID":"d114ea8f-b67e-44f7-a263-29d40b8cf812","Type":"ContainerStarted","Data":"42e687e716bd3a0d4a395f6c7738fcdb3b83523b93f8e6a4e50e7dabae0d7684"}
Sep 16 17:06:54.093360 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:54.093254    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4" podStartSLOduration=1.093230285 podStartE2EDuration="1.093230285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:06:54.092654031 +0000 UTC m=+1813.854615478" watchObservedRunningTime="2026-09-16 17:06:54.093230285 +0000 UTC m=+1813.855191755"
Sep 16 17:06:54.713742 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:54.713706    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:55.719149 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:55.719112    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:56.081672 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:56.081563    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:06:56.082589 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:06:56.082570    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:07:16.195934 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:16.195902    2556 generic.go:356] "Generic (PLEG): container finished" podID="d114ea8f-b67e-44f7-a263-29d40b8cf812" containerID="6907613684a191f1e40f3e253fa7951b648cd7859ba0ce0f84af03b7bf105107" exitCode=0
Sep 16 17:07:16.196347 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:16.195984    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4" event={"ID":"d114ea8f-b67e-44f7-a263-29d40b8cf812","Type":"ContainerDied","Data":"6907613684a191f1e40f3e253fa7951b648cd7859ba0ce0f84af03b7bf105107"}
Sep 16 17:07:16.196527 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:16.196509    2556 scope.go:117] "RemoveContainer" containerID="6907613684a191f1e40f3e253fa7951b648cd7859ba0ce0f84af03b7bf105107"
Sep 16 17:07:16.197683 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:16.197655    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 17:07:16.197788 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:16.197737    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 17:07:16.197850 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:16.197783    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 17:07:17.203491 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:17.203457    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4" event={"ID":"d114ea8f-b67e-44f7-a263-29d40b8cf812","Type":"ContainerStarted","Data":"5457a9a565a71050668eacc4810828c97edd94d6ad04a9b4dc67d2ff4ad20316"}
Sep 16 17:07:17.713564 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:17.713529    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:07:17.713731 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:17.713700    2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 17:07:17.713776 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:17.713759    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4" podUID="d114ea8f-b67e-44f7-a263-29d40b8cf812" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 17:07:18.713598 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:18.713562    2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 17:07:18.713999 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:18.713620    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4" podUID="d114ea8f-b67e-44f7-a263-29d40b8cf812" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 17:07:19.713650 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:19.713609    2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 17:07:19.714142 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:19.713679    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4" podUID="d114ea8f-b67e-44f7-a263-29d40b8cf812" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 17:07:20.713491 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:20.713453    2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 17:07:20.713676 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:20.713515    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4" podUID="d114ea8f-b67e-44f7-a263-29d40b8cf812" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 17:07:21.714312 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:21.714278    2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 17:07:21.714686 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:21.714331    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4" podUID="d114ea8f-b67e-44f7-a263-29d40b8cf812" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 17:07:22.713319 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:22.713279    2556 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Sep 16 17:07:22.713501 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:22.713334    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4" podUID="d114ea8f-b67e-44f7-a263-29d40b8cf812" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Sep 16 17:07:23.717540 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:23.717509    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:07:23.717910 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:23.717790    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:07:23.718560 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:23.718543    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:07:44.398520 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:44.398483    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"]
Sep 16 17:07:44.399057 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:44.398712    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4" podUID="d114ea8f-b67e-44f7-a263-29d40b8cf812" containerName="istio-proxy" containerID="cri-o://5457a9a565a71050668eacc4810828c97edd94d6ad04a9b4dc67d2ff4ad20316" gracePeriod=30
Sep 16 17:07:49.647931 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.647902    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:07:49.727127 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.727100    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-data\") pod \"d114ea8f-b67e-44f7-a263-29d40b8cf812\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") "
Sep 16 17:07:49.727291 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.727152    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-podinfo\") pod \"d114ea8f-b67e-44f7-a263-29d40b8cf812\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") "
Sep 16 17:07:49.727291 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.727181    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-workload-certs\") pod \"d114ea8f-b67e-44f7-a263-29d40b8cf812\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") "
Sep 16 17:07:49.727291 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.727220    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxd4w\" (UniqueName: \"kubernetes.io/projected/d114ea8f-b67e-44f7-a263-29d40b8cf812-kube-api-access-hxd4w\") pod \"d114ea8f-b67e-44f7-a263-29d40b8cf812\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") "
Sep 16 17:07:49.727291 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.727248    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-workload-socket\") pod \"d114ea8f-b67e-44f7-a263-29d40b8cf812\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") "
Sep 16 17:07:49.727510 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.727303    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-token\") pod \"d114ea8f-b67e-44f7-a263-29d40b8cf812\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") "
Sep 16 17:07:49.727510 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.727351    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d114ea8f-b67e-44f7-a263-29d40b8cf812-istiod-ca-cert\") pod \"d114ea8f-b67e-44f7-a263-29d40b8cf812\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") "
Sep 16 17:07:49.727510 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.727399    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-envoy\") pod \"d114ea8f-b67e-44f7-a263-29d40b8cf812\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") "
Sep 16 17:07:49.727510 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.727399    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-data" (OuterVolumeSpecName: "istio-data") pod "d114ea8f-b67e-44f7-a263-29d40b8cf812" (UID: "d114ea8f-b67e-44f7-a263-29d40b8cf812"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:07:49.727510 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.727440    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-credential-socket\") pod \"d114ea8f-b67e-44f7-a263-29d40b8cf812\" (UID: \"d114ea8f-b67e-44f7-a263-29d40b8cf812\") "
Sep 16 17:07:49.727778 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.727503    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d114ea8f-b67e-44f7-a263-29d40b8cf812" (UID: "d114ea8f-b67e-44f7-a263-29d40b8cf812"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:07:49.727778 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.727595    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d114ea8f-b67e-44f7-a263-29d40b8cf812" (UID: "d114ea8f-b67e-44f7-a263-29d40b8cf812"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:07:49.727778 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.727719    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d114ea8f-b67e-44f7-a263-29d40b8cf812" (UID: "d114ea8f-b67e-44f7-a263-29d40b8cf812"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:07:49.727778 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.727738    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d114ea8f-b67e-44f7-a263-29d40b8cf812-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d114ea8f-b67e-44f7-a263-29d40b8cf812" (UID: "d114ea8f-b67e-44f7-a263-29d40b8cf812"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:07:49.728025 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.727805    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d114ea8f-b67e-44f7-a263-29d40b8cf812-istiod-ca-cert\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:07:49.728025 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.727821    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-credential-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:07:49.728025 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.727835    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-data\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:07:49.728025 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.727846    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-workload-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:07:49.728025 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.727854    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-workload-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:07:49.729473 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.729452    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d114ea8f-b67e-44f7-a263-29d40b8cf812" (UID: "d114ea8f-b67e-44f7-a263-29d40b8cf812"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:07:49.729473 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.729462    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-token" (OuterVolumeSpecName: "istio-token") pod "d114ea8f-b67e-44f7-a263-29d40b8cf812" (UID: "d114ea8f-b67e-44f7-a263-29d40b8cf812"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:07:49.729598 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.729485    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d114ea8f-b67e-44f7-a263-29d40b8cf812" (UID: "d114ea8f-b67e-44f7-a263-29d40b8cf812"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 17:07:49.729636 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.729605    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d114ea8f-b67e-44f7-a263-29d40b8cf812-kube-api-access-hxd4w" (OuterVolumeSpecName: "kube-api-access-hxd4w") pod "d114ea8f-b67e-44f7-a263-29d40b8cf812" (UID: "d114ea8f-b67e-44f7-a263-29d40b8cf812"). InnerVolumeSpecName "kube-api-access-hxd4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:07:49.828773 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.828741    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-podinfo\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:07:49.828773 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.828768    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxd4w\" (UniqueName: \"kubernetes.io/projected/d114ea8f-b67e-44f7-a263-29d40b8cf812-kube-api-access-hxd4w\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:07:49.828979 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.828778    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-token\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:07:49.828979 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:49.828804    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d114ea8f-b67e-44f7-a263-29d40b8cf812-istio-envoy\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:07:50.372197 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:50.372159    2556 generic.go:356] "Generic (PLEG): container finished" podID="d114ea8f-b67e-44f7-a263-29d40b8cf812" containerID="5457a9a565a71050668eacc4810828c97edd94d6ad04a9b4dc67d2ff4ad20316" exitCode=0
Sep 16 17:07:50.372393 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:50.372211    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"
Sep 16 17:07:50.372393 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:50.372245    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4" event={"ID":"d114ea8f-b67e-44f7-a263-29d40b8cf812","Type":"ContainerDied","Data":"5457a9a565a71050668eacc4810828c97edd94d6ad04a9b4dc67d2ff4ad20316"}
Sep 16 17:07:50.372393 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:50.372315    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4" event={"ID":"d114ea8f-b67e-44f7-a263-29d40b8cf812","Type":"ContainerDied","Data":"42e687e716bd3a0d4a395f6c7738fcdb3b83523b93f8e6a4e50e7dabae0d7684"}
Sep 16 17:07:50.372393 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:50.372344    2556 scope.go:117] "RemoveContainer" containerID="5457a9a565a71050668eacc4810828c97edd94d6ad04a9b4dc67d2ff4ad20316"
Sep 16 17:07:50.381498 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:50.381480    2556 scope.go:117] "RemoveContainer" containerID="6907613684a191f1e40f3e253fa7951b648cd7859ba0ce0f84af03b7bf105107"
Sep 16 17:07:50.389058 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:50.389039    2556 scope.go:117] "RemoveContainer" containerID="5457a9a565a71050668eacc4810828c97edd94d6ad04a9b4dc67d2ff4ad20316"
Sep 16 17:07:50.389312 ip-10-0-140-120 kubenswrapper[2556]: E0916 17:07:50.389293    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5457a9a565a71050668eacc4810828c97edd94d6ad04a9b4dc67d2ff4ad20316\": container with ID starting with 5457a9a565a71050668eacc4810828c97edd94d6ad04a9b4dc67d2ff4ad20316 not found: ID does not exist" containerID="5457a9a565a71050668eacc4810828c97edd94d6ad04a9b4dc67d2ff4ad20316"
Sep 16 17:07:50.389362 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:50.389319    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5457a9a565a71050668eacc4810828c97edd94d6ad04a9b4dc67d2ff4ad20316"} err="failed to get container status \"5457a9a565a71050668eacc4810828c97edd94d6ad04a9b4dc67d2ff4ad20316\": rpc error: code = NotFound desc = could not find container \"5457a9a565a71050668eacc4810828c97edd94d6ad04a9b4dc67d2ff4ad20316\": container with ID starting with 5457a9a565a71050668eacc4810828c97edd94d6ad04a9b4dc67d2ff4ad20316 not found: ID does not exist"
Sep 16 17:07:50.389362 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:50.389334    2556 scope.go:117] "RemoveContainer" containerID="6907613684a191f1e40f3e253fa7951b648cd7859ba0ce0f84af03b7bf105107"
Sep 16 17:07:50.389584 ip-10-0-140-120 kubenswrapper[2556]: E0916 17:07:50.389566    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6907613684a191f1e40f3e253fa7951b648cd7859ba0ce0f84af03b7bf105107\": container with ID starting with 6907613684a191f1e40f3e253fa7951b648cd7859ba0ce0f84af03b7bf105107 not found: ID does not exist" containerID="6907613684a191f1e40f3e253fa7951b648cd7859ba0ce0f84af03b7bf105107"
Sep 16 17:07:50.389635 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:50.389594    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6907613684a191f1e40f3e253fa7951b648cd7859ba0ce0f84af03b7bf105107"} err="failed to get container status \"6907613684a191f1e40f3e253fa7951b648cd7859ba0ce0f84af03b7bf105107\": rpc error: code = NotFound desc = could not find container \"6907613684a191f1e40f3e253fa7951b648cd7859ba0ce0f84af03b7bf105107\": container with ID starting with 6907613684a191f1e40f3e253fa7951b648cd7859ba0ce0f84af03b7bf105107 not found: ID does not exist"
Sep 16 17:07:50.394870 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:50.394849    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"]
Sep 16 17:07:50.400577 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:50.400554    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-48fbf0-openshift-default-79dfd9df68-8hjk4"]
Sep 16 17:07:50.852359 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:07:50.852325    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d114ea8f-b67e-44f7-a263-29d40b8cf812" path="/var/lib/kubelet/pods/d114ea8f-b67e-44f7-a263-29d40b8cf812/volumes"
Sep 16 17:08:03.434338 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:03.434125    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"]
Sep 16 17:08:03.435595 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:03.435533    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh" podUID="37794aad-c1f0-4d9b-bd68-786dab64dcd4" containerName="maas-api" containerID="cri-o://59acc7b58c3877466ccecf6afcee497a49a5a6e751d3a330547d417a1cb63769" gracePeriod=30
Sep 16 17:08:03.686159 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:03.686094    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"
Sep 16 17:08:03.861020 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:03.860982    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-978bx\" (UniqueName: \"kubernetes.io/projected/37794aad-c1f0-4d9b-bd68-786dab64dcd4-kube-api-access-978bx\") pod \"37794aad-c1f0-4d9b-bd68-786dab64dcd4\" (UID: \"37794aad-c1f0-4d9b-bd68-786dab64dcd4\") "
Sep 16 17:08:03.861197 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:03.861035    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37794aad-c1f0-4d9b-bd68-786dab64dcd4-maas-api-tls\") pod \"37794aad-c1f0-4d9b-bd68-786dab64dcd4\" (UID: \"37794aad-c1f0-4d9b-bd68-786dab64dcd4\") "
Sep 16 17:08:03.861197 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:03.861072    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37794aad-c1f0-4d9b-bd68-786dab64dcd4-openshift-service-ca\") pod \"37794aad-c1f0-4d9b-bd68-786dab64dcd4\" (UID: \"37794aad-c1f0-4d9b-bd68-786dab64dcd4\") "
Sep 16 17:08:03.861197 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:03.861158    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37794aad-c1f0-4d9b-bd68-786dab64dcd4-metrics-certs\") pod \"37794aad-c1f0-4d9b-bd68-786dab64dcd4\" (UID: \"37794aad-c1f0-4d9b-bd68-786dab64dcd4\") "
Sep 16 17:08:03.861781 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:03.861753    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37794aad-c1f0-4d9b-bd68-786dab64dcd4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "37794aad-c1f0-4d9b-bd68-786dab64dcd4" (UID: "37794aad-c1f0-4d9b-bd68-786dab64dcd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:08:03.863548 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:03.863517    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37794aad-c1f0-4d9b-bd68-786dab64dcd4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "37794aad-c1f0-4d9b-bd68-786dab64dcd4" (UID: "37794aad-c1f0-4d9b-bd68-786dab64dcd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:08:03.863651 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:03.863583    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37794aad-c1f0-4d9b-bd68-786dab64dcd4-kube-api-access-978bx" (OuterVolumeSpecName: "kube-api-access-978bx") pod "37794aad-c1f0-4d9b-bd68-786dab64dcd4" (UID: "37794aad-c1f0-4d9b-bd68-786dab64dcd4"). InnerVolumeSpecName "kube-api-access-978bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:08:03.863709 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:03.863669    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37794aad-c1f0-4d9b-bd68-786dab64dcd4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "37794aad-c1f0-4d9b-bd68-786dab64dcd4" (UID: "37794aad-c1f0-4d9b-bd68-786dab64dcd4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:08:03.961954 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:03.961860    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37794aad-c1f0-4d9b-bd68-786dab64dcd4-metrics-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:08:03.961954 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:03.961890    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-978bx\" (UniqueName: \"kubernetes.io/projected/37794aad-c1f0-4d9b-bd68-786dab64dcd4-kube-api-access-978bx\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:08:03.961954 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:03.961901    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/37794aad-c1f0-4d9b-bd68-786dab64dcd4-maas-api-tls\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:08:03.961954 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:03.961909    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/37794aad-c1f0-4d9b-bd68-786dab64dcd4-openshift-service-ca\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:08:04.443741 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:04.443696    2556 generic.go:356] "Generic (PLEG): container finished" podID="37794aad-c1f0-4d9b-bd68-786dab64dcd4" containerID="59acc7b58c3877466ccecf6afcee497a49a5a6e751d3a330547d417a1cb63769" exitCode=0
Sep 16 17:08:04.444158 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:04.443752    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"
Sep 16 17:08:04.444158 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:04.443755    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh" event={"ID":"37794aad-c1f0-4d9b-bd68-786dab64dcd4","Type":"ContainerDied","Data":"59acc7b58c3877466ccecf6afcee497a49a5a6e751d3a330547d417a1cb63769"}
Sep 16 17:08:04.444158 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:04.443806    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh" event={"ID":"37794aad-c1f0-4d9b-bd68-786dab64dcd4","Type":"ContainerDied","Data":"ab0de8f942efb909c18f70dffa5954a8188f5d312febf4b6207ab979df0d1b29"}
Sep 16 17:08:04.444158 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:04.443835    2556 scope.go:117] "RemoveContainer" containerID="59acc7b58c3877466ccecf6afcee497a49a5a6e751d3a330547d417a1cb63769"
Sep 16 17:08:04.453338 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:04.453319    2556 scope.go:117] "RemoveContainer" containerID="59acc7b58c3877466ccecf6afcee497a49a5a6e751d3a330547d417a1cb63769"
Sep 16 17:08:04.453604 ip-10-0-140-120 kubenswrapper[2556]: E0916 17:08:04.453586    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59acc7b58c3877466ccecf6afcee497a49a5a6e751d3a330547d417a1cb63769\": container with ID starting with 59acc7b58c3877466ccecf6afcee497a49a5a6e751d3a330547d417a1cb63769 not found: ID does not exist" containerID="59acc7b58c3877466ccecf6afcee497a49a5a6e751d3a330547d417a1cb63769"
Sep 16 17:08:04.453672 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:04.453610    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59acc7b58c3877466ccecf6afcee497a49a5a6e751d3a330547d417a1cb63769"} err="failed to get container status \"59acc7b58c3877466ccecf6afcee497a49a5a6e751d3a330547d417a1cb63769\": rpc error: code = NotFound desc = could not find container \"59acc7b58c3877466ccecf6afcee497a49a5a6e751d3a330547d417a1cb63769\": container with ID starting with 59acc7b58c3877466ccecf6afcee497a49a5a6e751d3a330547d417a1cb63769 not found: ID does not exist"
Sep 16 17:08:04.466197 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:04.466174    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"]
Sep 16 17:08:04.469988 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:04.469964    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-9ac315-57b8cffc9d-s78vh"]
Sep 16 17:08:04.849073 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:08:04.848995    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37794aad-c1f0-4d9b-bd68-786dab64dcd4" path="/var/lib/kubelet/pods/37794aad-c1f0-4d9b-bd68-786dab64dcd4/volumes"
Sep 16 17:09:06.933775 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:06.933741    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"]
Sep 16 17:09:06.934276 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:06.933969    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9" podUID="e00bc8c1-aef0-45c2-a034-0ad3fe638892" containerName="istio-proxy" containerID="cri-o://beeac25fdf5fa8ab945d8487a9f104aa0ec2cef5d95c793157dfb838e72288a1" gracePeriod=30
Sep 16 17:09:10.252579 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:10.252539    2556 patch_prober.go:28] interesting pod/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 17:09:10.252956 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:10.252610    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9" podUID="e00bc8c1-aef0-45c2-a034-0ad3fe638892" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 17:09:12.186923 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.186899    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:09:12.251310 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.251236    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-workload-socket\") pod \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") "
Sep 16 17:09:12.251310 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.251299    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62plr\" (UniqueName: \"kubernetes.io/projected/e00bc8c1-aef0-45c2-a034-0ad3fe638892-kube-api-access-62plr\") pod \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") "
Sep 16 17:09:12.251518 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.251322    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-data\") pod \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") "
Sep 16 17:09:12.251518 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.251338    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-workload-certs\") pod \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") "
Sep 16 17:09:12.251518 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.251358    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-podinfo\") pod \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") "
Sep 16 17:09:12.251518 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.251389    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-envoy\") pod \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") "
Sep 16 17:09:12.251518 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.251425    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istiod-ca-cert\") pod \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") "
Sep 16 17:09:12.251518 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.251483    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-token\") pod \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") "
Sep 16 17:09:12.251518 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.251512    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-credential-socket\") pod \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\" (UID: \"e00bc8c1-aef0-45c2-a034-0ad3fe638892\") "
Sep 16 17:09:12.251892 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.251511    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e00bc8c1-aef0-45c2-a034-0ad3fe638892" (UID: "e00bc8c1-aef0-45c2-a034-0ad3fe638892"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:09:12.251892 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.251640    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-data" (OuterVolumeSpecName: "istio-data") pod "e00bc8c1-aef0-45c2-a034-0ad3fe638892" (UID: "e00bc8c1-aef0-45c2-a034-0ad3fe638892"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:09:12.251892 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.251745    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e00bc8c1-aef0-45c2-a034-0ad3fe638892" (UID: "e00bc8c1-aef0-45c2-a034-0ad3fe638892"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:09:12.251892 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.251872    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e00bc8c1-aef0-45c2-a034-0ad3fe638892" (UID: "e00bc8c1-aef0-45c2-a034-0ad3fe638892"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:09:12.252149 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.251958    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e00bc8c1-aef0-45c2-a034-0ad3fe638892" (UID: "e00bc8c1-aef0-45c2-a034-0ad3fe638892"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:09:12.252149 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.252102    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-workload-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:12.252149 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.252123    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-data\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:12.252149 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.252137    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-workload-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:12.252149 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.252150    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istiod-ca-cert\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:12.252409 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.252162    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-credential-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:12.253689 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.253653    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e00bc8c1-aef0-45c2-a034-0ad3fe638892" (UID: "e00bc8c1-aef0-45c2-a034-0ad3fe638892"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 17:09:12.253806 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.253704    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-token" (OuterVolumeSpecName: "istio-token") pod "e00bc8c1-aef0-45c2-a034-0ad3fe638892" (UID: "e00bc8c1-aef0-45c2-a034-0ad3fe638892"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:09:12.253806 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.253734    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e00bc8c1-aef0-45c2-a034-0ad3fe638892" (UID: "e00bc8c1-aef0-45c2-a034-0ad3fe638892"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:09:12.253806 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.253738    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00bc8c1-aef0-45c2-a034-0ad3fe638892-kube-api-access-62plr" (OuterVolumeSpecName: "kube-api-access-62plr") pod "e00bc8c1-aef0-45c2-a034-0ad3fe638892" (UID: "e00bc8c1-aef0-45c2-a034-0ad3fe638892"). InnerVolumeSpecName "kube-api-access-62plr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:09:12.353285 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.353247    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62plr\" (UniqueName: \"kubernetes.io/projected/e00bc8c1-aef0-45c2-a034-0ad3fe638892-kube-api-access-62plr\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:12.353285 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.353282    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-podinfo\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:12.353285 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.353293    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-envoy\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:12.353464 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.353301    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00bc8c1-aef0-45c2-a034-0ad3fe638892-istio-token\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:12.770220 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.770180    2556 generic.go:356] "Generic (PLEG): container finished" podID="e00bc8c1-aef0-45c2-a034-0ad3fe638892" containerID="beeac25fdf5fa8ab945d8487a9f104aa0ec2cef5d95c793157dfb838e72288a1" exitCode=0
Sep 16 17:09:12.770386 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.770231    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9" event={"ID":"e00bc8c1-aef0-45c2-a034-0ad3fe638892","Type":"ContainerDied","Data":"beeac25fdf5fa8ab945d8487a9f104aa0ec2cef5d95c793157dfb838e72288a1"}
Sep 16 17:09:12.770386 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.770235    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"
Sep 16 17:09:12.770386 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.770280    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9" event={"ID":"e00bc8c1-aef0-45c2-a034-0ad3fe638892","Type":"ContainerDied","Data":"eae41b04d0ae8f976e973335a11a6ff86f55fcca30d6b83ff9a2d8f0cfc467ed"}
Sep 16 17:09:12.770386 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.770303    2556 scope.go:117] "RemoveContainer" containerID="beeac25fdf5fa8ab945d8487a9f104aa0ec2cef5d95c793157dfb838e72288a1"
Sep 16 17:09:12.779256 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.779237    2556 scope.go:117] "RemoveContainer" containerID="74eb4ea0a169b69df10638998eba744eb284266b22289e22fb4dddc7c2308aff"
Sep 16 17:09:12.787037 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.787017    2556 scope.go:117] "RemoveContainer" containerID="beeac25fdf5fa8ab945d8487a9f104aa0ec2cef5d95c793157dfb838e72288a1"
Sep 16 17:09:12.787296 ip-10-0-140-120 kubenswrapper[2556]: E0916 17:09:12.787275    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beeac25fdf5fa8ab945d8487a9f104aa0ec2cef5d95c793157dfb838e72288a1\": container with ID starting with beeac25fdf5fa8ab945d8487a9f104aa0ec2cef5d95c793157dfb838e72288a1 not found: ID does not exist" containerID="beeac25fdf5fa8ab945d8487a9f104aa0ec2cef5d95c793157dfb838e72288a1"
Sep 16 17:09:12.787353 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.787304    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beeac25fdf5fa8ab945d8487a9f104aa0ec2cef5d95c793157dfb838e72288a1"} err="failed to get container status \"beeac25fdf5fa8ab945d8487a9f104aa0ec2cef5d95c793157dfb838e72288a1\": rpc error: code = NotFound desc = could not find container \"beeac25fdf5fa8ab945d8487a9f104aa0ec2cef5d95c793157dfb838e72288a1\": container with ID starting with beeac25fdf5fa8ab945d8487a9f104aa0ec2cef5d95c793157dfb838e72288a1 not found: ID does not exist"
Sep 16 17:09:12.787353 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.787321    2556 scope.go:117] "RemoveContainer" containerID="74eb4ea0a169b69df10638998eba744eb284266b22289e22fb4dddc7c2308aff"
Sep 16 17:09:12.787570 ip-10-0-140-120 kubenswrapper[2556]: E0916 17:09:12.787553    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74eb4ea0a169b69df10638998eba744eb284266b22289e22fb4dddc7c2308aff\": container with ID starting with 74eb4ea0a169b69df10638998eba744eb284266b22289e22fb4dddc7c2308aff not found: ID does not exist" containerID="74eb4ea0a169b69df10638998eba744eb284266b22289e22fb4dddc7c2308aff"
Sep 16 17:09:12.787610 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.787578    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74eb4ea0a169b69df10638998eba744eb284266b22289e22fb4dddc7c2308aff"} err="failed to get container status \"74eb4ea0a169b69df10638998eba744eb284266b22289e22fb4dddc7c2308aff\": rpc error: code = NotFound desc = could not find container \"74eb4ea0a169b69df10638998eba744eb284266b22289e22fb4dddc7c2308aff\": container with ID starting with 74eb4ea0a169b69df10638998eba744eb284266b22289e22fb4dddc7c2308aff not found: ID does not exist"
Sep 16 17:09:12.791999 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.791977    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"]
Sep 16 17:09:12.795851 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.795831    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-9ac315-openshift-default-78556459c4-5zsc9"]
Sep 16 17:09:12.848686 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:12.848657    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e00bc8c1-aef0-45c2-a034-0ad3fe638892" path="/var/lib/kubelet/pods/e00bc8c1-aef0-45c2-a034-0ad3fe638892/volumes"
Sep 16 17:09:17.938047 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:17.938010    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql"]
Sep 16 17:09:17.938434 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:17.938263    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql" podUID="67aff6eb-30af-4e42-ae7f-02ac193f1b19" containerName="payload-processing" containerID="cri-o://6e646cdcf64254b1cfa68044c6bcbaabe541675c8f25192dc2d368a3148b7a64" gracePeriod=30
Sep 16 17:09:18.199665 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:18.199603    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql"
Sep 16 17:09:18.306298 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:18.306257    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67aff6eb-30af-4e42-ae7f-02ac193f1b19-plugins-config-volume\") pod \"67aff6eb-30af-4e42-ae7f-02ac193f1b19\" (UID: \"67aff6eb-30af-4e42-ae7f-02ac193f1b19\") "
Sep 16 17:09:18.306494 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:18.306332    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rt9jx\" (UniqueName: \"kubernetes.io/projected/67aff6eb-30af-4e42-ae7f-02ac193f1b19-kube-api-access-rt9jx\") pod \"67aff6eb-30af-4e42-ae7f-02ac193f1b19\" (UID: \"67aff6eb-30af-4e42-ae7f-02ac193f1b19\") "
Sep 16 17:09:18.306737 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:18.306705    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67aff6eb-30af-4e42-ae7f-02ac193f1b19-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67aff6eb-30af-4e42-ae7f-02ac193f1b19" (UID: "67aff6eb-30af-4e42-ae7f-02ac193f1b19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:09:18.308384 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:18.308356    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67aff6eb-30af-4e42-ae7f-02ac193f1b19-kube-api-access-rt9jx" (OuterVolumeSpecName: "kube-api-access-rt9jx") pod "67aff6eb-30af-4e42-ae7f-02ac193f1b19" (UID: "67aff6eb-30af-4e42-ae7f-02ac193f1b19"). InnerVolumeSpecName "kube-api-access-rt9jx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:09:18.406920 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:18.406888    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67aff6eb-30af-4e42-ae7f-02ac193f1b19-plugins-config-volume\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:18.406920 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:18.406916    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt9jx\" (UniqueName: \"kubernetes.io/projected/67aff6eb-30af-4e42-ae7f-02ac193f1b19-kube-api-access-rt9jx\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:18.799524 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:18.799484    2556 generic.go:356] "Generic (PLEG): container finished" podID="67aff6eb-30af-4e42-ae7f-02ac193f1b19" containerID="6e646cdcf64254b1cfa68044c6bcbaabe541675c8f25192dc2d368a3148b7a64" exitCode=0
Sep 16 17:09:18.799704 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:18.799548    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql"
Sep 16 17:09:18.799704 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:18.799572    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql" event={"ID":"67aff6eb-30af-4e42-ae7f-02ac193f1b19","Type":"ContainerDied","Data":"6e646cdcf64254b1cfa68044c6bcbaabe541675c8f25192dc2d368a3148b7a64"}
Sep 16 17:09:18.799704 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:18.799651    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql" event={"ID":"67aff6eb-30af-4e42-ae7f-02ac193f1b19","Type":"ContainerDied","Data":"1d4f2a06d2d20759c2b63785cda3d7ac153669b8904c1303a6a64fbbf627ae21"}
Sep 16 17:09:18.799704 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:18.799673    2556 scope.go:117] "RemoveContainer" containerID="6e646cdcf64254b1cfa68044c6bcbaabe541675c8f25192dc2d368a3148b7a64"
Sep 16 17:09:18.808653 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:18.808635    2556 scope.go:117] "RemoveContainer" containerID="6e646cdcf64254b1cfa68044c6bcbaabe541675c8f25192dc2d368a3148b7a64"
Sep 16 17:09:18.808926 ip-10-0-140-120 kubenswrapper[2556]: E0916 17:09:18.808901    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e646cdcf64254b1cfa68044c6bcbaabe541675c8f25192dc2d368a3148b7a64\": container with ID starting with 6e646cdcf64254b1cfa68044c6bcbaabe541675c8f25192dc2d368a3148b7a64 not found: ID does not exist" containerID="6e646cdcf64254b1cfa68044c6bcbaabe541675c8f25192dc2d368a3148b7a64"
Sep 16 17:09:18.809025 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:18.808935    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e646cdcf64254b1cfa68044c6bcbaabe541675c8f25192dc2d368a3148b7a64"} err="failed to get container status \"6e646cdcf64254b1cfa68044c6bcbaabe541675c8f25192dc2d368a3148b7a64\": rpc error: code = NotFound desc = could not find container \"6e646cdcf64254b1cfa68044c6bcbaabe541675c8f25192dc2d368a3148b7a64\": container with ID starting with 6e646cdcf64254b1cfa68044c6bcbaabe541675c8f25192dc2d368a3148b7a64 not found: ID does not exist"
Sep 16 17:09:18.820819 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:18.820792    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql"]
Sep 16 17:09:18.824740 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:18.824717    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6586c3-88c878bb9-qsvql"]
Sep 16 17:09:18.848900 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:18.848868    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67aff6eb-30af-4e42-ae7f-02ac193f1b19" path="/var/lib/kubelet/pods/67aff6eb-30af-4e42-ae7f-02ac193f1b19/volumes"
Sep 16 17:09:35.360363 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:35.360126    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-52fd8c-29826300-f5jl8"]
Sep 16 17:09:35.362192 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:35.362170    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-52fd8c-29826300-f5jl8"]
Sep 16 17:09:36.850156 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:36.850121    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1223020-cfe7-427e-a1da-0c51dc1f493b" path="/var/lib/kubelet/pods/a1223020-cfe7-427e-a1da-0c51dc1f493b/volumes"
Sep 16 17:09:40.918088 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:40.918055    2556 scope.go:117] "RemoveContainer" containerID="9128183d82e45afd426a5d693586341be550e2786d6d329c7dace962fa547b2f"
Sep 16 17:09:42.718110 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:42.718078    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"]
Sep 16 17:09:42.718523 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:42.718342    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r" podUID="576f4074-7fe9-42cb-810b-bb18acdbfa69" containerName="istio-proxy" containerID="cri-o://73d45c78c4fc180761f3aeee8bee01c20090f56ef42df521a84fe38d357c919c" gracePeriod=30
Sep 16 17:09:47.932246 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:47.932215    2556 generic.go:356] "Generic (PLEG): container finished" podID="576f4074-7fe9-42cb-810b-bb18acdbfa69" containerID="73d45c78c4fc180761f3aeee8bee01c20090f56ef42df521a84fe38d357c919c" exitCode=0
Sep 16 17:09:47.932696 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:47.932283    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r" event={"ID":"576f4074-7fe9-42cb-810b-bb18acdbfa69","Type":"ContainerDied","Data":"73d45c78c4fc180761f3aeee8bee01c20090f56ef42df521a84fe38d357c919c"}
Sep 16 17:09:47.932696 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:47.932346    2556 scope.go:117] "RemoveContainer" containerID="bb92001a47da9b6413d668a98f3cd3db175359678daabc5f6458f0987412bb6b"
Sep 16 17:09:47.963642 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:47.963621    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 17:09:48.062145 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.062075    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j92xj\" (UniqueName: \"kubernetes.io/projected/576f4074-7fe9-42cb-810b-bb18acdbfa69-kube-api-access-j92xj\") pod \"576f4074-7fe9-42cb-810b-bb18acdbfa69\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") "
Sep 16 17:09:48.062145 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.062114    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-workload-socket\") pod \"576f4074-7fe9-42cb-810b-bb18acdbfa69\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") "
Sep 16 17:09:48.062145 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.062133    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576f4074-7fe9-42cb-810b-bb18acdbfa69-istiod-ca-cert\") pod \"576f4074-7fe9-42cb-810b-bb18acdbfa69\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") "
Sep 16 17:09:48.062435 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.062184    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-credential-socket\") pod \"576f4074-7fe9-42cb-810b-bb18acdbfa69\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") "
Sep 16 17:09:48.062435 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.062209    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-data\") pod \"576f4074-7fe9-42cb-810b-bb18acdbfa69\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") "
Sep 16 17:09:48.062435 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.062245    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-podinfo\") pod \"576f4074-7fe9-42cb-810b-bb18acdbfa69\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") "
Sep 16 17:09:48.062435 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.062296    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-workload-certs\") pod \"576f4074-7fe9-42cb-810b-bb18acdbfa69\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") "
Sep 16 17:09:48.062435 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.062322    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-envoy\") pod \"576f4074-7fe9-42cb-810b-bb18acdbfa69\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") "
Sep 16 17:09:48.062435 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.062354    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-token\") pod \"576f4074-7fe9-42cb-810b-bb18acdbfa69\" (UID: \"576f4074-7fe9-42cb-810b-bb18acdbfa69\") "
Sep 16 17:09:48.062435 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.062412    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "576f4074-7fe9-42cb-810b-bb18acdbfa69" (UID: "576f4074-7fe9-42cb-810b-bb18acdbfa69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:09:48.062435 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.062423    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "576f4074-7fe9-42cb-810b-bb18acdbfa69" (UID: "576f4074-7fe9-42cb-810b-bb18acdbfa69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:09:48.062809 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.062472    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-data" (OuterVolumeSpecName: "istio-data") pod "576f4074-7fe9-42cb-810b-bb18acdbfa69" (UID: "576f4074-7fe9-42cb-810b-bb18acdbfa69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:09:48.062809 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.062538    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/576f4074-7fe9-42cb-810b-bb18acdbfa69-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "576f4074-7fe9-42cb-810b-bb18acdbfa69" (UID: "576f4074-7fe9-42cb-810b-bb18acdbfa69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:09:48.062809 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.062583    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "576f4074-7fe9-42cb-810b-bb18acdbfa69" (UID: "576f4074-7fe9-42cb-810b-bb18acdbfa69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:09:48.062809 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.062700    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-workload-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:48.062809 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.062717    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-workload-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:48.062809 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.062732    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576f4074-7fe9-42cb-810b-bb18acdbfa69-istiod-ca-cert\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:48.062809 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.062746    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-credential-socket\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:48.062809 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.062761    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-data\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:48.064188 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.064160    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576f4074-7fe9-42cb-810b-bb18acdbfa69-kube-api-access-j92xj" (OuterVolumeSpecName: "kube-api-access-j92xj") pod "576f4074-7fe9-42cb-810b-bb18acdbfa69" (UID: "576f4074-7fe9-42cb-810b-bb18acdbfa69"). InnerVolumeSpecName "kube-api-access-j92xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:09:48.064432 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.064413    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "576f4074-7fe9-42cb-810b-bb18acdbfa69" (UID: "576f4074-7fe9-42cb-810b-bb18acdbfa69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:09:48.064482 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.064434    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "576f4074-7fe9-42cb-810b-bb18acdbfa69" (UID: "576f4074-7fe9-42cb-810b-bb18acdbfa69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 16 17:09:48.064576 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.064562    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-token" (OuterVolumeSpecName: "istio-token") pod "576f4074-7fe9-42cb-810b-bb18acdbfa69" (UID: "576f4074-7fe9-42cb-810b-bb18acdbfa69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:09:48.163132 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.163104    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-podinfo\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:48.163132 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.163130    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-envoy\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:48.163282 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.163143    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576f4074-7fe9-42cb-810b-bb18acdbfa69-istio-token\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:48.163282 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.163155    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j92xj\" (UniqueName: \"kubernetes.io/projected/576f4074-7fe9-42cb-810b-bb18acdbfa69-kube-api-access-j92xj\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:48.919518 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.919482    2556 patch_prober.go:28] interesting pod/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 17:09:48.919637 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.919557    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r" podUID="576f4074-7fe9-42cb-810b-bb18acdbfa69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 17:09:48.938082 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.938054    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"
Sep 16 17:09:48.938408 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.938094    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r" event={"ID":"576f4074-7fe9-42cb-810b-bb18acdbfa69","Type":"ContainerDied","Data":"81ada1ef4924b6d3d9672cfd146bfb7537f07ca2b3703a3a5e20096c708fae73"}
Sep 16 17:09:48.938408 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.938154    2556 scope.go:117] "RemoveContainer" containerID="73d45c78c4fc180761f3aeee8bee01c20090f56ef42df521a84fe38d357c919c"
Sep 16 17:09:48.955879 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.955856    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"]
Sep 16 17:09:48.959718 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:48.959681    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-52fd8c-openshift-default-69c94dbf4d-j2h5r"]
Sep 16 17:09:50.847832 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:50.847799    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="576f4074-7fe9-42cb-810b-bb18acdbfa69" path="/var/lib/kubelet/pods/576f4074-7fe9-42cb-810b-bb18acdbfa69/volumes"
Sep 16 17:09:53.579571 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:53.579540    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml"]
Sep 16 17:09:53.580058 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:53.579748    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml" podUID="e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a" containerName="payload-processing" containerID="cri-o://e5a2c7307d9e816e6bf0acb11216479692b941d4bd050ac57dd15264be241987" gracePeriod=30
Sep 16 17:09:53.834551 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:53.834497    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml"
Sep 16 17:09:53.901391 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:53.901355    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2jb8\" (UniqueName: \"kubernetes.io/projected/e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a-kube-api-access-f2jb8\") pod \"e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a\" (UID: \"e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a\") "
Sep 16 17:09:53.901583 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:53.901452    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a-plugins-config-volume\") pod \"e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a\" (UID: \"e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a\") "
Sep 16 17:09:53.901843 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:53.901819    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a" (UID: "e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:09:53.903482 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:53.903452    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a-kube-api-access-f2jb8" (OuterVolumeSpecName: "kube-api-access-f2jb8") pod "e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a" (UID: "e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a"). InnerVolumeSpecName "kube-api-access-f2jb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:09:53.965757 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:53.965723    2556 generic.go:356] "Generic (PLEG): container finished" podID="e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a" containerID="e5a2c7307d9e816e6bf0acb11216479692b941d4bd050ac57dd15264be241987" exitCode=0
Sep 16 17:09:53.965922 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:53.965766    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml" event={"ID":"e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a","Type":"ContainerDied","Data":"e5a2c7307d9e816e6bf0acb11216479692b941d4bd050ac57dd15264be241987"}
Sep 16 17:09:53.965922 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:53.965804    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml" event={"ID":"e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a","Type":"ContainerDied","Data":"534b433bce13d626071cf0ba0bd5ed59fd30ed62c4dc2de4c86d145f3b0d5350"}
Sep 16 17:09:53.965922 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:53.965824    2556 scope.go:117] "RemoveContainer" containerID="e5a2c7307d9e816e6bf0acb11216479692b941d4bd050ac57dd15264be241987"
Sep 16 17:09:53.965922 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:53.965767    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml"
Sep 16 17:09:53.974279 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:53.974258    2556 scope.go:117] "RemoveContainer" containerID="e5a2c7307d9e816e6bf0acb11216479692b941d4bd050ac57dd15264be241987"
Sep 16 17:09:53.974606 ip-10-0-140-120 kubenswrapper[2556]: E0916 17:09:53.974584    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a2c7307d9e816e6bf0acb11216479692b941d4bd050ac57dd15264be241987\": container with ID starting with e5a2c7307d9e816e6bf0acb11216479692b941d4bd050ac57dd15264be241987 not found: ID does not exist" containerID="e5a2c7307d9e816e6bf0acb11216479692b941d4bd050ac57dd15264be241987"
Sep 16 17:09:53.974660 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:53.974617    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a2c7307d9e816e6bf0acb11216479692b941d4bd050ac57dd15264be241987"} err="failed to get container status \"e5a2c7307d9e816e6bf0acb11216479692b941d4bd050ac57dd15264be241987\": rpc error: code = NotFound desc = could not find container \"e5a2c7307d9e816e6bf0acb11216479692b941d4bd050ac57dd15264be241987\": container with ID starting with e5a2c7307d9e816e6bf0acb11216479692b941d4bd050ac57dd15264be241987 not found: ID does not exist"
Sep 16 17:09:53.987736 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:53.987695    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml"]
Sep 16 17:09:53.991585 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:53.991564    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-34d301-69c8766cb7-22nml"]
Sep 16 17:09:54.002483 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:54.002463    2556 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a-plugins-config-volume\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:54.002549 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:54.002483    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2jb8\" (UniqueName: \"kubernetes.io/projected/e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a-kube-api-access-f2jb8\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:09:54.847970 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:09:54.847920    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a" path="/var/lib/kubelet/pods/e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a/volumes"
Sep 16 17:12:35.703091 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:12:35.703017    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv"]
Sep 16 17:12:35.705665 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:12:35.703214    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv" podUID="a7aeffa2-475a-47d0-9556-bee3e5534928" containerName="manager" containerID="cri-o://28b0ada8e48fc956ffec86876fe823896cb949f6f4ceb554168887461e96d617" gracePeriod=10
Sep 16 17:12:36.695665 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:12:36.695636    2556 generic.go:356] "Generic (PLEG): container finished" podID="a7aeffa2-475a-47d0-9556-bee3e5534928" containerID="28b0ada8e48fc956ffec86876fe823896cb949f6f4ceb554168887461e96d617" exitCode=0
Sep 16 17:12:36.695805 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:12:36.695711    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv" event={"ID":"a7aeffa2-475a-47d0-9556-bee3e5534928","Type":"ContainerDied","Data":"28b0ada8e48fc956ffec86876fe823896cb949f6f4ceb554168887461e96d617"}
Sep 16 17:12:36.749914 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:12:36.749892    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv"
Sep 16 17:12:36.846874 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:12:36.846791    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7aeffa2-475a-47d0-9556-bee3e5534928-extensions-socket-volume\") pod \"a7aeffa2-475a-47d0-9556-bee3e5534928\" (UID: \"a7aeffa2-475a-47d0-9556-bee3e5534928\") "
Sep 16 17:12:36.847047 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:12:36.846897    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph572\" (UniqueName: \"kubernetes.io/projected/a7aeffa2-475a-47d0-9556-bee3e5534928-kube-api-access-ph572\") pod \"a7aeffa2-475a-47d0-9556-bee3e5534928\" (UID: \"a7aeffa2-475a-47d0-9556-bee3e5534928\") "
Sep 16 17:12:36.847227 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:12:36.847202    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7aeffa2-475a-47d0-9556-bee3e5534928-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a7aeffa2-475a-47d0-9556-bee3e5534928" (UID: "a7aeffa2-475a-47d0-9556-bee3e5534928"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:12:36.848817 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:12:36.848791    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7aeffa2-475a-47d0-9556-bee3e5534928-kube-api-access-ph572" (OuterVolumeSpecName: "kube-api-access-ph572") pod "a7aeffa2-475a-47d0-9556-bee3e5534928" (UID: "a7aeffa2-475a-47d0-9556-bee3e5534928"). InnerVolumeSpecName "kube-api-access-ph572". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:12:36.947972 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:12:36.947926    2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7aeffa2-475a-47d0-9556-bee3e5534928-extensions-socket-volume\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:12:36.947972 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:12:36.947969    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph572\" (UniqueName: \"kubernetes.io/projected/a7aeffa2-475a-47d0-9556-bee3e5534928-kube-api-access-ph572\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:12:37.701361 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:12:37.701329    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv"
Sep 16 17:12:37.701528 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:12:37.701357    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv" event={"ID":"a7aeffa2-475a-47d0-9556-bee3e5534928","Type":"ContainerDied","Data":"0bc48070c3c7a64ec47b5bd86c433cbd8e7f8200abb2259df4bc4c6710012d18"}
Sep 16 17:12:37.701528 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:12:37.701416    2556 scope.go:117] "RemoveContainer" containerID="28b0ada8e48fc956ffec86876fe823896cb949f6f4ceb554168887461e96d617"
Sep 16 17:12:37.722915 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:12:37.722888    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv"]
Sep 16 17:12:37.727592 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:12:37.727571    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-2mvqv"]
Sep 16 17:12:38.848249 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:12:38.848216    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7aeffa2-475a-47d0-9556-bee3e5534928" path="/var/lib/kubelet/pods/a7aeffa2-475a-47d0-9556-bee3e5534928/volumes"
Sep 16 17:13:36.382759 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.382725    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m7z4d"]
Sep 16 17:13:36.383204 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383059    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d114ea8f-b67e-44f7-a263-29d40b8cf812" containerName="istio-proxy"
Sep 16 17:13:36.383204 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383071    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d114ea8f-b67e-44f7-a263-29d40b8cf812" containerName="istio-proxy"
Sep 16 17:13:36.383204 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383084    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37794aad-c1f0-4d9b-bd68-786dab64dcd4" containerName="maas-api"
Sep 16 17:13:36.383204 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383090    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="37794aad-c1f0-4d9b-bd68-786dab64dcd4" containerName="maas-api"
Sep 16 17:13:36.383204 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383100    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e00bc8c1-aef0-45c2-a034-0ad3fe638892" containerName="istio-proxy"
Sep 16 17:13:36.383204 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383105    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e00bc8c1-aef0-45c2-a034-0ad3fe638892" containerName="istio-proxy"
Sep 16 17:13:36.383204 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383113    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d114ea8f-b67e-44f7-a263-29d40b8cf812" containerName="istio-proxy"
Sep 16 17:13:36.383204 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383119    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d114ea8f-b67e-44f7-a263-29d40b8cf812" containerName="istio-proxy"
Sep 16 17:13:36.383204 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383128    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7aeffa2-475a-47d0-9556-bee3e5534928" containerName="manager"
Sep 16 17:13:36.383204 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383133    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7aeffa2-475a-47d0-9556-bee3e5534928" containerName="manager"
Sep 16 17:13:36.383204 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383140    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a" containerName="payload-processing"
Sep 16 17:13:36.383204 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383147    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a" containerName="payload-processing"
Sep 16 17:13:36.383204 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383157    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="576f4074-7fe9-42cb-810b-bb18acdbfa69" containerName="istio-proxy"
Sep 16 17:13:36.383204 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383162    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="576f4074-7fe9-42cb-810b-bb18acdbfa69" containerName="istio-proxy"
Sep 16 17:13:36.383204 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383167    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67aff6eb-30af-4e42-ae7f-02ac193f1b19" containerName="payload-processing"
Sep 16 17:13:36.383204 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383172    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="67aff6eb-30af-4e42-ae7f-02ac193f1b19" containerName="payload-processing"
Sep 16 17:13:36.383751 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383223    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="576f4074-7fe9-42cb-810b-bb18acdbfa69" containerName="istio-proxy"
Sep 16 17:13:36.383751 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383231    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="576f4074-7fe9-42cb-810b-bb18acdbfa69" containerName="istio-proxy"
Sep 16 17:13:36.383751 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383238    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4a62cc7-1785-4a1d-92e4-4d64e5d8e27a" containerName="payload-processing"
Sep 16 17:13:36.383751 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383245    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7aeffa2-475a-47d0-9556-bee3e5534928" containerName="manager"
Sep 16 17:13:36.383751 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383250    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="37794aad-c1f0-4d9b-bd68-786dab64dcd4" containerName="maas-api"
Sep 16 17:13:36.383751 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383255    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e00bc8c1-aef0-45c2-a034-0ad3fe638892" containerName="istio-proxy"
Sep 16 17:13:36.383751 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383261    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d114ea8f-b67e-44f7-a263-29d40b8cf812" containerName="istio-proxy"
Sep 16 17:13:36.383751 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383266    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="67aff6eb-30af-4e42-ae7f-02ac193f1b19" containerName="payload-processing"
Sep 16 17:13:36.383751 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.383271    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d114ea8f-b67e-44f7-a263-29d40b8cf812" containerName="istio-proxy"
Sep 16 17:13:36.386187 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.386171    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m7z4d"
Sep 16 17:13:36.390448 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.390422    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-ngnlg\""
Sep 16 17:13:36.402118 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.402094    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m7z4d"]
Sep 16 17:13:36.423734 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.423700    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvllf\" (UniqueName: \"kubernetes.io/projected/d0140c4f-ab3b-4aa4-b206-c45a1a97e7f2-kube-api-access-cvllf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m7z4d\" (UID: \"d0140c4f-ab3b-4aa4-b206-c45a1a97e7f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m7z4d"
Sep 16 17:13:36.423851 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.423784    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0140c4f-ab3b-4aa4-b206-c45a1a97e7f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m7z4d\" (UID: \"d0140c4f-ab3b-4aa4-b206-c45a1a97e7f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m7z4d"
Sep 16 17:13:36.524333 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.524302    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0140c4f-ab3b-4aa4-b206-c45a1a97e7f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m7z4d\" (UID: \"d0140c4f-ab3b-4aa4-b206-c45a1a97e7f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m7z4d"
Sep 16 17:13:36.524483 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.524344    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvllf\" (UniqueName: \"kubernetes.io/projected/d0140c4f-ab3b-4aa4-b206-c45a1a97e7f2-kube-api-access-cvllf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m7z4d\" (UID: \"d0140c4f-ab3b-4aa4-b206-c45a1a97e7f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m7z4d"
Sep 16 17:13:36.524642 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.524622    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d0140c4f-ab3b-4aa4-b206-c45a1a97e7f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m7z4d\" (UID: \"d0140c4f-ab3b-4aa4-b206-c45a1a97e7f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m7z4d"
Sep 16 17:13:36.532495 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.532467    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvllf\" (UniqueName: \"kubernetes.io/projected/d0140c4f-ab3b-4aa4-b206-c45a1a97e7f2-kube-api-access-cvllf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-m7z4d\" (UID: \"d0140c4f-ab3b-4aa4-b206-c45a1a97e7f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m7z4d"
Sep 16 17:13:36.697510 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.697480    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m7z4d"
Sep 16 17:13:36.973596 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:36.973567    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m7z4d"]
Sep 16 17:13:36.976113 ip-10-0-140-120 kubenswrapper[2556]: W0916 17:13:36.976083    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0140c4f_ab3b_4aa4_b206_c45a1a97e7f2.slice/crio-2b365a6d0dd8e366ae003859f4b217bd60acb386c0c59441c8fa5754ae316f24 WatchSource:0}: Error finding container 2b365a6d0dd8e366ae003859f4b217bd60acb386c0c59441c8fa5754ae316f24: Status 404 returned error can't find the container with id 2b365a6d0dd8e366ae003859f4b217bd60acb386c0c59441c8fa5754ae316f24
Sep 16 17:13:37.966281 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:37.966238    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m7z4d" event={"ID":"d0140c4f-ab3b-4aa4-b206-c45a1a97e7f2","Type":"ContainerStarted","Data":"813dd117f6be16a6acc5064ba9361d3caf4f230f9d564a5db8fdcbc5ca67b13a"}
Sep 16 17:13:37.966674 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:37.966312    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m7z4d"
Sep 16 17:13:37.966674 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:37.966338    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m7z4d" event={"ID":"d0140c4f-ab3b-4aa4-b206-c45a1a97e7f2","Type":"ContainerStarted","Data":"2b365a6d0dd8e366ae003859f4b217bd60acb386c0c59441c8fa5754ae316f24"}
Sep 16 17:13:37.992099 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:37.992048    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m7z4d" podStartSLOduration=1.992034707 podStartE2EDuration="1.992034707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:13:37.990895859 +0000 UTC m=+2217.752857306" watchObservedRunningTime="2026-09-16 17:13:37.992034707 +0000 UTC m=+2217.753996153"
Sep 16 17:13:48.973340 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:13:48.973306    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-m7z4d"
Sep 16 17:14:44.321588 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.321552    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"]
Sep 16 17:14:44.322131 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.322086    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e00bc8c1-aef0-45c2-a034-0ad3fe638892" containerName="istio-proxy"
Sep 16 17:14:44.322131 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.322106    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e00bc8c1-aef0-45c2-a034-0ad3fe638892" containerName="istio-proxy"
Sep 16 17:14:44.322250 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.322134    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="576f4074-7fe9-42cb-810b-bb18acdbfa69" containerName="istio-proxy"
Sep 16 17:14:44.322250 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.322143    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="576f4074-7fe9-42cb-810b-bb18acdbfa69" containerName="istio-proxy"
Sep 16 17:14:44.322250 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.322240    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e00bc8c1-aef0-45c2-a034-0ad3fe638892" containerName="istio-proxy"
Sep 16 17:14:44.325139 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.325117    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:44.328043 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.328023    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-8c71e597-kserve-self-signed-certs\""
Sep 16 17:14:44.345270 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.345239    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"]
Sep 16 17:14:44.370050 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.370025    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-dshm\") pod \"e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") " pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:44.370167 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.370065    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb038c4d-f10d-4c52-8e98-677a045c46f3-tls-certs\") pod \"e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") " pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:44.370167 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.370102    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-tmp-dir\") pod \"e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") " pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:44.370287 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.370173    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-model-cache\") pod \"e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") " pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:44.370287 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.370220    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-home\") pod \"e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") " pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:44.370287 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.370265    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgdbf\" (UniqueName: \"kubernetes.io/projected/cb038c4d-f10d-4c52-8e98-677a045c46f3-kube-api-access-cgdbf\") pod \"e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") " pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:44.471027 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.470998    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb038c4d-f10d-4c52-8e98-677a045c46f3-tls-certs\") pod \"e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") " pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:44.471144 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.471051    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-tmp-dir\") pod \"e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") " pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:44.471144 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.471087    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-model-cache\") pod \"e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") " pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:44.471144 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.471119    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-home\") pod \"e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") " pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:44.471432 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.471403    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgdbf\" (UniqueName: \"kubernetes.io/projected/cb038c4d-f10d-4c52-8e98-677a045c46f3-kube-api-access-cgdbf\") pod \"e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") " pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:44.471559 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.471481    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-dshm\") pod \"e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") " pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:44.471559 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.471552    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-tmp-dir\") pod \"e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") " pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:44.471684 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.471579    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-model-cache\") pod \"e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") " pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:44.471684 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.471614    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-home\") pod \"e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") " pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:44.473565 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.473536    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-dshm\") pod \"e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") " pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:44.473565 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.473560    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb038c4d-f10d-4c52-8e98-677a045c46f3-tls-certs\") pod \"e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") " pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:44.484383 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.484362    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgdbf\" (UniqueName: \"kubernetes.io/projected/cb038c4d-f10d-4c52-8e98-677a045c46f3-kube-api-access-cgdbf\") pod \"e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") " pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:44.634529 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.634454    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:44.912098 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:44.912072    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"]
Sep 16 17:14:44.914633 ip-10-0-140-120 kubenswrapper[2556]: W0916 17:14:44.914606    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb038c4d_f10d_4c52_8e98_677a045c46f3.slice/crio-ca7923e77266d6e343f2828686c9aa6aa41790864ec467b4d49375027ae36594 WatchSource:0}: Error finding container ca7923e77266d6e343f2828686c9aa6aa41790864ec467b4d49375027ae36594: Status 404 returned error can't find the container with id ca7923e77266d6e343f2828686c9aa6aa41790864ec467b4d49375027ae36594
Sep 16 17:14:45.270006 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:45.269927    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v" event={"ID":"cb038c4d-f10d-4c52-8e98-677a045c46f3","Type":"ContainerStarted","Data":"7f92a937f6294893794432c538a2c834fa9e86e7d8deb7492b92878fd47d1ef3"}
Sep 16 17:14:45.270006 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:45.270011    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v" event={"ID":"cb038c4d-f10d-4c52-8e98-677a045c46f3","Type":"ContainerStarted","Data":"ca7923e77266d6e343f2828686c9aa6aa41790864ec467b4d49375027ae36594"}
Sep 16 17:14:54.634902 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:54.634867    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:54.634902 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:54.634912    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:54.648322 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:54.648295    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:14:54.664570 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:54.664519    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v" podStartSLOduration=10.664501839 podStartE2EDuration="10.664501839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:14:45.298732228 +0000 UTC m=+2285.060693674" watchObservedRunningTime="2026-09-16 17:14:54.664501839 +0000 UTC m=+2294.426463277"
Sep 16 17:14:55.327274 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:14:55.327240    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:15:02.376889 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:02.376848    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"]
Sep 16 17:15:17.395579 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:17.395510    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v" podUID="cb038c4d-f10d-4c52-8e98-677a045c46f3" containerName="main" containerID="cri-o://7f92a937f6294893794432c538a2c834fa9e86e7d8deb7492b92878fd47d1ef3" gracePeriod=45
Sep 16 17:15:17.638437 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:17.638415    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:15:17.763751 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:17.763720    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-tmp-dir\") pod \"cb038c4d-f10d-4c52-8e98-677a045c46f3\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") "
Sep 16 17:15:17.763917 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:17.763763    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-dshm\") pod \"cb038c4d-f10d-4c52-8e98-677a045c46f3\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") "
Sep 16 17:15:17.763917 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:17.763808    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-model-cache\") pod \"cb038c4d-f10d-4c52-8e98-677a045c46f3\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") "
Sep 16 17:15:17.763917 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:17.763838    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb038c4d-f10d-4c52-8e98-677a045c46f3-tls-certs\") pod \"cb038c4d-f10d-4c52-8e98-677a045c46f3\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") "
Sep 16 17:15:17.763917 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:17.763856    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-home\") pod \"cb038c4d-f10d-4c52-8e98-677a045c46f3\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") "
Sep 16 17:15:17.763917 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:17.763875    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgdbf\" (UniqueName: \"kubernetes.io/projected/cb038c4d-f10d-4c52-8e98-677a045c46f3-kube-api-access-cgdbf\") pod \"cb038c4d-f10d-4c52-8e98-677a045c46f3\" (UID: \"cb038c4d-f10d-4c52-8e98-677a045c46f3\") "
Sep 16 17:15:17.764227 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:17.764064    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cb038c4d-f10d-4c52-8e98-677a045c46f3" (UID: "cb038c4d-f10d-4c52-8e98-677a045c46f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:15:17.764227 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:17.764135    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "cb038c4d-f10d-4c52-8e98-677a045c46f3" (UID: "cb038c4d-f10d-4c52-8e98-677a045c46f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:15:17.764227 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:17.764173    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-home" (OuterVolumeSpecName: "home") pod "cb038c4d-f10d-4c52-8e98-677a045c46f3" (UID: "cb038c4d-f10d-4c52-8e98-677a045c46f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:15:17.766168 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:17.766141    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-dshm" (OuterVolumeSpecName: "dshm") pod "cb038c4d-f10d-4c52-8e98-677a045c46f3" (UID: "cb038c4d-f10d-4c52-8e98-677a045c46f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 17:15:17.766256 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:17.766244    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb038c4d-f10d-4c52-8e98-677a045c46f3-kube-api-access-cgdbf" (OuterVolumeSpecName: "kube-api-access-cgdbf") pod "cb038c4d-f10d-4c52-8e98-677a045c46f3" (UID: "cb038c4d-f10d-4c52-8e98-677a045c46f3"). InnerVolumeSpecName "kube-api-access-cgdbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:15:17.766318 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:17.766278    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb038c4d-f10d-4c52-8e98-677a045c46f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cb038c4d-f10d-4c52-8e98-677a045c46f3" (UID: "cb038c4d-f10d-4c52-8e98-677a045c46f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:15:17.865138 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:17.865112    2556 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-model-cache\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:15:17.865138 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:17.865136    2556 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb038c4d-f10d-4c52-8e98-677a045c46f3-tls-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:15:17.865138 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:17.865145    2556 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-home\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:15:17.865328 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:17.865153    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgdbf\" (UniqueName: \"kubernetes.io/projected/cb038c4d-f10d-4c52-8e98-677a045c46f3-kube-api-access-cgdbf\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:15:17.865328 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:17.865162    2556 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-tmp-dir\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:15:17.865328 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:17.865170    2556 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb038c4d-f10d-4c52-8e98-677a045c46f3-dshm\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:15:18.422644 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:18.422604    2556 generic.go:356] "Generic (PLEG): container finished" podID="cb038c4d-f10d-4c52-8e98-677a045c46f3" containerID="7f92a937f6294893794432c538a2c834fa9e86e7d8deb7492b92878fd47d1ef3" exitCode=0
Sep 16 17:15:18.423099 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:18.422652    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"
Sep 16 17:15:18.423099 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:18.422685    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v" event={"ID":"cb038c4d-f10d-4c52-8e98-677a045c46f3","Type":"ContainerDied","Data":"7f92a937f6294893794432c538a2c834fa9e86e7d8deb7492b92878fd47d1ef3"}
Sep 16 17:15:18.423099 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:18.422733    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v" event={"ID":"cb038c4d-f10d-4c52-8e98-677a045c46f3","Type":"ContainerDied","Data":"ca7923e77266d6e343f2828686c9aa6aa41790864ec467b4d49375027ae36594"}
Sep 16 17:15:18.423099 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:18.422751    2556 scope.go:117] "RemoveContainer" containerID="7f92a937f6294893794432c538a2c834fa9e86e7d8deb7492b92878fd47d1ef3"
Sep 16 17:15:18.431732 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:18.431715    2556 scope.go:117] "RemoveContainer" containerID="7f92a937f6294893794432c538a2c834fa9e86e7d8deb7492b92878fd47d1ef3"
Sep 16 17:15:18.432020 ip-10-0-140-120 kubenswrapper[2556]: E0916 17:15:18.431985    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f92a937f6294893794432c538a2c834fa9e86e7d8deb7492b92878fd47d1ef3\": container with ID starting with 7f92a937f6294893794432c538a2c834fa9e86e7d8deb7492b92878fd47d1ef3 not found: ID does not exist" containerID="7f92a937f6294893794432c538a2c834fa9e86e7d8deb7492b92878fd47d1ef3"
Sep 16 17:15:18.432085 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:18.432027    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f92a937f6294893794432c538a2c834fa9e86e7d8deb7492b92878fd47d1ef3"} err="failed to get container status \"7f92a937f6294893794432c538a2c834fa9e86e7d8deb7492b92878fd47d1ef3\": rpc error: code = NotFound desc = could not find container \"7f92a937f6294893794432c538a2c834fa9e86e7d8deb7492b92878fd47d1ef3\": container with ID starting with 7f92a937f6294893794432c538a2c834fa9e86e7d8deb7492b92878fd47d1ef3 not found: ID does not exist"
Sep 16 17:15:18.443259 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:18.443233    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"]
Sep 16 17:15:18.444694 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:18.444674    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-8c71e597-kserve-7cffcdb5f-6tb4v"]
Sep 16 17:15:18.848966 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:18.848902    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb038c4d-f10d-4c52-8e98-677a045c46f3" path="/var/lib/kubelet/pods/cb038c4d-f10d-4c52-8e98-677a045c46f3/volumes"
Sep 16 17:15:21.374959 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.374907    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5449d58978-km7rr"]
Sep 16 17:15:21.375453 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.375433    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb038c4d-f10d-4c52-8e98-677a045c46f3" containerName="main"
Sep 16 17:15:21.375551 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.375458    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb038c4d-f10d-4c52-8e98-677a045c46f3" containerName="main"
Sep 16 17:15:21.375594 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.375587    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb038c4d-f10d-4c52-8e98-677a045c46f3" containerName="main"
Sep 16 17:15:21.380752 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.380666    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5449d58978-km7rr"
Sep 16 17:15:21.390709 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.390677    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5449d58978-km7rr"]
Sep 16 17:15:21.496363 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.496328    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4adb113b-2127-40cb-82b0-e272186f4e76-maas-api-tls\") pod \"maas-api-5449d58978-km7rr\" (UID: \"4adb113b-2127-40cb-82b0-e272186f4e76\") " pod="odh-ai-gateway-infra/maas-api-5449d58978-km7rr"
Sep 16 17:15:21.496363 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.496365    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4adb113b-2127-40cb-82b0-e272186f4e76-metrics-certs\") pod \"maas-api-5449d58978-km7rr\" (UID: \"4adb113b-2127-40cb-82b0-e272186f4e76\") " pod="odh-ai-gateway-infra/maas-api-5449d58978-km7rr"
Sep 16 17:15:21.496574 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.496440    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cws5\" (UniqueName: \"kubernetes.io/projected/4adb113b-2127-40cb-82b0-e272186f4e76-kube-api-access-5cws5\") pod \"maas-api-5449d58978-km7rr\" (UID: \"4adb113b-2127-40cb-82b0-e272186f4e76\") " pod="odh-ai-gateway-infra/maas-api-5449d58978-km7rr"
Sep 16 17:15:21.496574 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.496474    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4adb113b-2127-40cb-82b0-e272186f4e76-openshift-service-ca\") pod \"maas-api-5449d58978-km7rr\" (UID: \"4adb113b-2127-40cb-82b0-e272186f4e76\") " pod="odh-ai-gateway-infra/maas-api-5449d58978-km7rr"
Sep 16 17:15:21.597898 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.597862    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cws5\" (UniqueName: \"kubernetes.io/projected/4adb113b-2127-40cb-82b0-e272186f4e76-kube-api-access-5cws5\") pod \"maas-api-5449d58978-km7rr\" (UID: \"4adb113b-2127-40cb-82b0-e272186f4e76\") " pod="odh-ai-gateway-infra/maas-api-5449d58978-km7rr"
Sep 16 17:15:21.597898 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.597900    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4adb113b-2127-40cb-82b0-e272186f4e76-openshift-service-ca\") pod \"maas-api-5449d58978-km7rr\" (UID: \"4adb113b-2127-40cb-82b0-e272186f4e76\") " pod="odh-ai-gateway-infra/maas-api-5449d58978-km7rr"
Sep 16 17:15:21.598129 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.597986    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4adb113b-2127-40cb-82b0-e272186f4e76-maas-api-tls\") pod \"maas-api-5449d58978-km7rr\" (UID: \"4adb113b-2127-40cb-82b0-e272186f4e76\") " pod="odh-ai-gateway-infra/maas-api-5449d58978-km7rr"
Sep 16 17:15:21.598129 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.598021    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4adb113b-2127-40cb-82b0-e272186f4e76-metrics-certs\") pod \"maas-api-5449d58978-km7rr\" (UID: \"4adb113b-2127-40cb-82b0-e272186f4e76\") " pod="odh-ai-gateway-infra/maas-api-5449d58978-km7rr"
Sep 16 17:15:21.598551 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.598522    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4adb113b-2127-40cb-82b0-e272186f4e76-openshift-service-ca\") pod \"maas-api-5449d58978-km7rr\" (UID: \"4adb113b-2127-40cb-82b0-e272186f4e76\") " pod="odh-ai-gateway-infra/maas-api-5449d58978-km7rr"
Sep 16 17:15:21.600274 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.600250    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4adb113b-2127-40cb-82b0-e272186f4e76-maas-api-tls\") pod \"maas-api-5449d58978-km7rr\" (UID: \"4adb113b-2127-40cb-82b0-e272186f4e76\") " pod="odh-ai-gateway-infra/maas-api-5449d58978-km7rr"
Sep 16 17:15:21.600448 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.600425    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4adb113b-2127-40cb-82b0-e272186f4e76-metrics-certs\") pod \"maas-api-5449d58978-km7rr\" (UID: \"4adb113b-2127-40cb-82b0-e272186f4e76\") " pod="odh-ai-gateway-infra/maas-api-5449d58978-km7rr"
Sep 16 17:15:21.606078 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.606057    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cws5\" (UniqueName: \"kubernetes.io/projected/4adb113b-2127-40cb-82b0-e272186f4e76-kube-api-access-5cws5\") pod \"maas-api-5449d58978-km7rr\" (UID: \"4adb113b-2127-40cb-82b0-e272186f4e76\") " pod="odh-ai-gateway-infra/maas-api-5449d58978-km7rr"
Sep 16 17:15:21.697379 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.697347    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5449d58978-km7rr"
Sep 16 17:15:21.972040 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.972011    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5449d58978-km7rr"]
Sep 16 17:15:21.974755 ip-10-0-140-120 kubenswrapper[2556]: W0916 17:15:21.974723    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4adb113b_2127_40cb_82b0_e272186f4e76.slice/crio-6092c218e6ccc3c3604d2a16e213eec90cf71746db7ad882312976fb000a16f3 WatchSource:0}: Error finding container 6092c218e6ccc3c3604d2a16e213eec90cf71746db7ad882312976fb000a16f3: Status 404 returned error can't find the container with id 6092c218e6ccc3c3604d2a16e213eec90cf71746db7ad882312976fb000a16f3
Sep 16 17:15:21.976134 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:21.976116    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 17:15:22.443444 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:22.443413    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5449d58978-km7rr" event={"ID":"4adb113b-2127-40cb-82b0-e272186f4e76","Type":"ContainerStarted","Data":"6092c218e6ccc3c3604d2a16e213eec90cf71746db7ad882312976fb000a16f3"}
Sep 16 17:15:23.449695 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:23.449656    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5449d58978-km7rr" event={"ID":"4adb113b-2127-40cb-82b0-e272186f4e76","Type":"ContainerStarted","Data":"26edb6b407f248f48ebb4bd1458bcf00dbbff6450d744fb5629ea4bbf683af87"}
Sep 16 17:15:23.450137 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:23.449784    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5449d58978-km7rr"
Sep 16 17:15:23.467881 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:23.467835    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5449d58978-km7rr" podStartSLOduration=2.041082712 podStartE2EDuration="2.467821565s" totalImagesPullingTime="426.738853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 17:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:15:23.466579672 +0000 UTC m=+2323.228541125" watchObservedRunningTime="2026-09-16 17:15:23.467821565 +0000 UTC m=+2323.229783075"
Sep 16 17:15:29.459361 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:29.459329    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5449d58978-km7rr"
Sep 16 17:15:29.517252 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:29.517218    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"]
Sep 16 17:15:29.517489 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:29.517463    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f" podUID="5df3cb22-5542-468b-b4e8-53f3dbadf022" containerName="maas-api" containerID="cri-o://0b96b72294528bb5393fc5435fdae1c0101ff906084c73aabef82776c8ef7478" gracePeriod=30
Sep 16 17:15:29.818680 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:29.818574    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f" podUID="5df3cb22-5542-468b-b4e8-53f3dbadf022" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.32:8443/health\": dial tcp 10.134.0.32:8443: connect: connection refused"
Sep 16 17:15:30.761653 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:30.761628    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"
Sep 16 17:15:30.776687 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:30.776662    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5df3cb22-5542-468b-b4e8-53f3dbadf022-maas-api-tls\") pod \"5df3cb22-5542-468b-b4e8-53f3dbadf022\" (UID: \"5df3cb22-5542-468b-b4e8-53f3dbadf022\") "
Sep 16 17:15:30.776795 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:30.776703    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5df3cb22-5542-468b-b4e8-53f3dbadf022-metrics-certs\") pod \"5df3cb22-5542-468b-b4e8-53f3dbadf022\" (UID: \"5df3cb22-5542-468b-b4e8-53f3dbadf022\") "
Sep 16 17:15:30.776795 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:30.776724    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-494fb\" (UniqueName: \"kubernetes.io/projected/5df3cb22-5542-468b-b4e8-53f3dbadf022-kube-api-access-494fb\") pod \"5df3cb22-5542-468b-b4e8-53f3dbadf022\" (UID: \"5df3cb22-5542-468b-b4e8-53f3dbadf022\") "
Sep 16 17:15:30.776795 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:30.776751    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df3cb22-5542-468b-b4e8-53f3dbadf022-openshift-service-ca\") pod \"5df3cb22-5542-468b-b4e8-53f3dbadf022\" (UID: \"5df3cb22-5542-468b-b4e8-53f3dbadf022\") "
Sep 16 17:15:30.777173 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:30.777149    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5df3cb22-5542-468b-b4e8-53f3dbadf022-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5df3cb22-5542-468b-b4e8-53f3dbadf022" (UID: "5df3cb22-5542-468b-b4e8-53f3dbadf022"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 17:15:30.778707 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:30.778679    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5df3cb22-5542-468b-b4e8-53f3dbadf022-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5df3cb22-5542-468b-b4e8-53f3dbadf022" (UID: "5df3cb22-5542-468b-b4e8-53f3dbadf022"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:15:30.778707 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:30.778691    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5df3cb22-5542-468b-b4e8-53f3dbadf022-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5df3cb22-5542-468b-b4e8-53f3dbadf022" (UID: "5df3cb22-5542-468b-b4e8-53f3dbadf022"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 17:15:30.778875 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:30.778692    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5df3cb22-5542-468b-b4e8-53f3dbadf022-kube-api-access-494fb" (OuterVolumeSpecName: "kube-api-access-494fb") pod "5df3cb22-5542-468b-b4e8-53f3dbadf022" (UID: "5df3cb22-5542-468b-b4e8-53f3dbadf022"). InnerVolumeSpecName "kube-api-access-494fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 17:15:30.877570 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:30.877545    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5df3cb22-5542-468b-b4e8-53f3dbadf022-maas-api-tls\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:15:30.877570 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:30.877567    2556 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5df3cb22-5542-468b-b4e8-53f3dbadf022-metrics-certs\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:15:30.877723 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:30.877576    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-494fb\" (UniqueName: \"kubernetes.io/projected/5df3cb22-5542-468b-b4e8-53f3dbadf022-kube-api-access-494fb\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:15:30.877723 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:30.877585    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5df3cb22-5542-468b-b4e8-53f3dbadf022-openshift-service-ca\") on node \"ip-10-0-140-120.ec2.internal\" DevicePath \"\""
Sep 16 17:15:31.486368 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:31.486333    2556 generic.go:356] "Generic (PLEG): container finished" podID="5df3cb22-5542-468b-b4e8-53f3dbadf022" containerID="0b96b72294528bb5393fc5435fdae1c0101ff906084c73aabef82776c8ef7478" exitCode=0
Sep 16 17:15:31.486546 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:31.486381    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"
Sep 16 17:15:31.486546 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:31.486417    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f" event={"ID":"5df3cb22-5542-468b-b4e8-53f3dbadf022","Type":"ContainerDied","Data":"0b96b72294528bb5393fc5435fdae1c0101ff906084c73aabef82776c8ef7478"}
Sep 16 17:15:31.486546 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:31.486468    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f" event={"ID":"5df3cb22-5542-468b-b4e8-53f3dbadf022","Type":"ContainerDied","Data":"606736a05dcff0a2891163279654cda08fe359ddc4941bd07592fdce0def1875"}
Sep 16 17:15:31.486546 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:31.486491    2556 scope.go:117] "RemoveContainer" containerID="0b96b72294528bb5393fc5435fdae1c0101ff906084c73aabef82776c8ef7478"
Sep 16 17:15:31.494669 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:31.494652    2556 scope.go:117] "RemoveContainer" containerID="0b96b72294528bb5393fc5435fdae1c0101ff906084c73aabef82776c8ef7478"
Sep 16 17:15:31.494910 ip-10-0-140-120 kubenswrapper[2556]: E0916 17:15:31.494891    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b96b72294528bb5393fc5435fdae1c0101ff906084c73aabef82776c8ef7478\": container with ID starting with 0b96b72294528bb5393fc5435fdae1c0101ff906084c73aabef82776c8ef7478 not found: ID does not exist" containerID="0b96b72294528bb5393fc5435fdae1c0101ff906084c73aabef82776c8ef7478"
Sep 16 17:15:31.494989 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:31.494917    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b96b72294528bb5393fc5435fdae1c0101ff906084c73aabef82776c8ef7478"} err="failed to get container status \"0b96b72294528bb5393fc5435fdae1c0101ff906084c73aabef82776c8ef7478\": rpc error: code = NotFound desc = could not find container \"0b96b72294528bb5393fc5435fdae1c0101ff906084c73aabef82776c8ef7478\": container with ID starting with 0b96b72294528bb5393fc5435fdae1c0101ff906084c73aabef82776c8ef7478 not found: ID does not exist"
Sep 16 17:15:31.502205 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:31.502183    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"]
Sep 16 17:15:31.506696 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:31.506679    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5665c7f4f4-mpw4f"]
Sep 16 17:15:32.848112 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:32.848079    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5df3cb22-5542-468b-b4e8-53f3dbadf022" path="/var/lib/kubelet/pods/5df3cb22-5542-468b-b4e8-53f3dbadf022/volumes"
Sep 16 17:15:56.274980 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:56.274932    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vdj6m_9a522445-b1b9-4528-b4bf-5144cd3aae87/server/0.log"
Sep 16 17:15:56.387453 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:56.387421    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rsbqk_333d82ae-12bc-47a8-8a8a-f6d4621b53a2/manager/0.log"
Sep 16 17:15:56.495498 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:56.495472    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54d7c7f658-2rwx4_6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0/copy-manifests/0.log"
Sep 16 17:15:56.507298 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:56.507267    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54d7c7f658-2rwx4_6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0/manager/0.log"
Sep 16 17:15:57.176643 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:57.176611    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5449d58978-km7rr_4adb113b-2127-40cb-82b0-e272186f4e76/maas-api/0.log"
Sep 16 17:15:57.500416 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:57.500391    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-c4m7h_7df80f59-1527-4045-98d0-32cbb77dc7db/postgres/0.log"
Sep 16 17:15:58.838003 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:58.837973    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-m7z4d_d0140c4f-ab3b-4aa4-b206-c45a1a97e7f2/manager/0.log"
Sep 16 17:15:59.093236 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:59.093158    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-82jzb_49273cd9-7389-4997-ae50-4d38382174c8/manager/0.log"
Sep 16 17:15:59.559646 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:59.559616    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-27bhq_81ff471a-3fb8-4c64-bfff-fd1be577fe05/discovery/0.log"
Sep 16 17:15:59.768687 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:15:59.768655    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b966d4c98-vw7ts_f27d87e1-4421-46fb-812b-f159f43dc302/kube-auth-proxy/0.log"
Sep 16 17:16:00.127755 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:00.127722    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mpcpf_bb67f300-0327-4e84-9998-045e6b646f34/payload-processing/0.log"
Sep 16 17:16:00.793448 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:00.793422    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-xw5hg_2cf92945-7a38-4b2c-86bc-761d2187a2aa/main/0.log"
Sep 16 17:16:01.108171 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:01.108092    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-9gh99_ee454389-f8a1-4351-ae5b-2c342a3c63a0/main/0.log"
Sep 16 17:16:10.303499 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:10.303444    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kmw95_cd9b4fb9-8db8-432c-a6d6-d7a1d32f7792/global-pull-secret-syncer/0.log"
Sep 16 17:16:10.452424 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:10.452394    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tdf5s_0b7cc8be-e4bc-477e-8166-f539c64992a4/konnectivity-agent/0.log"
Sep 16 17:16:10.564894 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:10.564821    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-120.ec2.internal_4181901193613f3e77dd46929737ec7c/haproxy/0.log"
Sep 16 17:16:14.969969 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:14.969929    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-m7z4d_d0140c4f-ab3b-4aa4-b206-c45a1a97e7f2/manager/0.log"
Sep 16 17:16:15.085751 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:15.085722    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-82jzb_49273cd9-7389-4997-ae50-4d38382174c8/manager/0.log"
Sep 16 17:16:16.938231 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:16.938202    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-xg2dr_5b4d8056-88a6-4900-889a-ef4abb61518a/monitoring-plugin/0.log"
Sep 16 17:16:16.977003 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:16.976966    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5w9v5_5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf/node-exporter/0.log"
Sep 16 17:16:16.999806 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:16.999781    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5w9v5_5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf/kube-rbac-proxy/0.log"
Sep 16 17:16:17.027915 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:17.027895    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5w9v5_5c1f9fe1-0c7f-4c63-b4c6-e155e5a079bf/init-textfile/0.log"
Sep 16 17:16:17.661751 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:17.661724    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-qkmb6_6c1388e8-16e7-4f78-b30e-033c4f9b5256/prometheus-operator/0.log"
Sep 16 17:16:17.690351 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:17.690323    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-qkmb6_6c1388e8-16e7-4f78-b30e-033c4f9b5256/kube-rbac-proxy/0.log"
Sep 16 17:16:17.834098 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:17.834072    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9c68594d-fmnln_707aeedc-ba66-41c1-a191-409f10661609/thanos-query/0.log"
Sep 16 17:16:17.856719 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:17.856692    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9c68594d-fmnln_707aeedc-ba66-41c1-a191-409f10661609/kube-rbac-proxy-web/0.log"
Sep 16 17:16:17.880825 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:17.880800    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9c68594d-fmnln_707aeedc-ba66-41c1-a191-409f10661609/kube-rbac-proxy/0.log"
Sep 16 17:16:17.904475 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:17.904458    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9c68594d-fmnln_707aeedc-ba66-41c1-a191-409f10661609/prom-label-proxy/0.log"
Sep 16 17:16:17.933526 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:17.933475    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9c68594d-fmnln_707aeedc-ba66-41c1-a191-409f10661609/kube-rbac-proxy-rules/0.log"
Sep 16 17:16:17.955410 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:17.955387    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-9c68594d-fmnln_707aeedc-ba66-41c1-a191-409f10661609/kube-rbac-proxy-metrics/0.log"
Sep 16 17:16:19.941574 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:19.941544    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"]
Sep 16 17:16:19.941929 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:19.941906    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5df3cb22-5542-468b-b4e8-53f3dbadf022" containerName="maas-api"
Sep 16 17:16:19.941929 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:19.941921    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5df3cb22-5542-468b-b4e8-53f3dbadf022" containerName="maas-api"
Sep 16 17:16:19.942024 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:19.942005    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5df3cb22-5542-468b-b4e8-53f3dbadf022" containerName="maas-api"
Sep 16 17:16:19.944858 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:19.944842    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"
Sep 16 17:16:19.947752 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:19.947734    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mqgpr\"/\"kube-root-ca.crt\""
Sep 16 17:16:19.947850 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:19.947733    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mqgpr\"/\"default-dockercfg-bb76x\""
Sep 16 17:16:19.947925 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:19.947910    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mqgpr\"/\"openshift-service-ca.crt\""
Sep 16 17:16:19.955037 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:19.955016    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"]
Sep 16 17:16:19.988771 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:19.988743    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6db30a3b-6c29-4b1a-b67d-21d568fa99ae-podres\") pod \"perf-node-gather-daemonset-pjtff\" (UID: \"6db30a3b-6c29-4b1a-b67d-21d568fa99ae\") " pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"
Sep 16 17:16:19.988875 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:19.988791    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6db30a3b-6c29-4b1a-b67d-21d568fa99ae-lib-modules\") pod \"perf-node-gather-daemonset-pjtff\" (UID: \"6db30a3b-6c29-4b1a-b67d-21d568fa99ae\") " pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"
Sep 16 17:16:19.988923 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:19.988871    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm45q\" (UniqueName: \"kubernetes.io/projected/6db30a3b-6c29-4b1a-b67d-21d568fa99ae-kube-api-access-rm45q\") pod \"perf-node-gather-daemonset-pjtff\" (UID: \"6db30a3b-6c29-4b1a-b67d-21d568fa99ae\") " pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"
Sep 16 17:16:19.988923 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:19.988906    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6db30a3b-6c29-4b1a-b67d-21d568fa99ae-sys\") pod \"perf-node-gather-daemonset-pjtff\" (UID: \"6db30a3b-6c29-4b1a-b67d-21d568fa99ae\") " pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"
Sep 16 17:16:19.989041 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:19.988923    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6db30a3b-6c29-4b1a-b67d-21d568fa99ae-proc\") pod \"perf-node-gather-daemonset-pjtff\" (UID: \"6db30a3b-6c29-4b1a-b67d-21d568fa99ae\") " pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"
Sep 16 17:16:20.090179 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:20.090146    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6db30a3b-6c29-4b1a-b67d-21d568fa99ae-lib-modules\") pod \"perf-node-gather-daemonset-pjtff\" (UID: \"6db30a3b-6c29-4b1a-b67d-21d568fa99ae\") " pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"
Sep 16 17:16:20.090312 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:20.090207    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm45q\" (UniqueName: \"kubernetes.io/projected/6db30a3b-6c29-4b1a-b67d-21d568fa99ae-kube-api-access-rm45q\") pod \"perf-node-gather-daemonset-pjtff\" (UID: \"6db30a3b-6c29-4b1a-b67d-21d568fa99ae\") " pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"
Sep 16 17:16:20.090312 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:20.090235    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6db30a3b-6c29-4b1a-b67d-21d568fa99ae-sys\") pod \"perf-node-gather-daemonset-pjtff\" (UID: \"6db30a3b-6c29-4b1a-b67d-21d568fa99ae\") " pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"
Sep 16 17:16:20.090312 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:20.090257    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6db30a3b-6c29-4b1a-b67d-21d568fa99ae-proc\") pod \"perf-node-gather-daemonset-pjtff\" (UID: \"6db30a3b-6c29-4b1a-b67d-21d568fa99ae\") " pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"
Sep 16 17:16:20.090471 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:20.090315    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6db30a3b-6c29-4b1a-b67d-21d568fa99ae-lib-modules\") pod \"perf-node-gather-daemonset-pjtff\" (UID: \"6db30a3b-6c29-4b1a-b67d-21d568fa99ae\") " pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"
Sep 16 17:16:20.090471 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:20.090344    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6db30a3b-6c29-4b1a-b67d-21d568fa99ae-sys\") pod \"perf-node-gather-daemonset-pjtff\" (UID: \"6db30a3b-6c29-4b1a-b67d-21d568fa99ae\") " pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"
Sep 16 17:16:20.090471 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:20.090386    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6db30a3b-6c29-4b1a-b67d-21d568fa99ae-podres\") pod \"perf-node-gather-daemonset-pjtff\" (UID: \"6db30a3b-6c29-4b1a-b67d-21d568fa99ae\") " pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"
Sep 16 17:16:20.090471 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:20.090392    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6db30a3b-6c29-4b1a-b67d-21d568fa99ae-proc\") pod \"perf-node-gather-daemonset-pjtff\" (UID: \"6db30a3b-6c29-4b1a-b67d-21d568fa99ae\") " pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"
Sep 16 17:16:20.090471 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:20.090458    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6db30a3b-6c29-4b1a-b67d-21d568fa99ae-podres\") pod \"perf-node-gather-daemonset-pjtff\" (UID: \"6db30a3b-6c29-4b1a-b67d-21d568fa99ae\") " pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"
Sep 16 17:16:20.099017 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:20.098999    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm45q\" (UniqueName: \"kubernetes.io/projected/6db30a3b-6c29-4b1a-b67d-21d568fa99ae-kube-api-access-rm45q\") pod \"perf-node-gather-daemonset-pjtff\" (UID: \"6db30a3b-6c29-4b1a-b67d-21d568fa99ae\") " pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"
Sep 16 17:16:20.255018 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:20.254930    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"
Sep 16 17:16:20.526138 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:20.526065    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-qdq8l_ccb79307-8c26-492a-8c6f-ce741345a6be/download-server/0.log"
Sep 16 17:16:20.528730 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:20.528709    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"]
Sep 16 17:16:20.531092 ip-10-0-140-120 kubenswrapper[2556]: W0916 17:16:20.531067    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6db30a3b_6c29_4b1a_b67d_21d568fa99ae.slice/crio-e16243555251ccdf679d5395ac816f26cbcec367e3a9d80d9cea11884472bfe3 WatchSource:0}: Error finding container e16243555251ccdf679d5395ac816f26cbcec367e3a9d80d9cea11884472bfe3: Status 404 returned error can't find the container with id e16243555251ccdf679d5395ac816f26cbcec367e3a9d80d9cea11884472bfe3
Sep 16 17:16:20.708246 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:20.708209    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff" event={"ID":"6db30a3b-6c29-4b1a-b67d-21d568fa99ae","Type":"ContainerStarted","Data":"25411b69e6abbf4f68626b922d0759c2ac79f65fcffc4a2f26b92e90c6829b49"}
Sep 16 17:16:20.708392 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:20.708264    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff" event={"ID":"6db30a3b-6c29-4b1a-b67d-21d568fa99ae","Type":"ContainerStarted","Data":"e16243555251ccdf679d5395ac816f26cbcec367e3a9d80d9cea11884472bfe3"}
Sep 16 17:16:20.708392 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:20.708286    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"
Sep 16 17:16:20.725412 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:20.725368    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff" podStartSLOduration=1.72535368 podStartE2EDuration="1.72535368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 17:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 17:16:20.72344712 +0000 UTC m=+2380.485408567" watchObservedRunningTime="2026-09-16 17:16:20.72535368 +0000 UTC m=+2380.487315126"
Sep 16 17:16:22.473723 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:22.473689    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z2tqs_611f8865-ebde-45ab-b546-9f0be6d42d81/dns/0.log"
Sep 16 17:16:22.494591 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:22.494565    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z2tqs_611f8865-ebde-45ab-b546-9f0be6d42d81/kube-rbac-proxy/0.log"
Sep 16 17:16:22.570015 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:22.569988    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-knvfz_02df9d68-24e0-43d9-8774-273f8b55c906/dns-node-resolver/0.log"
Sep 16 17:16:23.170349 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:23.170315    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qf2sz_321e79fa-919d-436e-a11d-a4525450a85c/node-ca/0.log"
Sep 16 17:16:24.426089 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:24.426058    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-27bhq_81ff471a-3fb8-4c64-bfff-fd1be577fe05/discovery/0.log"
Sep 16 17:16:24.481138 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:24.481114    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b966d4c98-vw7ts_f27d87e1-4421-46fb-812b-f159f43dc302/kube-auth-proxy/0.log"
Sep 16 17:16:24.597012 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:24.596980    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mpcpf_bb67f300-0327-4e84-9998-045e6b646f34/payload-processing/0.log"
Sep 16 17:16:25.520034 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:25.520005    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n4drw_01475a36-0404-44f7-b7c9-7293e3729dd4/serve-healthcheck-canary/0.log"
Sep 16 17:16:26.144654 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:26.144623    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p6srw_5dd963a7-182d-4dae-a474-aa7fc60faf2a/kube-rbac-proxy/0.log"
Sep 16 17:16:26.165896 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:26.165873    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p6srw_5dd963a7-182d-4dae-a474-aa7fc60faf2a/exporter/0.log"
Sep 16 17:16:26.187669 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:26.187645    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-p6srw_5dd963a7-182d-4dae-a474-aa7fc60faf2a/extractor/0.log"
Sep 16 17:16:26.722025 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:26.722000    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mqgpr/perf-node-gather-daemonset-pjtff"
Sep 16 17:16:28.716336 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:28.716306    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vdj6m_9a522445-b1b9-4528-b4bf-5144cd3aae87/server/0.log"
Sep 16 17:16:28.796099 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:28.796069    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rsbqk_333d82ae-12bc-47a8-8a8a-f6d4621b53a2/manager/0.log"
Sep 16 17:16:28.879393 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:28.879361    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54d7c7f658-2rwx4_6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0/manager/0.log"
Sep 16 17:16:28.898975 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:28.898952    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54d7c7f658-2rwx4_6d7e8dc3-46fe-4fc2-98fa-7a2b25ec46e0/copy-manifests/0.log"
Sep 16 17:16:30.761347 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:30.761320    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-27bhq_81ff471a-3fb8-4c64-bfff-fd1be577fe05/discovery/0.log"
Sep 16 17:16:30.786040 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:30.786008    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b966d4c98-vw7ts_f27d87e1-4421-46fb-812b-f159f43dc302/kube-auth-proxy/0.log"
Sep 16 17:16:30.879140 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:30.879099    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mpcpf_bb67f300-0327-4e84-9998-045e6b646f34/payload-processing/0.log"
Sep 16 17:16:32.789269 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:32.789238    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-27bhq_81ff471a-3fb8-4c64-bfff-fd1be577fe05/discovery/0.log"
Sep 16 17:16:32.831666 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:32.831643    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b966d4c98-vw7ts_f27d87e1-4421-46fb-812b-f159f43dc302/kube-auth-proxy/0.log"
Sep 16 17:16:32.914820 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:32.914794    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mpcpf_bb67f300-0327-4e84-9998-045e6b646f34/payload-processing/0.log"
Sep 16 17:16:36.913195 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:36.913166    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g7ksk_2009b752-ad4d-4e71-8a91-3b3b41c44725/kube-multus-additional-cni-plugins/0.log"
Sep 16 17:16:36.935990 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:36.935961    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g7ksk_2009b752-ad4d-4e71-8a91-3b3b41c44725/egress-router-binary-copy/0.log"
Sep 16 17:16:36.956913 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:36.956849    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g7ksk_2009b752-ad4d-4e71-8a91-3b3b41c44725/cni-plugins/0.log"
Sep 16 17:16:36.976758 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:36.976732    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g7ksk_2009b752-ad4d-4e71-8a91-3b3b41c44725/bond-cni-plugin/0.log"
Sep 16 17:16:36.998408 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:36.998389    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g7ksk_2009b752-ad4d-4e71-8a91-3b3b41c44725/routeoverride-cni/0.log"
Sep 16 17:16:37.019998 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:37.019982    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g7ksk_2009b752-ad4d-4e71-8a91-3b3b41c44725/whereabouts-cni-bincopy/0.log"
Sep 16 17:16:37.040574 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:37.040555    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g7ksk_2009b752-ad4d-4e71-8a91-3b3b41c44725/whereabouts-cni/0.log"
Sep 16 17:16:37.457437 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:37.457402    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lktnv_e5adc7d5-f860-4c8d-992d-963756104953/kube-multus/0.log"
Sep 16 17:16:37.579220 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:37.579185    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hk8q9_c04662fe-4176-4a0b-ac5e-89ead5941975/network-metrics-daemon/0.log"
Sep 16 17:16:37.598152 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:37.598123    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hk8q9_c04662fe-4176-4a0b-ac5e-89ead5941975/kube-rbac-proxy/0.log"
Sep 16 17:16:38.503774 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:38.503741    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n42qf_9280e2b8-f49e-4e07-b1af-a26cc9108ff9/ovn-controller/0.log"
Sep 16 17:16:38.526245 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:38.526222    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n42qf_9280e2b8-f49e-4e07-b1af-a26cc9108ff9/ovn-acl-logging/0.log"
Sep 16 17:16:38.554059 ip-10-0-140-120 kubenswrapper[2556]: I0916 17:16:38.554038    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n42qf_9280e2b8-f49e-4e07-b1af-a26cc9108ff9/kube-rbac-proxy-node/0.log"