Sep 14 12:56:16.288819 ip-10-0-136-96 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 14 12:56:16.288834 ip-10-0-136-96 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 14 12:56:16.288868 ip-10-0-136-96 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 14 12:56:16.289362 ip-10-0-136-96 systemd[1]: Failed to start Kubernetes Kubelet. Sep 14 12:56:26.482921 ip-10-0-136-96 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 14 12:56:26.482940 ip-10-0-136-96 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 1caa42143225416d9071b4e613bcc5b6 -- Sep 14 12:58:17.748641 ip-10-0-136-96 systemd[1]: Starting Kubernetes Kubelet... Sep 14 12:58:18.188094 ip-10-0-136-96 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 12:58:18.188094 ip-10-0-136-96 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 14 12:58:18.188094 ip-10-0-136-96 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 12:58:18.188094 ip-10-0-136-96 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 14 12:58:18.188094 ip-10-0-136-96 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 14 12:58:18.190536 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.190453 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 14 12:58:18.196477 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196461 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 12:58:18.196477 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196477 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 12:58:18.196548 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196482 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 12:58:18.196548 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196486 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 12:58:18.196548 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196489 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 12:58:18.196548 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196492 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 12:58:18.196548 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196495 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 12:58:18.196548 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196497 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 12:58:18.196548 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196500 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 12:58:18.196548 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196503 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 12:58:18.196548 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196505 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 12:58:18.196548 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196508 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 12:58:18.196548 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196511 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 12:58:18.196548 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196513 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 12:58:18.196548 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196516 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 12:58:18.196548 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196518 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 12:58:18.196548 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196526 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 12:58:18.196548 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196529 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 12:58:18.196548 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196532 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 12:58:18.196548 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196534 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 12:58:18.196548 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196537 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 12:58:18.196548 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196539 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 12:58:18.197031 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196542 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 12:58:18.197031 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196545 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 12:58:18.197031 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196547 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 12:58:18.197031 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196550 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 12:58:18.197031 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196553 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 12:58:18.197031 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196556 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 12:58:18.197031 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196559 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 12:58:18.197031 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196562 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 12:58:18.197031 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196565 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 12:58:18.197031 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196567 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 12:58:18.197031 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196570 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 12:58:18.197031 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196572 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 12:58:18.197031 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196575 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 12:58:18.197031 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196577 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 12:58:18.197031 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196580 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 12:58:18.197031 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196582 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 12:58:18.197031 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196585 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 12:58:18.197031 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196587 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 12:58:18.197031 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196590 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 12:58:18.197031 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196593 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 12:58:18.197551 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196595 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 12:58:18.197551 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196597 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 12:58:18.197551 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196600 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 12:58:18.197551 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196603 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 12:58:18.197551 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196606 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 12:58:18.197551 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196610 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 12:58:18.197551 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196614 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 12:58:18.197551 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196619 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 12:58:18.197551 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196623 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 12:58:18.197551 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196626 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 12:58:18.197551 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196629 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 12:58:18.197551 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196631 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 14 12:58:18.197551 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196634 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 12:58:18.197551 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196638 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 12:58:18.197551 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196641 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 12:58:18.197551 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196643 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 12:58:18.197551 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196646 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 12:58:18.197551 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196648 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 12:58:18.197551 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196651 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 12:58:18.198039 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196653 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 12:58:18.198039 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196656 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 12:58:18.198039 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196658 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 12:58:18.198039 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196661 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 12:58:18.198039 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196663 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 12:58:18.198039 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196667 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 12:58:18.198039 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196670 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 12:58:18.198039 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196672 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 12:58:18.198039 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196675 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 12:58:18.198039 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196677 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 12:58:18.198039 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196680 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 12:58:18.198039 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196683 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 12:58:18.198039 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196685 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 12:58:18.198039 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196688 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 12:58:18.198039 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196690 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 12:58:18.198039 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196693 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 12:58:18.198039 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196695 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 12:58:18.198039 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196697 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 12:58:18.198039 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196700 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 12:58:18.198039 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196702 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 12:58:18.198539 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196705 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 12:58:18.198539 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196707 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 12:58:18.198539 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196710 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 12:58:18.198539 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196712 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 12:58:18.198539 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196715 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 12:58:18.198539 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196718 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 12:58:18.198539 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196721 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 12:58:18.198539 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196724 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 12:58:18.198539 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196726 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 12:58:18.198539 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196729 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 12:58:18.198539 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196732 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 12:58:18.198539 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196734 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 12:58:18.198539 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196737 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 12:58:18.198539 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196739 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 12:58:18.198539 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196742 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 12:58:18.198539 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196744 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 12:58:18.198539 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196746 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 12:58:18.198539 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.196750 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 12:58:18.198539 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197505 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 12:58:18.199008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197513 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 12:58:18.199008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197516 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 12:58:18.199008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197520 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 12:58:18.199008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197523 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 12:58:18.199008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197527 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 12:58:18.199008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197530 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 12:58:18.199008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197534 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 12:58:18.199008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197537 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 12:58:18.199008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197540 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 12:58:18.199008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197543 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 12:58:18.199008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197546 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 12:58:18.199008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197549 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 12:58:18.199008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197553 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 12:58:18.199008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197556 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 12:58:18.199008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197559 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 12:58:18.199008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197561 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 12:58:18.199008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197564 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 14 12:58:18.199008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197567 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 12:58:18.199008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197570 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 12:58:18.199510 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197573 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 12:58:18.199510 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197576 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 12:58:18.199510 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197578 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 12:58:18.199510 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197581 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 12:58:18.199510 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197583 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 12:58:18.199510 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197586 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 12:58:18.199510 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197588 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 12:58:18.199510 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197591 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 12:58:18.199510 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197593 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 12:58:18.199510 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197596 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 12:58:18.199510 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197599 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 12:58:18.199510 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197602 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 12:58:18.199510 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197613 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 12:58:18.199510 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197616 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 12:58:18.199510 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197618 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 12:58:18.199510 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197621 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 12:58:18.199510 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197623 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 12:58:18.199510 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197626 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 12:58:18.199510 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197629 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 12:58:18.199510 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197632 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 12:58:18.200008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197634 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 12:58:18.200008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197637 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 12:58:18.200008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197639 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 12:58:18.200008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197642 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 12:58:18.200008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197645 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 12:58:18.200008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197647 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 12:58:18.200008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197649 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 12:58:18.200008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197652 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 12:58:18.200008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197654 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 12:58:18.200008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197657 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 12:58:18.200008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197660 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 12:58:18.200008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197663 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 12:58:18.200008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197665 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 12:58:18.200008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197668 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 12:58:18.200008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197670 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 12:58:18.200008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197673 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 12:58:18.200008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197675 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 12:58:18.200008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197677 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 12:58:18.200008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197681 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 12:58:18.200008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197683 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 12:58:18.200522 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197687 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 12:58:18.200522 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197690 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 12:58:18.200522 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197693 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 12:58:18.200522 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197696 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 12:58:18.200522 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197698 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 12:58:18.200522 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197701 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 12:58:18.200522 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197704 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 12:58:18.200522 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197706 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 12:58:18.200522 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197709 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 12:58:18.200522 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197711 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 12:58:18.200522 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197714 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 12:58:18.200522 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197717 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 12:58:18.200522 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197719 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 12:58:18.200522 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197722 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 12:58:18.200522 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197724 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 12:58:18.200522 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197727 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 12:58:18.200522 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197730 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 12:58:18.200522 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197733 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 12:58:18.200522 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197735 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 12:58:18.200991 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197737 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 12:58:18.200991 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197740 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 12:58:18.200991 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197742 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 12:58:18.200991 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197745 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 12:58:18.200991 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197748 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 12:58:18.200991 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197751 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 12:58:18.200991 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197753 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 12:58:18.200991 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197755 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 12:58:18.200991 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197758 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 12:58:18.200991 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197760 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 12:58:18.200991 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197763 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 12:58:18.200991 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197765 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 12:58:18.200991 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197768 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 12:58:18.200991 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197770 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 12:58:18.200991 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197773 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 12:58:18.200991 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197775 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 12:58:18.200991 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197778 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 12:58:18.200991 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197780 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 12:58:18.200991 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197782 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 12:58:18.200991 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.197785 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 12:58:18.201535 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198173 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 12:58:18.201535 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198179 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 12:58:18.201535 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198182 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 12:58:18.201535 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198185 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 12:58:18.201535 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198188 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 12:58:18.201535 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198191 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 12:58:18.201535 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198194 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 12:58:18.201535 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198196 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 12:58:18.201535 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198199 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 12:58:18.201535 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198201 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 12:58:18.201535 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198204 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 12:58:18.201535 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198206 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 12:58:18.201535 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198209 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 12:58:18.201535 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198211 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 12:58:18.201535 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198214 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 12:58:18.201535 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198216 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 12:58:18.201535 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198219 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 12:58:18.201535 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198222 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 12:58:18.201535 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198225 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 12:58:18.202020 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198228 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 12:58:18.202020 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198230 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 12:58:18.202020 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198233 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 12:58:18.202020 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198236 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 12:58:18.202020 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198238 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 12:58:18.202020 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198240 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 12:58:18.202020 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198243 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 12:58:18.202020 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198245 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 12:58:18.202020 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198248 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 12:58:18.202020 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198250 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 12:58:18.202020 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198253 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 12:58:18.202020 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198255 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 12:58:18.202020 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198257 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 12:58:18.202020 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198260 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 12:58:18.202020 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198263 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 12:58:18.202020 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198265 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 12:58:18.202020 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198268 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 12:58:18.202020 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198270 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 12:58:18.202020 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198273 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 14 12:58:18.202020 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198275 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 12:58:18.202540 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198278 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 12:58:18.202540 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198281 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 12:58:18.202540 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198285 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 12:58:18.202540 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198288 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 12:58:18.202540 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198290 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 12:58:18.202540 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198293 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 12:58:18.202540 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198296 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 12:58:18.202540 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198298 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 12:58:18.202540 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198301 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 12:58:18.202540 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198304 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 12:58:18.202540 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198306 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 12:58:18.202540 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198309 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 12:58:18.202540 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198311 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 12:58:18.202540 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198313 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 12:58:18.202540 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198315 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 12:58:18.202540 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198318 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 12:58:18.202540 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198320 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 12:58:18.202540 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198323 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 12:58:18.202540 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198325 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 12:58:18.203068 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198327 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 12:58:18.203068 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198346 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 12:58:18.203068 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198350 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 12:58:18.203068 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198353 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 12:58:18.203068 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198356 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 12:58:18.203068 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198359 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 12:58:18.203068 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198362 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 12:58:18.203068 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198365 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 12:58:18.203068 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198367 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 12:58:18.203068 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198370 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 12:58:18.203068 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198373 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 12:58:18.203068 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198376 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 12:58:18.203068 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198379 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 12:58:18.203068 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198383 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 12:58:18.203068 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198386 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 12:58:18.203068 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198389 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 12:58:18.203068 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198391 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 12:58:18.203068 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198394 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 12:58:18.203068 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198397 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 12:58:18.203068 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198399 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 12:58:18.203584 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198401 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 12:58:18.203584 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198405 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 12:58:18.203584 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198408 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 12:58:18.203584 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198410 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 12:58:18.203584 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198414 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 12:58:18.203584 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198416 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 12:58:18.203584 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198419 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 12:58:18.203584 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198421 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 12:58:18.203584 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198424 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 12:58:18.203584 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198427 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 12:58:18.203584 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198429 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 12:58:18.203584 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198431 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 12:58:18.203584 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198434 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 12:58:18.203584 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198437 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 12:58:18.203584 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198441 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 12:58:18.203584 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198444 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 12:58:18.203584 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198446 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 12:58:18.203584 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198449 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 12:58:18.203584 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198451 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 12:58:18.203584 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198453 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.198456 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198522 2564 flags.go:64] FLAG: --address="0.0.0.0" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198559 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198565 2564 flags.go:64] FLAG: --anonymous-auth="true" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198570 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198575 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198578 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198582 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198586 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198589 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198592 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198596 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198599 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198603 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198605 2564 flags.go:64] FLAG: --cgroup-root="" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198608 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198611 2564 flags.go:64] FLAG: --client-ca-file="" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198614 2564 flags.go:64] FLAG: --cloud-provider="external" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198616 2564 flags.go:64] FLAG: --cluster-dns="[]" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198621 2564 flags.go:64] FLAG: --cluster-domain="" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198624 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198626 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198629 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 14 12:58:18.204067 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198633 2564 flags.go:64] FLAG: --container-log-max-files="5" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198636 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198639 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198642 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198645 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198648 2564 flags.go:64] FLAG: --contention-profiling="false" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198651 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198653 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198656 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198659 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198663 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198666 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198669 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198671 2564 flags.go:64] FLAG: --enable-load-reader="false" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198674 2564 flags.go:64] FLAG: --enable-server="true" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198678 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198682 2564 flags.go:64] FLAG: --event-burst="100" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198686 2564 flags.go:64] FLAG: --event-qps="50" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198688 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198691 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198694 2564 flags.go:64] FLAG: --eviction-hard="" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198700 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198702 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198705 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198708 2564 flags.go:64] FLAG: --eviction-soft="" Sep 14 12:58:18.204875 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198711 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198714 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198717 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198720 2564 flags.go:64] FLAG: --experimental-mounter-path="" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198722 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198726 2564 flags.go:64] FLAG: --fail-swap-on="true" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198729 2564 flags.go:64] FLAG: --feature-gates="" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198732 2564 flags.go:64] FLAG: --file-check-frequency="20s" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198735 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198738 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198741 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198744 2564 flags.go:64] FLAG: --healthz-port="10248" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198747 2564 flags.go:64] FLAG: --help="false" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198750 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-136-96.ec2.internal" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198753 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198756 2564 flags.go:64] FLAG: --http-check-frequency="20s" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198759 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198762 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198765 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198768 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198771 2564 flags.go:64] FLAG: --image-service-endpoint="" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198774 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198777 2564 flags.go:64] FLAG: --kube-api-burst="100" Sep 14 12:58:18.205504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198781 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198784 2564 flags.go:64] FLAG: --kube-api-qps="50" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198787 2564 flags.go:64] FLAG: --kube-reserved="" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198790 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198793 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198797 2564 flags.go:64] FLAG: --kubelet-cgroups="" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198800 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198803 2564 flags.go:64] FLAG: --lock-file="" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198805 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198808 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198811 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198816 2564 flags.go:64] FLAG: --log-json-split-stream="false" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198819 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198821 2564 flags.go:64] FLAG: --log-text-split-stream="false" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198824 2564 flags.go:64] FLAG: --logging-format="text" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198827 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198830 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198833 2564 flags.go:64] FLAG: --manifest-url="" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198836 2564 flags.go:64] FLAG: --manifest-url-header="" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198840 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198843 2564 flags.go:64] FLAG: --max-open-files="1000000" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198846 2564 flags.go:64] FLAG: --max-pods="110" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198849 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198852 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198855 2564 flags.go:64] FLAG: --memory-manager-policy="None" Sep 14 12:58:18.206142 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198858 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198861 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198864 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198867 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198873 2564 flags.go:64] FLAG: --node-status-max-images="50" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198876 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198879 2564 flags.go:64] FLAG: --oom-score-adj="-999" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198882 2564 flags.go:64] FLAG: --pod-cidr="" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198885 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198891 2564 flags.go:64] FLAG: --pod-manifest-path="" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198894 2564 flags.go:64] FLAG: --pod-max-pids="-1" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198897 2564 flags.go:64] FLAG: --pods-per-core="0" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198901 2564 flags.go:64] FLAG: --port="10250" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198904 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198907 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07a0351472234e95e" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198909 2564 flags.go:64] FLAG: --qos-reserved="" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198912 2564 flags.go:64] FLAG: --read-only-port="10255" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198915 2564 flags.go:64] FLAG: --register-node="true" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198918 2564 flags.go:64] FLAG: --register-with-taints="" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198921 2564 flags.go:64] FLAG: --registry-burst="10" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198924 2564 flags.go:64] FLAG: --registry-qps="5" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198927 2564 flags.go:64] FLAG: --reserved-cpus="" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198929 2564 flags.go:64] FLAG: --reserved-memory="" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198933 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198936 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 14 12:58:18.206786 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198938 2564 flags.go:64] FLAG: --rotate-certificates="false" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198941 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198944 2564 flags.go:64] FLAG: --runonce="false" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198946 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198949 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198952 2564 flags.go:64] FLAG: --seccomp-default="false" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198955 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198958 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198960 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198963 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198966 2564 flags.go:64] FLAG: --storage-driver-password="root" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198969 2564 flags.go:64] FLAG: --storage-driver-secure="false" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198971 2564 flags.go:64] FLAG: --storage-driver-table="stats" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198976 2564 flags.go:64] FLAG: --storage-driver-user="root" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198978 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198981 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198988 2564 flags.go:64] FLAG: --system-cgroups="" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198991 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.198996 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.199001 2564 flags.go:64] FLAG: --tls-cert-file="" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.199004 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.199008 2564 flags.go:64] FLAG: --tls-min-version="" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.199011 2564 flags.go:64] FLAG: --tls-private-key-file="" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.199014 2564 flags.go:64] FLAG: --topology-manager-policy="none" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.199017 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 14 12:58:18.207422 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.199019 2564 flags.go:64] FLAG: --topology-manager-scope="container" Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.199022 2564 flags.go:64] FLAG: --v="2" Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.199027 2564 flags.go:64] FLAG: --version="false" Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.199031 2564 flags.go:64] FLAG: --vmodule="" Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.199034 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.199037 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199148 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199151 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199154 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199157 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199160 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199162 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199165 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199167 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199170 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199172 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199175 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199177 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199180 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199183 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199189 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 12:58:18.208082 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199192 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 12:58:18.208613 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199195 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 12:58:18.208613 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199198 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 12:58:18.208613 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199201 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 12:58:18.208613 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199203 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 12:58:18.208613 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199207 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 12:58:18.208613 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199209 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 12:58:18.208613 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199212 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 12:58:18.208613 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199214 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 12:58:18.208613 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199217 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 12:58:18.208613 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199219 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 12:58:18.208613 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199222 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 12:58:18.208613 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199224 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 12:58:18.208613 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199227 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 12:58:18.208613 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199230 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 12:58:18.208613 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199232 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 12:58:18.208613 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199235 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 12:58:18.208613 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199237 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 12:58:18.208613 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199240 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 12:58:18.208613 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199242 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 12:58:18.208613 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199245 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 12:58:18.209121 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199247 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 12:58:18.209121 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199250 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 12:58:18.209121 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199252 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 12:58:18.209121 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199254 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 12:58:18.209121 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199257 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 12:58:18.209121 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199259 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 12:58:18.209121 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199262 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 12:58:18.209121 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199264 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 12:58:18.209121 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199267 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 12:58:18.209121 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199269 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 12:58:18.209121 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199273 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 12:58:18.209121 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199275 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 12:58:18.209121 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199278 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 12:58:18.209121 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199282 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 12:58:18.209121 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199285 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 12:58:18.209121 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199289 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 12:58:18.209121 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199293 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 12:58:18.209121 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199296 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 12:58:18.209121 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199299 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 12:58:18.209599 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199301 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 12:58:18.209599 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199304 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 12:58:18.209599 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199306 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 12:58:18.209599 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199309 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 12:58:18.209599 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199311 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 12:58:18.209599 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199313 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 12:58:18.209599 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199316 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 12:58:18.209599 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199318 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 12:58:18.209599 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199321 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 12:58:18.209599 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199323 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 12:58:18.209599 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199326 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 12:58:18.209599 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199328 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 12:58:18.209599 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199331 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 12:58:18.209599 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199351 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 12:58:18.209599 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199354 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 12:58:18.209599 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199357 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 12:58:18.209599 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199360 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 12:58:18.209599 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199362 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 12:58:18.209599 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199365 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 12:58:18.209599 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199367 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 12:58:18.210102 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199371 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 12:58:18.210102 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199374 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 12:58:18.210102 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199377 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 12:58:18.210102 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199381 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 12:58:18.210102 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199384 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 12:58:18.210102 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199386 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 12:58:18.210102 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199389 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 12:58:18.210102 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199392 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 12:58:18.210102 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199394 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 12:58:18.210102 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199398 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 12:58:18.210102 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199401 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 12:58:18.210102 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199404 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 12:58:18.210102 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199406 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 12:58:18.210102 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199409 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 12:58:18.210102 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199411 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 12:58:18.210102 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199414 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 12:58:18.210102 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199416 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 12:58:18.210102 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199419 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 12:58:18.210102 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199421 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 12:58:18.210635 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199424 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 14 12:58:18.210635 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199426 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 12:58:18.210635 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199429 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 12:58:18.210635 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199431 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 12:58:18.210635 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.199434 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 12:58:18.210635 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.200029 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 12:58:18.210635 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.207173 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 14 12:58:18.210635 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.207186 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 14 12:58:18.210635 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.207990 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 14 12:58:18.210635 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.207997 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 14 12:58:18.210635 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208053 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 12:58:18.210635 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208058 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 12:58:18.210635 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208061 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 12:58:18.210635 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208064 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 12:58:18.210635 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208068 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 12:58:18.211010 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208071 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 12:58:18.211010 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208074 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 12:58:18.211010 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208076 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 12:58:18.211010 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208079 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 12:58:18.211010 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208082 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 12:58:18.211010 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208085 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 12:58:18.211010 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208087 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 12:58:18.211010 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208090 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 12:58:18.211010 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208093 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 12:58:18.211010 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208096 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 12:58:18.211010 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208098 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 12:58:18.211010 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208101 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 12:58:18.211010 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208103 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 12:58:18.211010 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208106 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 12:58:18.211010 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208110 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 12:58:18.211010 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208113 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 12:58:18.211010 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208116 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 12:58:18.211010 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208119 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 12:58:18.211010 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208121 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 12:58:18.211528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208124 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 12:58:18.211528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208127 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 14 12:58:18.211528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208130 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 12:58:18.211528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208132 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 12:58:18.211528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208135 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 12:58:18.211528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208139 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 12:58:18.211528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208142 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 12:58:18.211528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208145 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 12:58:18.211528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208147 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 12:58:18.211528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208150 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 12:58:18.211528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208152 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 12:58:18.211528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208155 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 12:58:18.211528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208157 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 12:58:18.211528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208161 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 12:58:18.211528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208163 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 12:58:18.211528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208167 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 12:58:18.211528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208171 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 12:58:18.211528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208174 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 12:58:18.211528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208176 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 12:58:18.211528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208179 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 12:58:18.212021 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208181 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 12:58:18.212021 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208184 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 12:58:18.212021 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208187 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 12:58:18.212021 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208189 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 12:58:18.212021 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208192 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 12:58:18.212021 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208194 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 12:58:18.212021 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208197 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 12:58:18.212021 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208199 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 12:58:18.212021 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208202 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 12:58:18.212021 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208204 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 12:58:18.212021 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208207 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 12:58:18.212021 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208209 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 12:58:18.212021 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208212 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 12:58:18.212021 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208215 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 12:58:18.212021 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208218 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 12:58:18.212021 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208220 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 12:58:18.212021 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208223 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 12:58:18.212021 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208225 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 12:58:18.212021 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208228 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 12:58:18.212021 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208231 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 12:58:18.212566 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208234 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 12:58:18.212566 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208236 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 12:58:18.212566 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208240 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 12:58:18.212566 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208244 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 12:58:18.212566 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208247 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 12:58:18.212566 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208250 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 12:58:18.212566 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208253 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 12:58:18.212566 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208256 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 12:58:18.212566 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208258 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 12:58:18.212566 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208262 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 12:58:18.212566 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208264 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 12:58:18.212566 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208266 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 12:58:18.212566 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208269 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 12:58:18.212566 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208272 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 12:58:18.212566 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208274 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 12:58:18.212566 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208276 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 12:58:18.212566 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208279 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 12:58:18.212566 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208281 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 12:58:18.212566 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208284 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 12:58:18.212566 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208286 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 12:58:18.213056 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208289 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 12:58:18.213056 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208291 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 12:58:18.213056 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208293 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 12:58:18.213056 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208296 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 12:58:18.213056 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208298 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 12:58:18.213056 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208301 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 12:58:18.213056 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208303 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 12:58:18.213056 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208306 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 12:58:18.213056 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208309 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 12:58:18.213056 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208311 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 12:58:18.213056 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208313 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 12:58:18.213056 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208316 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 12:58:18.213056 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208319 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 12:58:18.213056 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208321 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 12:58:18.213056 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208323 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 12:58:18.213443 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.208328 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 12:58:18.213443 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208466 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 14 12:58:18.213443 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208471 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 14 12:58:18.213443 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208474 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 14 12:58:18.213443 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208476 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 14 12:58:18.213443 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208479 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 14 12:58:18.213443 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208482 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 14 12:58:18.213443 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208484 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 14 12:58:18.213443 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208487 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 14 12:58:18.213443 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208489 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 14 12:58:18.213443 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208491 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 14 12:58:18.213443 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208494 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 14 12:58:18.213443 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208496 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 14 12:58:18.213443 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208499 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 14 12:58:18.213443 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208501 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 14 12:58:18.213820 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208504 2564 feature_gate.go:328] unrecognized feature gate: Example2 Sep 14 12:58:18.213820 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208506 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 14 12:58:18.213820 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208509 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 14 12:58:18.213820 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208511 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 14 12:58:18.213820 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208514 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 14 12:58:18.213820 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208516 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 14 12:58:18.213820 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208519 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 14 12:58:18.213820 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208521 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 14 12:58:18.213820 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208525 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 14 12:58:18.213820 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208529 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 14 12:58:18.213820 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208532 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 14 12:58:18.213820 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208534 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 14 12:58:18.213820 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208537 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 14 12:58:18.213820 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208539 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 14 12:58:18.213820 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208542 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 14 12:58:18.213820 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208544 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 14 12:58:18.213820 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208547 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 14 12:58:18.213820 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208551 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 14 12:58:18.213820 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208554 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 14 12:58:18.214283 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208557 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 14 12:58:18.214283 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208559 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 14 12:58:18.214283 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208562 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 14 12:58:18.214283 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208565 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 14 12:58:18.214283 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208567 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 14 12:58:18.214283 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208569 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 14 12:58:18.214283 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208572 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 14 12:58:18.214283 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208574 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 14 12:58:18.214283 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208577 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 14 12:58:18.214283 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208579 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 14 12:58:18.214283 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208581 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 14 12:58:18.214283 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208584 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 14 12:58:18.214283 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208586 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 14 12:58:18.214283 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208589 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 14 12:58:18.214283 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208591 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 14 12:58:18.214283 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208593 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 14 12:58:18.214283 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208596 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 14 12:58:18.214283 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208598 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 14 12:58:18.214283 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208601 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 14 12:58:18.214283 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208603 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 14 12:58:18.214855 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208605 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 14 12:58:18.214855 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208608 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 14 12:58:18.214855 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208611 2564 feature_gate.go:328] unrecognized feature gate: Example Sep 14 12:58:18.214855 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208613 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 14 12:58:18.214855 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208616 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 14 12:58:18.214855 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208618 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 14 12:58:18.214855 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208621 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 14 12:58:18.214855 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208623 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 14 12:58:18.214855 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208626 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 14 12:58:18.214855 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208629 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 14 12:58:18.214855 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208631 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 14 12:58:18.214855 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208634 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 14 12:58:18.214855 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208636 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 14 12:58:18.214855 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208639 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 14 12:58:18.214855 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208641 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 14 12:58:18.214855 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208643 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 14 12:58:18.214855 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208646 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 14 12:58:18.214855 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208648 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 14 12:58:18.214855 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208650 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 14 12:58:18.214855 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208653 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 14 12:58:18.215332 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208655 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 14 12:58:18.215332 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208657 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 14 12:58:18.215332 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208660 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 14 12:58:18.215332 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208662 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 14 12:58:18.215332 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208665 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 14 12:58:18.215332 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208667 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 14 12:58:18.215332 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208670 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 14 12:58:18.215332 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208672 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 14 12:58:18.215332 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208674 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 14 12:58:18.215332 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208677 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 14 12:58:18.215332 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208679 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 14 12:58:18.215332 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208682 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 14 12:58:18.215332 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208684 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 14 12:58:18.215332 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208687 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 14 12:58:18.215332 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208689 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 14 12:58:18.215332 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208692 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 14 12:58:18.215332 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208694 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 14 12:58:18.215332 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208697 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 14 12:58:18.215332 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208700 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 14 12:58:18.215805 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208704 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 14 12:58:18.215805 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208706 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 14 12:58:18.215805 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208710 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 14 12:58:18.215805 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208713 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 14 12:58:18.215805 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208716 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 14 12:58:18.215805 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208719 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 14 12:58:18.215805 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:18.208721 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 14 12:58:18.215805 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.208726 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 14 12:58:18.215805 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.208819 2564 server.go:962] "Client rotation is on, will bootstrap in background" Sep 14 12:58:18.215805 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.210792 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 14 12:58:18.215805 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.212025 2564 server.go:1019] "Starting client certificate rotation" Sep 14 12:58:18.215805 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.212128 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 14 12:58:18.215805 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.212193 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 14 12:58:18.216131 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.214780 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 14 12:58:18.220062 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.220042 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 14 12:58:18.235208 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.235190 2564 log.go:25] "Validated CRI v1 runtime API" Sep 14 12:58:18.238214 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.238199 2564 log.go:25] "Validated CRI v1 image API" Sep 14 12:58:18.239128 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.239102 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 14 12:58:18.243559 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.243530 2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c9a151eb-f818-48e0-b96b-d53f225df7c0:/dev/nvme0n1p3 fa90e750-19ca-451b-ba74-6047cfaabcc6:/dev/nvme0n1p4] Sep 14 12:58:18.243632 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.243558 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 14 12:58:18.244094 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.244076 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 14 12:58:18.249222 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.248459 2564 manager.go:216] Machine: {Timestamp:2026-09-14 12:58:18.247131993 +0000 UTC m=+0.373615179 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3043717 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2942a997910be1960224e792124606 SystemUUID:ec2942a9-9791-0be1-9602-24e792124606 BootID:1caa4214-3225-416d-9071-b4e613bcc5b6 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d5:6d:fc:a5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d5:6d:fc:a5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:e0:c4:d0:5e:41 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 14 12:58:18.249222 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.249211 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 14 12:58:18.249414 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.249310 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 14 12:58:18.250879 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.250858 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 14 12:58:18.251008 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.250881 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-96.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 14 12:58:18.251631 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.251621 2564 topology_manager.go:138] "Creating topology manager with none policy" Sep 14 12:58:18.251668 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.251633 2564 container_manager_linux.go:309] "Creating device plugin manager" Sep 14 12:58:18.251668 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.251641 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 14 12:58:18.251668 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.251655 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 14 12:58:18.252736 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.252725 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 14 12:58:18.255903 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.255891 2564 state_mem.go:36] "Initialized new in-memory state store" Sep 14 12:58:18.256013 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.256004 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 14 12:58:18.256046 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.256033 2564 kubelet.go:486] "Attempting to sync node with API server" Sep 14 12:58:18.256046 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.256042 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 14 12:58:18.256096 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.256056 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 14 12:58:18.256096 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.256077 2564 kubelet.go:398] "Adding apiserver pod source" Sep 14 12:58:18.256096 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.256091 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 14 12:58:18.256426 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.256411 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 14 12:58:18.256480 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.256436 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 14 12:58:18.258589 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.258571 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 14 12:58:18.260844 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.260831 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 14 12:58:18.260936 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.260861 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 14 12:58:18.262373 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.262361 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 14 12:58:18.262425 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.262378 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 14 12:58:18.262425 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.262384 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 14 12:58:18.262425 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.262389 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 14 12:58:18.262425 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.262397 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 14 12:58:18.262425 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.262402 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 14 12:58:18.262425 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.262415 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 14 12:58:18.262425 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.262421 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 14 12:58:18.262425 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.262427 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 14 12:58:18.262633 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.262433 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 14 12:58:18.262633 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.262441 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 14 12:58:18.262633 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.262447 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 14 12:58:18.263015 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.263006 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 14 12:58:18.263015 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.263014 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 14 12:58:18.267259 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.267239 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-96.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 14 12:58:18.267382 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.267248 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 14 12:58:18.268652 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.268511 2564 server.go:1268] "Started kubelet" Sep 14 12:58:18.268726 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.268585 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 14 12:58:18.268800 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.268601 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 14 12:58:18.268850 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.268823 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 14 12:58:18.269399 ip-10-0-136-96 systemd[1]: Started Kubernetes Kubelet. Sep 14 12:58:18.269975 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.269953 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 14 12:58:18.270842 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.270827 2564 server.go:310] "Adding debug handlers to kubelet server" Sep 14 12:58:18.271726 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.271701 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-thtbp" Sep 14 12:58:18.272673 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.272656 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 14 12:58:18.272876 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.272858 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 14 12:58:18.274301 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.272910 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-96.ec2.internal.18d5315c06ae441c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-96.ec2.internal,UID:ip-10-0-136-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-96.ec2.internal,},FirstTimestamp:2026-09-14 12:58:18.268476444 +0000 UTC m=+0.394959627,LastTimestamp:2026-09-14 12:58:18.268476444 +0000 UTC m=+0.394959627,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-96.ec2.internal,}" Sep 14 12:58:18.275670 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.275653 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 14 12:58:18.276401 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.276384 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 14 12:58:18.276512 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.276498 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 14 12:58:18.276985 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.276967 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 14 12:58:18.276985 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.276969 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 14 12:58:18.277125 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.276995 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 14 12:58:18.277125 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.277080 2564 reconstruct.go:97] "Volume reconstruction finished" Sep 14 12:58:18.277125 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.277089 2564 reconciler.go:29] "Reconciler: start to sync state" Sep 14 12:58:18.277125 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.277112 2564 factory.go:55] Registering systemd factory Sep 14 12:58:18.277125 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.277123 2564 factory.go:223] Registration of the systemd container factory successfully Sep 14 12:58:18.277309 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.277298 2564 factory.go:153] Registering CRI-O factory Sep 14 12:58:18.277370 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.277312 2564 factory.go:223] Registration of the crio container factory successfully Sep 14 12:58:18.277427 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.277376 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 14 12:58:18.277427 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.277394 2564 factory.go:103] Registering Raw factory Sep 14 12:58:18.277427 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.277405 2564 manager.go:1196] Started watching for new ooms in manager Sep 14 12:58:18.277806 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.277793 2564 manager.go:349] Starting recovery of all containers Sep 14 12:58:18.279123 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.279105 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-thtbp" Sep 14 12:58:18.279502 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.279464 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-96.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 14 12:58:18.279582 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.279564 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 14 12:58:18.288468 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.288454 2564 manager.go:354] Recovery completed Sep 14 12:58:18.292886 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.292872 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 12:58:18.294986 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.294972 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory" Sep 14 12:58:18.295046 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.295002 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 12:58:18.295046 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.295011 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID" Sep 14 12:58:18.295660 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.295648 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 14 12:58:18.295660 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.295660 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 14 12:58:18.295764 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.295674 2564 state_mem.go:36] "Initialized new in-memory state store" Sep 14 12:58:18.295764 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.295650 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 12:58:18.296398 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.296380 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory" Sep 14 12:58:18.296459 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.296406 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 12:58:18.296459 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.296416 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID" Sep 14 12:58:18.296877 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.296863 2564 policy_none.go:49] "None policy: Start" Sep 14 12:58:18.296927 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.296883 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 14 12:58:18.296927 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.296897 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 14 12:58:18.297167 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.297102 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-96.ec2.internal.18d5315c0842c84a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-96.ec2.internal,UID:ip-10-0-136-96.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-96.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-96.ec2.internal,},FirstTimestamp:2026-09-14 12:58:18.294986826 +0000 UTC m=+0.421470009,LastTimestamp:2026-09-14 12:58:18.294986826 +0000 UTC m=+0.421470009,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-96.ec2.internal,}" Sep 14 12:58:18.298134 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.298113 2564 policy_none.go:47] "Start" Sep 14 12:58:18.344589 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.336906 2564 manager.go:337] "Starting Device Plugin manager" Sep 14 12:58:18.344589 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.336939 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 14 12:58:18.344589 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.336952 2564 server.go:87] "Starting device plugin registration server" Sep 14 12:58:18.344589 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.337208 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 14 12:58:18.344589 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.337218 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 14 12:58:18.344589 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.337313 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 14 12:58:18.344589 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.337406 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 14 12:58:18.344589 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.337415 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 14 12:58:18.344589 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.337935 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 14 12:58:18.344589 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.337974 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-96.ec2.internal\" not found" Sep 14 12:58:18.344589 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.339715 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 14 12:58:18.344589 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.341116 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 14 12:58:18.344589 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.341161 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 14 12:58:18.344589 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.341181 2564 kubelet.go:2448] "Starting kubelet main sync loop" Sep 14 12:58:18.344589 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.341213 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 14 12:58:18.344589 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.342934 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 12:58:18.437585 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.437553 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 12:58:18.438657 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.438580 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory" Sep 14 12:58:18.438657 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.438613 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 12:58:18.438657 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.438628 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID" Sep 14 12:58:18.438657 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.438658 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-96.ec2.internal" Sep 14 12:58:18.441742 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.441724 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal"] Sep 14 12:58:18.441798 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.441790 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 12:58:18.442640 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.442627 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory" Sep 14 12:58:18.442705 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.442652 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 12:58:18.442705 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.442662 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID" Sep 14 12:58:18.444011 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.444000 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 12:58:18.444143 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.444127 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" Sep 14 12:58:18.444203 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.444166 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 12:58:18.444818 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.444798 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory" Sep 14 12:58:18.444905 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.444830 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 12:58:18.444905 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.444840 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID" Sep 14 12:58:18.444905 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.444798 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory" Sep 14 12:58:18.445040 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.444922 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 12:58:18.445040 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.444940 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID" Sep 14 12:58:18.446312 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.446297 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" Sep 14 12:58:18.446417 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.446327 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 14 12:58:18.446803 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.446789 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-96.ec2.internal" Sep 14 12:58:18.446856 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.446811 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-96.ec2.internal\": node \"ip-10-0-136-96.ec2.internal\" not found" Sep 14 12:58:18.447150 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.447134 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientMemory" Sep 14 12:58:18.447243 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.447166 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasNoDiskPressure" Sep 14 12:58:18.447243 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.447182 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeHasSufficientPID" Sep 14 12:58:18.458073 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.458052 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 14 12:58:18.469961 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.469939 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-96.ec2.internal\" not found" node="ip-10-0-136-96.ec2.internal" Sep 14 12:58:18.474276 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.474260 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-96.ec2.internal\" not found" node="ip-10-0-136-96.ec2.internal" Sep 14 12:58:18.477444 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.477430 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e46b070927ba07e09c7fdfca6f7f4f91-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal\" (UID: \"e46b070927ba07e09c7fdfca6f7f4f91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" Sep 14 12:58:18.477498 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.477452 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e46b070927ba07e09c7fdfca6f7f4f91-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal\" (UID: \"e46b070927ba07e09c7fdfca6f7f4f91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" Sep 14 12:58:18.477498 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.477469 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd507eff9b3716da03d385309d794351-config\") pod \"kube-apiserver-proxy-ip-10-0-136-96.ec2.internal\" (UID: \"bd507eff9b3716da03d385309d794351\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" Sep 14 12:58:18.558356 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.558291 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 14 12:58:18.578512 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.578471 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e46b070927ba07e09c7fdfca6f7f4f91-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal\" (UID: \"e46b070927ba07e09c7fdfca6f7f4f91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" Sep 14 12:58:18.578634 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.578520 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e46b070927ba07e09c7fdfca6f7f4f91-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal\" (UID: \"e46b070927ba07e09c7fdfca6f7f4f91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" Sep 14 12:58:18.578634 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.578546 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd507eff9b3716da03d385309d794351-config\") pod \"kube-apiserver-proxy-ip-10-0-136-96.ec2.internal\" (UID: \"bd507eff9b3716da03d385309d794351\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" Sep 14 12:58:18.578634 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.578600 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd507eff9b3716da03d385309d794351-config\") pod \"kube-apiserver-proxy-ip-10-0-136-96.ec2.internal\" (UID: \"bd507eff9b3716da03d385309d794351\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" Sep 14 12:58:18.578634 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.578604 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e46b070927ba07e09c7fdfca6f7f4f91-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal\" (UID: \"e46b070927ba07e09c7fdfca6f7f4f91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" Sep 14 12:58:18.578634 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.578602 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e46b070927ba07e09c7fdfca6f7f4f91-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal\" (UID: \"e46b070927ba07e09c7fdfca6f7f4f91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" Sep 14 12:58:18.659048 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.659025 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 14 12:58:18.760131 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.760046 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 14 12:58:18.773222 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.773197 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" Sep 14 12:58:18.777317 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:18.777170 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" Sep 14 12:58:18.860185 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.860137 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 14 12:58:18.960609 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:18.960579 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 14 12:58:19.061213 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:19.061131 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 14 12:58:19.113686 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:19.113662 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 12:58:19.161287 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:19.161255 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 14 12:58:19.211743 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:19.211718 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 14 12:58:19.212162 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:19.211868 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 14 12:58:19.212162 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:19.211871 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 14 12:58:19.262125 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:19.262098 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 14 12:58:19.284959 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:19.284909 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-13 12:53:19 +0000 UTC" deadline="2028-05-09 14:52:23.828082751 +0000 UTC" Sep 14 12:58:19.284959 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:19.284934 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14473h54m4.543151271s" Sep 14 12:58:19.347256 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:19.347226 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46b070927ba07e09c7fdfca6f7f4f91.slice/crio-9551cd5c206fab4b08287f5b9b043128065c277bb804e0cc21f7405628e65154 WatchSource:0}: Error finding container 9551cd5c206fab4b08287f5b9b043128065c277bb804e0cc21f7405628e65154: Status 404 returned error can't find the container with id 9551cd5c206fab4b08287f5b9b043128065c277bb804e0cc21f7405628e65154 Sep 14 12:58:19.352375 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:19.352357 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 12:58:19.363063 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:19.363044 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 14 12:58:19.435528 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:19.435498 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd507eff9b3716da03d385309d794351.slice/crio-4af58b8bed22f09b59c60e9eda9a1f1b649ca335ba4fd3a571e22843cef0f92e WatchSource:0}: Error finding container 4af58b8bed22f09b59c60e9eda9a1f1b649ca335ba4fd3a571e22843cef0f92e: Status 404 returned error can't find the container with id 4af58b8bed22f09b59c60e9eda9a1f1b649ca335ba4fd3a571e22843cef0f92e Sep 14 12:58:19.463604 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:19.463580 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 14 12:58:19.563694 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:19.563660 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-96.ec2.internal\" not found" Sep 14 12:58:19.563838 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:19.563770 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 12:58:19.566681 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:19.566663 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 12:58:19.576847 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:19.576824 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" Sep 14 12:58:19.589708 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:19.589660 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 14 12:58:19.590801 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:19.590787 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" Sep 14 12:58:19.600504 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:19.600484 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 14 12:58:21.720966 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.720935 2564 apiserver.go:52] "Watching apiserver" Sep 14 12:58:21.731680 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.731651 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 14 12:58:21.732373 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.732341 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-slcjt","kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l","openshift-cluster-node-tuning-operator/tuned-v64ww","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal","openshift-multus/network-metrics-daemon-kv576","openshift-network-operator/iptables-alerter-rxz94","kube-system/global-pull-secret-syncer-72fhn","openshift-dns/node-resolver-g2k4m","openshift-image-registry/node-ca-4v75b","openshift-multus/multus-additional-cni-plugins-6h4bn","openshift-multus/multus-fs7nx","openshift-network-diagnostics/network-check-target-7t274","openshift-ovn-kubernetes/ovnkube-node-b62sc"] Sep 14 12:58:21.736584 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.736562 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rxz94" Sep 14 12:58:21.738893 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.738868 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.740115 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.739461 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 14 12:58:21.740115 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.739530 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 14 12:58:21.740115 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.739532 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 14 12:58:21.740115 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.739897 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 14 12:58:21.740115 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.739914 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qd26v\"" Sep 14 12:58:21.741138 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.741116 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-slcjt" Sep 14 12:58:21.741569 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.741552 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 14 12:58:21.741658 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.741590 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 14 12:58:21.741658 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.741609 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 14 12:58:21.741856 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.741843 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 14 12:58:21.741899 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.741852 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 14 12:58:21.742131 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.742114 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8h7cx\"" Sep 14 12:58:21.743609 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.743501 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.743865 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.743851 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 14 12:58:21.744003 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.743990 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 14 12:58:21.744570 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.744398 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-64cc5\"" Sep 14 12:58:21.745801 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.745785 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4v75b" Sep 14 12:58:21.746326 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.746312 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-t6ffv\"" Sep 14 12:58:21.746604 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.746586 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 14 12:58:21.746899 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.746885 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 14 12:58:21.748090 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.748073 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:21.748171 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:21.748147 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kv576" podUID="ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63" Sep 14 12:58:21.748523 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.748503 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 14 12:58:21.748599 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.748530 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 14 12:58:21.748775 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.748757 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 14 12:58:21.748775 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.748765 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h8mlx\"" Sep 14 12:58:21.750423 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.750405 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:21.750527 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:21.750482 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7t274" podUID="746cbe95-b362-49b9-a280-e4c1faf73d32" Sep 14 12:58:21.751870 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.751752 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-etc-sysctl-d\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.751870 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.751756 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 14 12:58:21.751870 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.751784 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00612ad7-80a8-41a4-90fd-28e632d42e8d-agent-certs\") pod \"konnectivity-agent-slcjt\" (UID: \"00612ad7-80a8-41a4-90fd-28e632d42e8d\") " pod="kube-system/konnectivity-agent-slcjt" Sep 14 12:58:21.751870 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.751815 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-sys\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.751870 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.751830 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9567688-99e2-46d9-b2ce-007605405aea-host-slash\") pod \"iptables-alerter-rxz94\" (UID: \"d9567688-99e2-46d9-b2ce-007605405aea\") " pod="openshift-network-operator/iptables-alerter-rxz94" Sep 14 12:58:21.751870 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.751846 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f44b8a44-25cf-4fec-a6ad-79aef4b55753-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.751870 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.751861 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26b2n\" (UniqueName: \"kubernetes.io/projected/f44b8a44-25cf-4fec-a6ad-79aef4b55753-kube-api-access-26b2n\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.751870 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.751877 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-etc-sysconfig\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.752318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.751890 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-etc-sysctl-conf\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.752318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.751903 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-etc-systemd\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.752318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.751916 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f44b8a44-25cf-4fec-a6ad-79aef4b55753-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.752318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.751993 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f44b8a44-25cf-4fec-a6ad-79aef4b55753-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.752318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.752011 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba142181-460d-4266-a361-bb44c7fbb63e-etc-tuned\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.752318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.752032 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92tn8\" (UniqueName: \"kubernetes.io/projected/ba142181-460d-4266-a361-bb44c7fbb63e-kube-api-access-92tn8\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.752318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.752059 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f44b8a44-25cf-4fec-a6ad-79aef4b55753-system-cni-dir\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.752318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.752073 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-etc-modprobe-d\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.752318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.752086 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-run\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.752318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.752099 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-var-lib-kubelet\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.752318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.752114 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f44b8a44-25cf-4fec-a6ad-79aef4b55753-cni-binary-copy\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.752318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.752128 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00612ad7-80a8-41a4-90fd-28e632d42e8d-konnectivity-ca\") pod \"konnectivity-agent-slcjt\" (UID: \"00612ad7-80a8-41a4-90fd-28e632d42e8d\") " pod="kube-system/konnectivity-agent-slcjt" Sep 14 12:58:21.752318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.752147 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-lib-modules\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.752318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.752159 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-host\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.752318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.752172 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9567688-99e2-46d9-b2ce-007605405aea-iptables-alerter-script\") pod \"iptables-alerter-rxz94\" (UID: \"d9567688-99e2-46d9-b2ce-007605405aea\") " pod="openshift-network-operator/iptables-alerter-rxz94" Sep 14 12:58:21.752318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.752186 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f44b8a44-25cf-4fec-a6ad-79aef4b55753-cnibin\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.752886 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.752204 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-etc-kubernetes\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.752886 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.752225 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba142181-460d-4266-a361-bb44c7fbb63e-tmp\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.752886 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.752245 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9xkv\" (UniqueName: \"kubernetes.io/projected/d9567688-99e2-46d9-b2ce-007605405aea-kube-api-access-n9xkv\") pod \"iptables-alerter-rxz94\" (UID: \"d9567688-99e2-46d9-b2ce-007605405aea\") " pod="openshift-network-operator/iptables-alerter-rxz94" Sep 14 12:58:21.753136 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.753121 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g2k4m" Sep 14 12:58:21.754798 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.754780 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.755899 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.755884 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 14 12:58:21.756024 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.755892 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qrtm6\"" Sep 14 12:58:21.756201 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.756173 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 14 12:58:21.757214 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.757181 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.758163 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.757367 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 14 12:58:21.758163 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.757411 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 14 12:58:21.758163 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.757507 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 14 12:58:21.758163 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.757514 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-znsxx\"" Sep 14 12:58:21.759424 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.759409 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:21.759512 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:21.759470 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72fhn" podUID="d49b2943-be96-4d9a-ab55-965845ba03f9" Sep 14 12:58:21.759570 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.759545 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 14 12:58:21.759667 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.759630 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7ls5v\"" Sep 14 12:58:21.761789 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.761770 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.764112 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.764083 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 14 12:58:21.764476 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.764459 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 14 12:58:21.764544 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.764534 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-57lj5\"" Sep 14 12:58:21.766054 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.765804 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 14 12:58:21.766054 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.765841 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 14 12:58:21.766054 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.765862 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 14 12:58:21.766054 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.765895 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 14 12:58:21.773795 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.773779 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-csfhj" Sep 14 12:58:21.786466 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.786449 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-csfhj" Sep 14 12:58:21.809971 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.809888 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" event={"ID":"bd507eff9b3716da03d385309d794351","Type":"ContainerStarted","Data":"4af58b8bed22f09b59c60e9eda9a1f1b649ca335ba4fd3a571e22843cef0f92e"} Sep 14 12:58:21.810909 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.810889 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" event={"ID":"e46b070927ba07e09c7fdfca6f7f4f91","Type":"ContainerStarted","Data":"9551cd5c206fab4b08287f5b9b043128065c277bb804e0cc21f7405628e65154"} Sep 14 12:58:21.842278 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.842248 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 14 12:58:21.852850 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.852826 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9567688-99e2-46d9-b2ce-007605405aea-iptables-alerter-script\") pod \"iptables-alerter-rxz94\" (UID: \"d9567688-99e2-46d9-b2ce-007605405aea\") " pod="openshift-network-operator/iptables-alerter-rxz94" Sep 14 12:58:21.852970 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.852860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f44b8a44-25cf-4fec-a6ad-79aef4b55753-cnibin\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.852970 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.852880 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.852970 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.852896 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-systemd-units\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.852970 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.852910 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-run-netns\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.852970 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.852948 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-run-ovn\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.853205 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.852963 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f44b8a44-25cf-4fec-a6ad-79aef4b55753-cnibin\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.853205 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.852987 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-etc-kubernetes\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.853205 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba142181-460d-4266-a361-bb44c7fbb63e-tmp\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.853205 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853032 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/570dacc2-2bf1-4b1a-8f03-3abb696a2ecf-host\") pod \"node-ca-4v75b\" (UID: \"570dacc2-2bf1-4b1a-8f03-3abb696a2ecf\") " pod="openshift-image-registry/node-ca-4v75b" Sep 14 12:58:21.853205 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-system-cni-dir\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.853205 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853085 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-run-systemd\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.853205 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853084 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-etc-kubernetes\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.853205 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853107 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-node-log\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.853205 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853163 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44km8\" (UniqueName: \"kubernetes.io/projected/578bd075-5e22-4a15-bde8-57bd2c202ba0-kube-api-access-44km8\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.853205 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853187 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs\") pod \"network-metrics-daemon-kv576\" (UID: \"ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63\") " pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:21.853685 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-etc-sysctl-d\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.853685 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853232 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00612ad7-80a8-41a4-90fd-28e632d42e8d-agent-certs\") pod \"konnectivity-agent-slcjt\" (UID: \"00612ad7-80a8-41a4-90fd-28e632d42e8d\") " pod="kube-system/konnectivity-agent-slcjt" Sep 14 12:58:21.853685 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853246 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d6daf95-87b4-4f08-8058-6ad1d9ea9dec-tmp-dir\") pod \"node-resolver-g2k4m\" (UID: \"0d6daf95-87b4-4f08-8058-6ad1d9ea9dec\") " pod="openshift-dns/node-resolver-g2k4m" Sep 14 12:58:21.853685 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853260 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-host-run-k8s-cni-cncf-io\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.853685 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853275 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-host-run-netns\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.853685 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853296 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-slash\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.853685 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrmjk\" (UniqueName: \"kubernetes.io/projected/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-kube-api-access-qrmjk\") pod \"network-metrics-daemon-kv576\" (UID: \"ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63\") " pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:21.853685 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853341 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9567688-99e2-46d9-b2ce-007605405aea-host-slash\") pod \"iptables-alerter-rxz94\" (UID: \"d9567688-99e2-46d9-b2ce-007605405aea\") " pod="openshift-network-operator/iptables-alerter-rxz94" Sep 14 12:58:21.853685 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853358 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f44b8a44-25cf-4fec-a6ad-79aef4b55753-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.853685 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853354 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-etc-sysctl-d\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.853685 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853377 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/570dacc2-2bf1-4b1a-8f03-3abb696a2ecf-serviceca\") pod \"node-ca-4v75b\" (UID: \"570dacc2-2bf1-4b1a-8f03-3abb696a2ecf\") " pod="openshift-image-registry/node-ca-4v75b" Sep 14 12:58:21.853685 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853386 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9567688-99e2-46d9-b2ce-007605405aea-host-slash\") pod \"iptables-alerter-rxz94\" (UID: \"d9567688-99e2-46d9-b2ce-007605405aea\") " pod="openshift-network-operator/iptables-alerter-rxz94" Sep 14 12:58:21.853685 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853400 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-sys-fs\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.853685 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853423 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-host-run-multus-certs\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.853685 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853448 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26b2n\" (UniqueName: \"kubernetes.io/projected/f44b8a44-25cf-4fec-a6ad-79aef4b55753-kube-api-access-26b2n\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.853685 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853471 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-etc-systemd\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.853685 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853498 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f44b8a44-25cf-4fec-a6ad-79aef4b55753-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.854474 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853522 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-etc-systemd\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.854474 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853522 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f44b8a44-25cf-4fec-a6ad-79aef4b55753-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.854474 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853564 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fllzn\" (UniqueName: \"kubernetes.io/projected/570dacc2-2bf1-4b1a-8f03-3abb696a2ecf-kube-api-access-fllzn\") pod \"node-ca-4v75b\" (UID: \"570dacc2-2bf1-4b1a-8f03-3abb696a2ecf\") " pod="openshift-image-registry/node-ca-4v75b" Sep 14 12:58:21.854474 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853592 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-etc-openvswitch\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.854474 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853619 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.854474 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853647 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92tn8\" (UniqueName: \"kubernetes.io/projected/ba142181-460d-4266-a361-bb44c7fbb63e-kube-api-access-92tn8\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.854474 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853670 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/921c1e9a-91e6-4f6d-846f-e146d92249ca-multus-daemon-config\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.854474 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853688 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f44b8a44-25cf-4fec-a6ad-79aef4b55753-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.854474 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853695 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret\") pod \"global-pull-secret-syncer-72fhn\" (UID: \"d49b2943-be96-4d9a-ab55-965845ba03f9\") " pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:21.854474 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853732 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-log-socket\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.854474 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853745 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 14 12:58:21.854474 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853757 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/578bd075-5e22-4a15-bde8-57bd2c202ba0-ovnkube-config\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.854474 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853781 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/578bd075-5e22-4a15-bde8-57bd2c202ba0-env-overrides\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.854474 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853780 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9567688-99e2-46d9-b2ce-007605405aea-iptables-alerter-script\") pod \"iptables-alerter-rxz94\" (UID: \"d9567688-99e2-46d9-b2ce-007605405aea\") " pod="openshift-network-operator/iptables-alerter-rxz94" Sep 14 12:58:21.854474 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853805 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/578bd075-5e22-4a15-bde8-57bd2c202ba0-ovn-node-metrics-cert\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.854474 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853831 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-etc-modprobe-d\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.854474 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853857 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-run\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.855232 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853881 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f44b8a44-25cf-4fec-a6ad-79aef4b55753-cni-binary-copy\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.855232 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-registration-dir\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.855232 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853951 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49hmq\" (UniqueName: \"kubernetes.io/projected/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-kube-api-access-49hmq\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.855232 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853951 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f44b8a44-25cf-4fec-a6ad-79aef4b55753-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.855232 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853976 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-multus-conf-dir\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.855232 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853975 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-run\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.855232 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853997 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f44b8a44-25cf-4fec-a6ad-79aef4b55753-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.855232 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.853998 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-kubelet\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.855232 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854039 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-run-openvswitch\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.855232 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854071 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-etc-modprobe-d\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.855232 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854071 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-var-lib-openvswitch\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.855232 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854108 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-run-ovn-kubernetes\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.855232 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854137 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-cni-netd\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.855232 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854159 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v2x2\" (UniqueName: \"kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2\") pod \"network-check-target-7t274\" (UID: \"746cbe95-b362-49b9-a280-e4c1faf73d32\") " pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:21.855232 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854204 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-host\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.855232 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854227 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-multus-cni-dir\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.855725 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854252 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9xkv\" (UniqueName: \"kubernetes.io/projected/d9567688-99e2-46d9-b2ce-007605405aea-kube-api-access-n9xkv\") pod \"iptables-alerter-rxz94\" (UID: \"d9567688-99e2-46d9-b2ce-007605405aea\") " pod="openshift-network-operator/iptables-alerter-rxz94" Sep 14 12:58:21.855725 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854268 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-host\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.855725 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854294 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-multus-socket-dir-parent\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.855725 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854313 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d49b2943-be96-4d9a-ab55-965845ba03f9-dbus\") pod \"global-pull-secret-syncer-72fhn\" (UID: \"d49b2943-be96-4d9a-ab55-965845ba03f9\") " pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:21.855725 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854345 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f44b8a44-25cf-4fec-a6ad-79aef4b55753-cni-binary-copy\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.855725 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854350 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.855725 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854394 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-cnibin\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.855725 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854422 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-host-var-lib-cni-multus\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.855725 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854445 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-cni-bin\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.855725 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854468 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-sys\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.855725 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854503 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0d6daf95-87b4-4f08-8058-6ad1d9ea9dec-hosts-file\") pod \"node-resolver-g2k4m\" (UID: \"0d6daf95-87b4-4f08-8058-6ad1d9ea9dec\") " pod="openshift-dns/node-resolver-g2k4m" Sep 14 12:58:21.855725 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854534 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/921c1e9a-91e6-4f6d-846f-e146d92249ca-cni-binary-copy\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.855725 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854557 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-host-var-lib-cni-bin\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.855725 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854556 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-sys\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.855725 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854581 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-host-var-lib-kubelet\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.855725 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854606 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-etc-sysconfig\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.855725 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-etc-sysctl-conf\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.856298 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854658 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-device-dir\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.856298 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854667 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-etc-sysconfig\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.856298 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854683 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/578bd075-5e22-4a15-bde8-57bd2c202ba0-ovnkube-script-lib\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.856298 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854710 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba142181-460d-4266-a361-bb44c7fbb63e-etc-tuned\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.856298 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854733 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f44b8a44-25cf-4fec-a6ad-79aef4b55753-system-cni-dir\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.856298 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854877 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-socket-dir\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.856298 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854903 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-var-lib-kubelet\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.856298 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854945 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00612ad7-80a8-41a4-90fd-28e632d42e8d-konnectivity-ca\") pod \"konnectivity-agent-slcjt\" (UID: \"00612ad7-80a8-41a4-90fd-28e632d42e8d\") " pod="kube-system/konnectivity-agent-slcjt" Sep 14 12:58:21.856298 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854734 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-etc-sysctl-conf\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.856298 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854768 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f44b8a44-25cf-4fec-a6ad-79aef4b55753-system-cni-dir\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.856298 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.854970 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fg9c\" (UniqueName: \"kubernetes.io/projected/0d6daf95-87b4-4f08-8058-6ad1d9ea9dec-kube-api-access-6fg9c\") pod \"node-resolver-g2k4m\" (UID: \"0d6daf95-87b4-4f08-8058-6ad1d9ea9dec\") " pod="openshift-dns/node-resolver-g2k4m" Sep 14 12:58:21.856298 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.855000 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-hostroot\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.856298 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.855023 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88m84\" (UniqueName: \"kubernetes.io/projected/921c1e9a-91e6-4f6d-846f-e146d92249ca-kube-api-access-88m84\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.856298 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.855046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-lib-modules\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.856298 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.855068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-etc-kubernetes\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.856298 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.855091 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d49b2943-be96-4d9a-ab55-965845ba03f9-kubelet-config\") pod \"global-pull-secret-syncer-72fhn\" (UID: \"d49b2943-be96-4d9a-ab55-965845ba03f9\") " pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:21.856298 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.855139 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-lib-modules\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.856878 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.855005 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba142181-460d-4266-a361-bb44c7fbb63e-var-lib-kubelet\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.856878 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.855385 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00612ad7-80a8-41a4-90fd-28e632d42e8d-konnectivity-ca\") pod \"konnectivity-agent-slcjt\" (UID: \"00612ad7-80a8-41a4-90fd-28e632d42e8d\") " pod="kube-system/konnectivity-agent-slcjt" Sep 14 12:58:21.857822 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.857410 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba142181-460d-4266-a361-bb44c7fbb63e-tmp\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.857822 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.857452 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba142181-460d-4266-a361-bb44c7fbb63e-etc-tuned\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.857822 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.857770 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00612ad7-80a8-41a4-90fd-28e632d42e8d-agent-certs\") pod \"konnectivity-agent-slcjt\" (UID: \"00612ad7-80a8-41a4-90fd-28e632d42e8d\") " pod="kube-system/konnectivity-agent-slcjt" Sep 14 12:58:21.862179 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.862161 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26b2n\" (UniqueName: \"kubernetes.io/projected/f44b8a44-25cf-4fec-a6ad-79aef4b55753-kube-api-access-26b2n\") pod \"multus-additional-cni-plugins-6h4bn\" (UID: \"f44b8a44-25cf-4fec-a6ad-79aef4b55753\") " pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:21.862487 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.862465 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92tn8\" (UniqueName: \"kubernetes.io/projected/ba142181-460d-4266-a361-bb44c7fbb63e-kube-api-access-92tn8\") pod \"tuned-v64ww\" (UID: \"ba142181-460d-4266-a361-bb44c7fbb63e\") " pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:21.862568 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.862492 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9xkv\" (UniqueName: \"kubernetes.io/projected/d9567688-99e2-46d9-b2ce-007605405aea-kube-api-access-n9xkv\") pod \"iptables-alerter-rxz94\" (UID: \"d9567688-99e2-46d9-b2ce-007605405aea\") " pod="openshift-network-operator/iptables-alerter-rxz94" Sep 14 12:58:21.955397 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955363 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-host-run-multus-certs\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.955397 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955403 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fllzn\" (UniqueName: \"kubernetes.io/projected/570dacc2-2bf1-4b1a-8f03-3abb696a2ecf-kube-api-access-fllzn\") pod \"node-ca-4v75b\" (UID: \"570dacc2-2bf1-4b1a-8f03-3abb696a2ecf\") " pod="openshift-image-registry/node-ca-4v75b" Sep 14 12:58:21.955635 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955428 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-etc-openvswitch\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.955635 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955450 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.955635 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955476 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/921c1e9a-91e6-4f6d-846f-e146d92249ca-multus-daemon-config\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.955635 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955478 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-host-run-multus-certs\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.955635 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955493 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret\") pod \"global-pull-secret-syncer-72fhn\" (UID: \"d49b2943-be96-4d9a-ab55-965845ba03f9\") " pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:21.955635 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955530 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-log-socket\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.955635 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/578bd075-5e22-4a15-bde8-57bd2c202ba0-ovnkube-config\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.955635 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955579 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/578bd075-5e22-4a15-bde8-57bd2c202ba0-env-overrides\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.955635 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:21.955586 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 12:58:21.955635 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955602 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/578bd075-5e22-4a15-bde8-57bd2c202ba0-ovn-node-metrics-cert\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.955635 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955630 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-registration-dir\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.956211 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:21.955661 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret podName:d49b2943-be96-4d9a-ab55-965845ba03f9 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:22.455628528 +0000 UTC m=+3.118524160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret") pod "global-pull-secret-syncer-72fhn" (UID: "d49b2943-be96-4d9a-ab55-965845ba03f9") : object "kube-system"/"original-pull-secret" not registered Sep 14 12:58:21.956211 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955688 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49hmq\" (UniqueName: \"kubernetes.io/projected/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-kube-api-access-49hmq\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.956211 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955711 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-multus-conf-dir\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.956211 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955728 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-kubelet\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.956211 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955743 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-run-openvswitch\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.956211 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955757 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-var-lib-openvswitch\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.956211 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-run-ovn-kubernetes\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.956211 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955791 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-cni-netd\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.956211 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955814 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v2x2\" (UniqueName: \"kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2\") pod \"network-check-target-7t274\" (UID: \"746cbe95-b362-49b9-a280-e4c1faf73d32\") " pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:21.956211 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955838 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-multus-cni-dir\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.956211 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955863 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-multus-socket-dir-parent\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.956211 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955884 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d49b2943-be96-4d9a-ab55-965845ba03f9-dbus\") pod \"global-pull-secret-syncer-72fhn\" (UID: \"d49b2943-be96-4d9a-ab55-965845ba03f9\") " pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:21.956211 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955937 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.956211 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955956 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-cnibin\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.956211 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955974 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-host-var-lib-cni-multus\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.956211 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955991 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-cni-bin\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.956806 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956016 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0d6daf95-87b4-4f08-8058-6ad1d9ea9dec-hosts-file\") pod \"node-resolver-g2k4m\" (UID: \"0d6daf95-87b4-4f08-8058-6ad1d9ea9dec\") " pod="openshift-dns/node-resolver-g2k4m" Sep 14 12:58:21.956806 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956061 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/921c1e9a-91e6-4f6d-846f-e146d92249ca-cni-binary-copy\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.956806 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956086 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-host-var-lib-cni-bin\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.956806 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956108 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-host-var-lib-kubelet\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.956806 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956129 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-device-dir\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.956806 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956145 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/578bd075-5e22-4a15-bde8-57bd2c202ba0-ovnkube-script-lib\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.956806 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956162 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-socket-dir\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.956806 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956171 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/578bd075-5e22-4a15-bde8-57bd2c202ba0-env-overrides\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.956806 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956185 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fg9c\" (UniqueName: \"kubernetes.io/projected/0d6daf95-87b4-4f08-8058-6ad1d9ea9dec-kube-api-access-6fg9c\") pod \"node-resolver-g2k4m\" (UID: \"0d6daf95-87b4-4f08-8058-6ad1d9ea9dec\") " pod="openshift-dns/node-resolver-g2k4m" Sep 14 12:58:21.956806 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956195 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/578bd075-5e22-4a15-bde8-57bd2c202ba0-ovnkube-config\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.956806 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956207 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-hostroot\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.956806 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956235 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88m84\" (UniqueName: \"kubernetes.io/projected/921c1e9a-91e6-4f6d-846f-e146d92249ca-kube-api-access-88m84\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.956806 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956250 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-log-socket\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.956806 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956260 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-etc-kubernetes\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.956806 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956312 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-etc-kubernetes\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.956806 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956354 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-multus-socket-dir-parent\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.956806 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956369 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d49b2943-be96-4d9a-ab55-965845ba03f9-dbus\") pod \"global-pull-secret-syncer-72fhn\" (UID: \"d49b2943-be96-4d9a-ab55-965845ba03f9\") " pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:21.956806 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956385 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d49b2943-be96-4d9a-ab55-965845ba03f9-kubelet-config\") pod \"global-pull-secret-syncer-72fhn\" (UID: \"d49b2943-be96-4d9a-ab55-965845ba03f9\") " pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:21.957690 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956451 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.957690 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956417 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.957690 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956486 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-systemd-units\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.957690 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956501 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.957690 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956510 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-run-netns\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.957690 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956509 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-host-var-lib-kubelet\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.957690 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956553 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-host-var-lib-cni-multus\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.957690 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956563 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-device-dir\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.957690 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956552 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.957690 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956585 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-run-ovn\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.957690 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956610 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-cni-bin\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.957690 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956616 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/570dacc2-2bf1-4b1a-8f03-3abb696a2ecf-host\") pod \"node-ca-4v75b\" (UID: \"570dacc2-2bf1-4b1a-8f03-3abb696a2ecf\") " pod="openshift-image-registry/node-ca-4v75b" Sep 14 12:58:21.957690 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956642 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-system-cni-dir\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.957690 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956663 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-run-systemd\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.957690 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956683 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-node-log\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.957690 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956675 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-socket-dir\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.957690 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956707 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44km8\" (UniqueName: \"kubernetes.io/projected/578bd075-5e22-4a15-bde8-57bd2c202ba0-kube-api-access-44km8\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.958391 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956732 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs\") pod \"network-metrics-daemon-kv576\" (UID: \"ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63\") " pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:21.958391 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956761 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d6daf95-87b4-4f08-8058-6ad1d9ea9dec-tmp-dir\") pod \"node-resolver-g2k4m\" (UID: \"0d6daf95-87b4-4f08-8058-6ad1d9ea9dec\") " pod="openshift-dns/node-resolver-g2k4m" Sep 14 12:58:21.958391 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956785 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-host-run-k8s-cni-cncf-io\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.958391 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-host-run-netns\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.958391 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956888 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-slash\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.958391 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.956948 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-hostroot\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.958391 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957001 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-run-netns\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.958391 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957015 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0d6daf95-87b4-4f08-8058-6ad1d9ea9dec-hosts-file\") pod \"node-resolver-g2k4m\" (UID: \"0d6daf95-87b4-4f08-8058-6ad1d9ea9dec\") " pod="openshift-dns/node-resolver-g2k4m" Sep 14 12:58:21.958391 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957039 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrmjk\" (UniqueName: \"kubernetes.io/projected/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-kube-api-access-qrmjk\") pod \"network-metrics-daemon-kv576\" (UID: \"ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63\") " pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:21.958391 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957084 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/570dacc2-2bf1-4b1a-8f03-3abb696a2ecf-serviceca\") pod \"node-ca-4v75b\" (UID: \"570dacc2-2bf1-4b1a-8f03-3abb696a2ecf\") " pod="openshift-image-registry/node-ca-4v75b" Sep 14 12:58:21.958391 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957097 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/578bd075-5e22-4a15-bde8-57bd2c202ba0-ovnkube-script-lib\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.958391 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-sys-fs\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.958391 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957149 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-host-var-lib-cni-bin\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.958391 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957190 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-sys-fs\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.958391 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957249 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-cnibin\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.958391 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957285 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-systemd-units\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.958391 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957285 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-etc-openvswitch\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.958391 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957320 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-run-ovn\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.959139 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957365 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/570dacc2-2bf1-4b1a-8f03-3abb696a2ecf-host\") pod \"node-ca-4v75b\" (UID: \"570dacc2-2bf1-4b1a-8f03-3abb696a2ecf\") " pod="openshift-image-registry/node-ca-4v75b" Sep 14 12:58:21.959139 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957414 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-system-cni-dir\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.959139 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957450 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-run-systemd\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.959139 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957487 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-node-log\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.959139 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957511 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/921c1e9a-91e6-4f6d-846f-e146d92249ca-cni-binary-copy\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.959139 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957527 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/570dacc2-2bf1-4b1a-8f03-3abb696a2ecf-serviceca\") pod \"node-ca-4v75b\" (UID: \"570dacc2-2bf1-4b1a-8f03-3abb696a2ecf\") " pod="openshift-image-registry/node-ca-4v75b" Sep 14 12:58:21.959139 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957575 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-host-run-k8s-cni-cncf-io\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.959139 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:21.957637 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 12:58:21.959139 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:21.957681 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs podName:ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:22.457665238 +0000 UTC m=+3.120560848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs") pod "network-metrics-daemon-kv576" (UID: "ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 12:58:21.959139 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957689 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-host-run-netns\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.959139 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957739 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/921c1e9a-91e6-4f6d-846f-e146d92249ca-multus-daemon-config\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.959139 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957743 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-run-openvswitch\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.959139 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957810 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-var-lib-openvswitch\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.959139 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957854 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-run-ovn-kubernetes\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.959139 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957897 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-cni-netd\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.959139 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957943 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d6daf95-87b4-4f08-8058-6ad1d9ea9dec-tmp-dir\") pod \"node-resolver-g2k4m\" (UID: \"0d6daf95-87b4-4f08-8058-6ad1d9ea9dec\") " pod="openshift-dns/node-resolver-g2k4m" Sep 14 12:58:21.959139 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.958020 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-multus-cni-dir\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.959139 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.958069 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d49b2943-be96-4d9a-ab55-965845ba03f9-kubelet-config\") pod \"global-pull-secret-syncer-72fhn\" (UID: \"d49b2943-be96-4d9a-ab55-965845ba03f9\") " pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:21.960038 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.958075 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/921c1e9a-91e6-4f6d-846f-e146d92249ca-multus-conf-dir\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.960038 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.955691 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-registration-dir\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:21.960038 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.958114 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-kubelet\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.960038 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.957734 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/578bd075-5e22-4a15-bde8-57bd2c202ba0-host-slash\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.960038 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.959142 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/578bd075-5e22-4a15-bde8-57bd2c202ba0-ovn-node-metrics-cert\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.963881 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.963861 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fllzn\" (UniqueName: \"kubernetes.io/projected/570dacc2-2bf1-4b1a-8f03-3abb696a2ecf-kube-api-access-fllzn\") pod \"node-ca-4v75b\" (UID: \"570dacc2-2bf1-4b1a-8f03-3abb696a2ecf\") " pod="openshift-image-registry/node-ca-4v75b" Sep 14 12:58:21.965239 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:21.965222 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 12:58:21.965340 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:21.965243 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 12:58:21.965340 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:21.965254 2564 projected.go:196] Error preparing data for projected volume kube-api-access-6v2x2 for pod openshift-network-diagnostics/network-check-target-7t274: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 12:58:21.965340 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:21.965320 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2 podName:746cbe95-b362-49b9-a280-e4c1faf73d32 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:22.465305918 +0000 UTC m=+3.128201563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6v2x2" (UniqueName: "kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2") pod "network-check-target-7t274" (UID: "746cbe95-b362-49b9-a280-e4c1faf73d32") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 12:58:21.966678 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.966657 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88m84\" (UniqueName: \"kubernetes.io/projected/921c1e9a-91e6-4f6d-846f-e146d92249ca-kube-api-access-88m84\") pod \"multus-fs7nx\" (UID: \"921c1e9a-91e6-4f6d-846f-e146d92249ca\") " pod="openshift-multus/multus-fs7nx" Sep 14 12:58:21.967677 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.967634 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44km8\" (UniqueName: \"kubernetes.io/projected/578bd075-5e22-4a15-bde8-57bd2c202ba0-kube-api-access-44km8\") pod \"ovnkube-node-b62sc\" (UID: \"578bd075-5e22-4a15-bde8-57bd2c202ba0\") " pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:21.967849 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.967834 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fg9c\" (UniqueName: \"kubernetes.io/projected/0d6daf95-87b4-4f08-8058-6ad1d9ea9dec-kube-api-access-6fg9c\") pod \"node-resolver-g2k4m\" (UID: \"0d6daf95-87b4-4f08-8058-6ad1d9ea9dec\") " pod="openshift-dns/node-resolver-g2k4m" Sep 14 12:58:21.969534 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.969513 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrmjk\" (UniqueName: \"kubernetes.io/projected/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-kube-api-access-qrmjk\") pod \"network-metrics-daemon-kv576\" (UID: \"ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63\") " pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:21.969534 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:21.969527 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49hmq\" (UniqueName: \"kubernetes.io/projected/dc3e8575-06c6-44d8-b486-ed4fca68c1cc-kube-api-access-49hmq\") pod \"aws-ebs-csi-driver-node-5t92l\" (UID: \"dc3e8575-06c6-44d8-b486-ed4fca68c1cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:22.048956 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.048862 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rxz94" Sep 14 12:58:22.055750 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.055725 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6h4bn" Sep 14 12:58:22.064676 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.064656 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-slcjt" Sep 14 12:58:22.069428 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.069407 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v64ww" Sep 14 12:58:22.075980 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.075960 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4v75b" Sep 14 12:58:22.082649 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.082632 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-g2k4m" Sep 14 12:58:22.089953 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.089918 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" Sep 14 12:58:22.097582 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.097562 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fs7nx" Sep 14 12:58:22.103461 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.103445 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:22.178960 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.178936 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 12:58:22.459652 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.459621 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs\") pod \"network-metrics-daemon-kv576\" (UID: \"ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63\") " pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:22.459823 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.459658 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret\") pod \"global-pull-secret-syncer-72fhn\" (UID: \"d49b2943-be96-4d9a-ab55-965845ba03f9\") " pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:22.459823 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:22.459773 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 12:58:22.459823 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:22.459790 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 12:58:22.460021 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:22.459834 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret podName:d49b2943-be96-4d9a-ab55-965845ba03f9 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:23.459820234 +0000 UTC m=+4.122715842 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret") pod "global-pull-secret-syncer-72fhn" (UID: "d49b2943-be96-4d9a-ab55-965845ba03f9") : object "kube-system"/"original-pull-secret" not registered Sep 14 12:58:22.460021 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:22.459849 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs podName:ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:23.459842537 +0000 UTC m=+4.122738145 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs") pod "network-metrics-daemon-kv576" (UID: "ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 12:58:22.560757 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.560716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v2x2\" (UniqueName: \"kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2\") pod \"network-check-target-7t274\" (UID: \"746cbe95-b362-49b9-a280-e4c1faf73d32\") " pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:22.560964 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:22.560871 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 12:58:22.560964 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:22.560892 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 12:58:22.560964 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:22.560901 2564 projected.go:196] Error preparing data for projected volume kube-api-access-6v2x2 for pod openshift-network-diagnostics/network-check-target-7t274: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 12:58:22.561125 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:22.560968 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2 podName:746cbe95-b362-49b9-a280-e4c1faf73d32 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:23.560953391 +0000 UTC m=+4.223849019 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6v2x2" (UniqueName: "kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2") pod "network-check-target-7t274" (UID: "746cbe95-b362-49b9-a280-e4c1faf73d32") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 12:58:22.719800 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:22.719777 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba142181_460d_4266_a361_bb44c7fbb63e.slice/crio-46680666daab5d8a543fae6da137b9ac8247619a1c199d1a57b445a9a7178180 WatchSource:0}: Error finding container 46680666daab5d8a543fae6da137b9ac8247619a1c199d1a57b445a9a7178180: Status 404 returned error can't find the container with id 46680666daab5d8a543fae6da137b9ac8247619a1c199d1a57b445a9a7178180 Sep 14 12:58:22.721269 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:22.721240 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9567688_99e2_46d9_b2ce_007605405aea.slice/crio-3a7322658220f65d7a51e6554bd480f23969450d89c0fddf7a7a5f18fbb3be02 WatchSource:0}: Error finding container 3a7322658220f65d7a51e6554bd480f23969450d89c0fddf7a7a5f18fbb3be02: Status 404 returned error can't find the container with id 3a7322658220f65d7a51e6554bd480f23969450d89c0fddf7a7a5f18fbb3be02 Sep 14 12:58:22.724788 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:22.724764 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod921c1e9a_91e6_4f6d_846f_e146d92249ca.slice/crio-11f26c110bf40377604e2eab77265ddc0a0ff48e3a5050aad34b4e84a921b04f WatchSource:0}: Error finding container 11f26c110bf40377604e2eab77265ddc0a0ff48e3a5050aad34b4e84a921b04f: Status 404 returned error can't find the container with id 11f26c110bf40377604e2eab77265ddc0a0ff48e3a5050aad34b4e84a921b04f Sep 14 12:58:22.725246 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:22.725187 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3e8575_06c6_44d8_b486_ed4fca68c1cc.slice/crio-ce10401434a1f9fee243fabf8a1bfd6b1aaf81cd337e0103fa508022f6b0fb34 WatchSource:0}: Error finding container ce10401434a1f9fee243fabf8a1bfd6b1aaf81cd337e0103fa508022f6b0fb34: Status 404 returned error can't find the container with id ce10401434a1f9fee243fabf8a1bfd6b1aaf81cd337e0103fa508022f6b0fb34 Sep 14 12:58:22.726145 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:22.726124 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44b8a44_25cf_4fec_a6ad_79aef4b55753.slice/crio-8f663b91f324a0bb8845a4152ac642d4d5414b04e4243d28114273b3bbcaa651 WatchSource:0}: Error finding container 8f663b91f324a0bb8845a4152ac642d4d5414b04e4243d28114273b3bbcaa651: Status 404 returned error can't find the container with id 8f663b91f324a0bb8845a4152ac642d4d5414b04e4243d28114273b3bbcaa651 Sep 14 12:58:22.727134 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:22.727088 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578bd075_5e22_4a15_bde8_57bd2c202ba0.slice/crio-8b37468d015c6ae62d5288caba9ca1d9010744333d766368a0520de6bde03e12 WatchSource:0}: Error finding container 8b37468d015c6ae62d5288caba9ca1d9010744333d766368a0520de6bde03e12: Status 404 returned error can't find the container with id 8b37468d015c6ae62d5288caba9ca1d9010744333d766368a0520de6bde03e12 Sep 14 12:58:22.728562 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:22.728538 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570dacc2_2bf1_4b1a_8f03_3abb696a2ecf.slice/crio-10310743bf9d9ac3ccccbb8869bd1fb91f5057daa97777c923e7536159520907 WatchSource:0}: Error finding container 10310743bf9d9ac3ccccbb8869bd1fb91f5057daa97777c923e7536159520907: Status 404 returned error can't find the container with id 10310743bf9d9ac3ccccbb8869bd1fb91f5057daa97777c923e7536159520907 Sep 14 12:58:22.730658 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:22.730257 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00612ad7_80a8_41a4_90fd_28e632d42e8d.slice/crio-ad49a204d329d6fe0be740cc53fd84bbfa2d958e96ea72039e23632222c54ce9 WatchSource:0}: Error finding container ad49a204d329d6fe0be740cc53fd84bbfa2d958e96ea72039e23632222c54ce9: Status 404 returned error can't find the container with id ad49a204d329d6fe0be740cc53fd84bbfa2d958e96ea72039e23632222c54ce9 Sep 14 12:58:22.731037 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:22.731000 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d6daf95_87b4_4f08_8058_6ad1d9ea9dec.slice/crio-413684e376140454b5c67c14ea095755b73374d5ecfa1013773196b64fe4cbf3 WatchSource:0}: Error finding container 413684e376140454b5c67c14ea095755b73374d5ecfa1013773196b64fe4cbf3: Status 404 returned error can't find the container with id 413684e376140454b5c67c14ea095755b73374d5ecfa1013773196b64fe4cbf3 Sep 14 12:58:22.790604 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.790576 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 12:53:21 +0000 UTC" deadline="2028-03-26 16:51:12.924368778 +0000 UTC" Sep 14 12:58:22.790604 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.790603 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13419h52m50.133769245s" Sep 14 12:58:22.804970 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.804949 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:22.805078 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:22.805060 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kv576" podUID="ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63" Sep 14 12:58:22.814195 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.814173 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" event={"ID":"dc3e8575-06c6-44d8-b486-ed4fca68c1cc","Type":"ContainerStarted","Data":"ce10401434a1f9fee243fabf8a1bfd6b1aaf81cd337e0103fa508022f6b0fb34"} Sep 14 12:58:22.815252 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.815231 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4v75b" event={"ID":"570dacc2-2bf1-4b1a-8f03-3abb696a2ecf","Type":"ContainerStarted","Data":"10310743bf9d9ac3ccccbb8869bd1fb91f5057daa97777c923e7536159520907"} Sep 14 12:58:22.816308 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.816290 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rxz94" event={"ID":"d9567688-99e2-46d9-b2ce-007605405aea","Type":"ContainerStarted","Data":"3a7322658220f65d7a51e6554bd480f23969450d89c0fddf7a7a5f18fbb3be02"} Sep 14 12:58:22.818624 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.818603 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g2k4m" event={"ID":"0d6daf95-87b4-4f08-8058-6ad1d9ea9dec","Type":"ContainerStarted","Data":"413684e376140454b5c67c14ea095755b73374d5ecfa1013773196b64fe4cbf3"} Sep 14 12:58:22.820342 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.820315 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-slcjt" event={"ID":"00612ad7-80a8-41a4-90fd-28e632d42e8d","Type":"ContainerStarted","Data":"ad49a204d329d6fe0be740cc53fd84bbfa2d958e96ea72039e23632222c54ce9"} Sep 14 12:58:22.821747 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.821727 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" event={"ID":"578bd075-5e22-4a15-bde8-57bd2c202ba0","Type":"ContainerStarted","Data":"8b37468d015c6ae62d5288caba9ca1d9010744333d766368a0520de6bde03e12"} Sep 14 12:58:22.823029 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.823005 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6h4bn" event={"ID":"f44b8a44-25cf-4fec-a6ad-79aef4b55753","Type":"ContainerStarted","Data":"8f663b91f324a0bb8845a4152ac642d4d5414b04e4243d28114273b3bbcaa651"} Sep 14 12:58:22.824334 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.824312 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fs7nx" event={"ID":"921c1e9a-91e6-4f6d-846f-e146d92249ca","Type":"ContainerStarted","Data":"11f26c110bf40377604e2eab77265ddc0a0ff48e3a5050aad34b4e84a921b04f"} Sep 14 12:58:22.825655 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.825633 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v64ww" event={"ID":"ba142181-460d-4266-a361-bb44c7fbb63e","Type":"ContainerStarted","Data":"46680666daab5d8a543fae6da137b9ac8247619a1c199d1a57b445a9a7178180"} Sep 14 12:58:22.906310 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:22.906278 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 14 12:58:23.466743 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:23.466702 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs\") pod \"network-metrics-daemon-kv576\" (UID: \"ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63\") " pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:23.466996 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:23.466763 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret\") pod \"global-pull-secret-syncer-72fhn\" (UID: \"d49b2943-be96-4d9a-ab55-965845ba03f9\") " pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:23.466996 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:23.466918 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 12:58:23.467122 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:23.466999 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret podName:d49b2943-be96-4d9a-ab55-965845ba03f9 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:25.466981298 +0000 UTC m=+6.129876923 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret") pod "global-pull-secret-syncer-72fhn" (UID: "d49b2943-be96-4d9a-ab55-965845ba03f9") : object "kube-system"/"original-pull-secret" not registered Sep 14 12:58:23.467409 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:23.467386 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 12:58:23.467503 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:23.467452 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs podName:ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:25.467428381 +0000 UTC m=+6.130324005 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs") pod "network-metrics-daemon-kv576" (UID: "ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 12:58:23.567709 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:23.567670 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v2x2\" (UniqueName: \"kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2\") pod \"network-check-target-7t274\" (UID: \"746cbe95-b362-49b9-a280-e4c1faf73d32\") " pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:23.567898 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:23.567827 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 12:58:23.567898 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:23.567845 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 12:58:23.567898 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:23.567857 2564 projected.go:196] Error preparing data for projected volume kube-api-access-6v2x2 for pod openshift-network-diagnostics/network-check-target-7t274: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 12:58:23.568088 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:23.567936 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2 podName:746cbe95-b362-49b9-a280-e4c1faf73d32 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:25.567900376 +0000 UTC m=+6.230796010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6v2x2" (UniqueName: "kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2") pod "network-check-target-7t274" (UID: "746cbe95-b362-49b9-a280-e4c1faf73d32") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 12:58:23.791501 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:23.791407 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-13 12:53:21 +0000 UTC" deadline="2028-02-27 07:39:06.144483258 +0000 UTC" Sep 14 12:58:23.791501 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:23.791453 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12738h40m42.35303511s" Sep 14 12:58:23.810459 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:23.810414 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:23.810603 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:23.810550 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72fhn" podUID="d49b2943-be96-4d9a-ab55-965845ba03f9" Sep 14 12:58:23.810951 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:23.810912 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:23.811042 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:23.811020 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7t274" podUID="746cbe95-b362-49b9-a280-e4c1faf73d32" Sep 14 12:58:23.843189 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:23.843158 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" event={"ID":"bd507eff9b3716da03d385309d794351","Type":"ContainerStarted","Data":"b275cbd27ed5a3688f82c49159d3404cb79d86e2965d326fe4bd3fea3c3d6e0e"} Sep 14 12:58:23.848961 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:23.848615 2564 generic.go:356] "Generic (PLEG): container finished" podID="e46b070927ba07e09c7fdfca6f7f4f91" containerID="ecf23c1829950c946521d319ddeab469d48f305753ed905e1f98e65da0f06727" exitCode=0 Sep 14 12:58:23.848961 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:23.848652 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" event={"ID":"e46b070927ba07e09c7fdfca6f7f4f91","Type":"ContainerDied","Data":"ecf23c1829950c946521d319ddeab469d48f305753ed905e1f98e65da0f06727"} Sep 14 12:58:23.865043 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:23.864981 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-96.ec2.internal" podStartSLOduration=2.864961336 podStartE2EDuration="2.864961336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:23.864327526 +0000 UTC m=+4.527223157" watchObservedRunningTime="2026-09-14 12:58:23.864961336 +0000 UTC m=+4.527856967" Sep 14 12:58:24.805345 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:24.805284 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:24.805854 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:24.805423 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kv576" podUID="ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63" Sep 14 12:58:24.886339 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:24.886018 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" event={"ID":"e46b070927ba07e09c7fdfca6f7f4f91","Type":"ContainerStarted","Data":"d8415cd8ddd7b72595907a079af9a88b030ab94fca7e47ac351756b372e98b61"} Sep 14 12:58:25.483251 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:25.482513 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs\") pod \"network-metrics-daemon-kv576\" (UID: \"ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63\") " pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:25.483251 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:25.482578 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret\") pod \"global-pull-secret-syncer-72fhn\" (UID: \"d49b2943-be96-4d9a-ab55-965845ba03f9\") " pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:25.483251 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:25.482698 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 12:58:25.483251 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:25.482762 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret podName:d49b2943-be96-4d9a-ab55-965845ba03f9 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:29.482742957 +0000 UTC m=+10.145638590 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret") pod "global-pull-secret-syncer-72fhn" (UID: "d49b2943-be96-4d9a-ab55-965845ba03f9") : object "kube-system"/"original-pull-secret" not registered Sep 14 12:58:25.483251 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:25.482834 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 12:58:25.483251 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:25.482868 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs podName:ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:29.482856963 +0000 UTC m=+10.145752588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs") pod "network-metrics-daemon-kv576" (UID: "ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 12:58:25.583480 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:25.583442 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v2x2\" (UniqueName: \"kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2\") pod \"network-check-target-7t274\" (UID: \"746cbe95-b362-49b9-a280-e4c1faf73d32\") " pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:25.583687 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:25.583665 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 12:58:25.583762 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:25.583699 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 12:58:25.583762 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:25.583713 2564 projected.go:196] Error preparing data for projected volume kube-api-access-6v2x2 for pod openshift-network-diagnostics/network-check-target-7t274: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 12:58:25.583857 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:25.583780 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2 podName:746cbe95-b362-49b9-a280-e4c1faf73d32 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:29.583759831 +0000 UTC m=+10.246655457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6v2x2" (UniqueName: "kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2") pod "network-check-target-7t274" (UID: "746cbe95-b362-49b9-a280-e4c1faf73d32") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 12:58:25.806122 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:25.806041 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:25.806489 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:25.806174 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7t274" podUID="746cbe95-b362-49b9-a280-e4c1faf73d32" Sep 14 12:58:25.806650 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:25.806627 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:25.806757 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:25.806733 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72fhn" podUID="d49b2943-be96-4d9a-ab55-965845ba03f9" Sep 14 12:58:26.805650 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:26.805620 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:26.805846 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:26.805729 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kv576" podUID="ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63" Sep 14 12:58:27.806692 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:27.806011 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:27.806692 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:27.806139 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7t274" podUID="746cbe95-b362-49b9-a280-e4c1faf73d32" Sep 14 12:58:27.806692 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:27.806551 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:27.806692 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:27.806644 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72fhn" podUID="d49b2943-be96-4d9a-ab55-965845ba03f9" Sep 14 12:58:28.806418 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:28.805690 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:28.806418 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:28.806064 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kv576" podUID="ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63" Sep 14 12:58:29.518965 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:29.518279 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret\") pod \"global-pull-secret-syncer-72fhn\" (UID: \"d49b2943-be96-4d9a-ab55-965845ba03f9\") " pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:29.518965 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:29.518375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs\") pod \"network-metrics-daemon-kv576\" (UID: \"ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63\") " pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:29.518965 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:29.518489 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 12:58:29.518965 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:29.518548 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs podName:ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:37.518528749 +0000 UTC m=+18.181424372 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs") pod "network-metrics-daemon-kv576" (UID: "ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 12:58:29.518965 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:29.518856 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 12:58:29.518965 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:29.518891 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret podName:d49b2943-be96-4d9a-ab55-965845ba03f9 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:37.518881592 +0000 UTC m=+18.181777201 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret") pod "global-pull-secret-syncer-72fhn" (UID: "d49b2943-be96-4d9a-ab55-965845ba03f9") : object "kube-system"/"original-pull-secret" not registered Sep 14 12:58:29.618730 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:29.618686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v2x2\" (UniqueName: \"kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2\") pod \"network-check-target-7t274\" (UID: \"746cbe95-b362-49b9-a280-e4c1faf73d32\") " pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:29.618891 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:29.618862 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 12:58:29.618891 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:29.618885 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 12:58:29.618980 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:29.618898 2564 projected.go:196] Error preparing data for projected volume kube-api-access-6v2x2 for pod openshift-network-diagnostics/network-check-target-7t274: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 12:58:29.618980 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:29.618963 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2 podName:746cbe95-b362-49b9-a280-e4c1faf73d32 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:37.618948749 +0000 UTC m=+18.281844370 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6v2x2" (UniqueName: "kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2") pod "network-check-target-7t274" (UID: "746cbe95-b362-49b9-a280-e4c1faf73d32") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 12:58:29.806083 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:29.806007 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:29.806229 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:29.806128 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7t274" podUID="746cbe95-b362-49b9-a280-e4c1faf73d32" Sep 14 12:58:29.806229 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:29.806214 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:29.806347 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:29.806317 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72fhn" podUID="d49b2943-be96-4d9a-ab55-965845ba03f9" Sep 14 12:58:30.806975 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:30.805241 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:30.806975 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:30.805388 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kv576" podUID="ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63" Sep 14 12:58:31.805186 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:31.805150 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:31.805368 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:31.805158 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:31.805368 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:31.805267 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72fhn" podUID="d49b2943-be96-4d9a-ab55-965845ba03f9" Sep 14 12:58:31.805368 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:31.805322 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7t274" podUID="746cbe95-b362-49b9-a280-e4c1faf73d32" Sep 14 12:58:32.805867 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:32.805836 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:32.806329 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:32.805974 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kv576" podUID="ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63" Sep 14 12:58:33.805741 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:33.805707 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:33.805892 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:33.805825 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7t274" podUID="746cbe95-b362-49b9-a280-e4c1faf73d32" Sep 14 12:58:33.805892 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:33.805873 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:33.806299 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:33.806005 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72fhn" podUID="d49b2943-be96-4d9a-ab55-965845ba03f9" Sep 14 12:58:34.805653 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:34.805618 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:34.805824 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:34.805752 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kv576" podUID="ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63" Sep 14 12:58:35.805958 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:35.805908 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:35.806409 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:35.805908 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:35.806409 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:35.806061 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7t274" podUID="746cbe95-b362-49b9-a280-e4c1faf73d32" Sep 14 12:58:35.806409 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:35.806111 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72fhn" podUID="d49b2943-be96-4d9a-ab55-965845ba03f9" Sep 14 12:58:36.805297 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:36.805252 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:36.805554 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:36.805380 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kv576" podUID="ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63" Sep 14 12:58:37.571871 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:37.571825 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs\") pod \"network-metrics-daemon-kv576\" (UID: \"ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63\") " pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:37.572320 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:37.571892 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret\") pod \"global-pull-secret-syncer-72fhn\" (UID: \"d49b2943-be96-4d9a-ab55-965845ba03f9\") " pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:37.572320 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:37.571987 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 12:58:37.572320 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:37.572044 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs podName:ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:53.572029712 +0000 UTC m=+34.234925320 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs") pod "network-metrics-daemon-kv576" (UID: "ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 12:58:37.572320 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:37.572063 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 12:58:37.572320 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:37.572135 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret podName:d49b2943-be96-4d9a-ab55-965845ba03f9 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:53.572116156 +0000 UTC m=+34.235011784 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret") pod "global-pull-secret-syncer-72fhn" (UID: "d49b2943-be96-4d9a-ab55-965845ba03f9") : object "kube-system"/"original-pull-secret" not registered Sep 14 12:58:37.673188 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:37.673156 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v2x2\" (UniqueName: \"kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2\") pod \"network-check-target-7t274\" (UID: \"746cbe95-b362-49b9-a280-e4c1faf73d32\") " pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:37.673378 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:37.673340 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 12:58:37.673378 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:37.673363 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 12:58:37.673378 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:37.673373 2564 projected.go:196] Error preparing data for projected volume kube-api-access-6v2x2 for pod openshift-network-diagnostics/network-check-target-7t274: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 12:58:37.673536 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:37.673425 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2 podName:746cbe95-b362-49b9-a280-e4c1faf73d32 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:53.673408828 +0000 UTC m=+34.336304453 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6v2x2" (UniqueName: "kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2") pod "network-check-target-7t274" (UID: "746cbe95-b362-49b9-a280-e4c1faf73d32") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 12:58:37.805400 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:37.805367 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:37.805551 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:37.805367 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:37.805551 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:37.805497 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7t274" podUID="746cbe95-b362-49b9-a280-e4c1faf73d32" Sep 14 12:58:37.805650 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:37.805580 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72fhn" podUID="d49b2943-be96-4d9a-ab55-965845ba03f9" Sep 14 12:58:38.805433 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:38.805402 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:38.805911 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:38.805518 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kv576" podUID="ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63" Sep 14 12:58:39.806222 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:39.806195 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:39.806700 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:39.806286 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72fhn" podUID="d49b2943-be96-4d9a-ab55-965845ba03f9" Sep 14 12:58:39.806700 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:39.806363 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:39.806700 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:39.806442 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7t274" podUID="746cbe95-b362-49b9-a280-e4c1faf73d32" Sep 14 12:58:40.805284 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:40.805255 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:40.805436 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:40.805353 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kv576" podUID="ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63" Sep 14 12:58:41.805771 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:41.805588 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:41.806607 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:41.805664 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:41.806607 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:41.805880 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72fhn" podUID="d49b2943-be96-4d9a-ab55-965845ba03f9" Sep 14 12:58:41.806607 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:41.805916 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7t274" podUID="746cbe95-b362-49b9-a280-e4c1faf73d32" Sep 14 12:58:41.924302 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:41.924242 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-g2k4m" event={"ID":"0d6daf95-87b4-4f08-8058-6ad1d9ea9dec","Type":"ContainerStarted","Data":"e169efa60104df6cb097e6ab36cf556f432a883c31db5c7574713b557ef71909"} Sep 14 12:58:41.925765 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:41.925740 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-slcjt" event={"ID":"00612ad7-80a8-41a4-90fd-28e632d42e8d","Type":"ContainerStarted","Data":"abe79303f1aa50a6a0c171d03c1fae251adced1a0222435d07540f7f910ab657"} Sep 14 12:58:41.929832 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:41.929790 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" event={"ID":"578bd075-5e22-4a15-bde8-57bd2c202ba0","Type":"ContainerStarted","Data":"ad05c1a2b1e158d3a8b6ac0d8d202011005ea1785cef56f8c12a220d607928cb"} Sep 14 12:58:41.929980 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:41.929920 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" event={"ID":"578bd075-5e22-4a15-bde8-57bd2c202ba0","Type":"ContainerStarted","Data":"df4a28bd1d4caff2d3f499feb83af837b8b9765a7dabedd5b127e7de632f465c"} Sep 14 12:58:41.930061 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:41.930006 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" event={"ID":"578bd075-5e22-4a15-bde8-57bd2c202ba0","Type":"ContainerStarted","Data":"50b06382cbf8549d93d163934fdbd5be21220332442206f374c32fcd1b1fe19b"} Sep 14 12:58:41.930113 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:41.930065 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" event={"ID":"578bd075-5e22-4a15-bde8-57bd2c202ba0","Type":"ContainerStarted","Data":"f29551faa5cd1b8ab7bf4cda94ab7521d8a1f4a864daf6cce948fe5a56295890"} Sep 14 12:58:41.930160 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:41.930126 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" event={"ID":"578bd075-5e22-4a15-bde8-57bd2c202ba0","Type":"ContainerStarted","Data":"88c4e2195fa12fcd008d2bda400b5e631a8faf8d612d380b8cdba04f3a4526f2"} Sep 14 12:58:41.930213 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:41.930190 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" event={"ID":"578bd075-5e22-4a15-bde8-57bd2c202ba0","Type":"ContainerStarted","Data":"8e4bc7c85347a29eb2ad8fcd4f65d176440616fdfc691b08628c50b94426c78c"} Sep 14 12:58:41.932036 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:41.932015 2564 generic.go:356] "Generic (PLEG): container finished" podID="f44b8a44-25cf-4fec-a6ad-79aef4b55753" containerID="608ffe3188a3af304f2f52f09cdc91987881bdc072102708069ceb7672b84874" exitCode=0 Sep 14 12:58:41.932114 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:41.932080 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6h4bn" event={"ID":"f44b8a44-25cf-4fec-a6ad-79aef4b55753","Type":"ContainerDied","Data":"608ffe3188a3af304f2f52f09cdc91987881bdc072102708069ceb7672b84874"} Sep 14 12:58:41.933566 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:41.933548 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fs7nx" event={"ID":"921c1e9a-91e6-4f6d-846f-e146d92249ca","Type":"ContainerStarted","Data":"a482ce182a62d4b0f21357d41a2c3a7c0403766ece41eb9079b3521d8057db96"} Sep 14 12:58:41.934952 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:41.934914 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v64ww" event={"ID":"ba142181-460d-4266-a361-bb44c7fbb63e","Type":"ContainerStarted","Data":"d652cc71cbea4c72c99d70cc50464fecdbdd5db87ce3c4c8bec6ba236e1a4dfb"} Sep 14 12:58:41.936246 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:41.936228 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" event={"ID":"dc3e8575-06c6-44d8-b486-ed4fca68c1cc","Type":"ContainerStarted","Data":"ce3683932046fbab8375ead64c879dcce0674916374dfc759c18f75a3457b394"} Sep 14 12:58:41.937484 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:41.937466 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4v75b" event={"ID":"570dacc2-2bf1-4b1a-8f03-3abb696a2ecf","Type":"ContainerStarted","Data":"b1a9d89670fef2aba40557eb1d416d87ca72e594ac822f8c6a550210bd487424"} Sep 14 12:58:41.939596 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:41.939562 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-96.ec2.internal" podStartSLOduration=20.939537632 podStartE2EDuration="20.939537632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:24.903768905 +0000 UTC m=+5.566664532" watchObservedRunningTime="2026-09-14 12:58:41.939537632 +0000 UTC m=+22.602433256" Sep 14 12:58:41.939794 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:41.939773 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-g2k4m" podStartSLOduration=3.78120272 podStartE2EDuration="21.93976558s" totalImagesPullingTime="18.15856286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:41.939739677 +0000 UTC m=+22.602635306" watchObservedRunningTime="2026-09-14 12:58:41.93976558 +0000 UTC m=+22.602661210" Sep 14 12:58:41.971418 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:41.971371 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fs7nx" podStartSLOduration=3.745058772 podStartE2EDuration="21.971355458s" totalImagesPullingTime="18.226296686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:41.971205661 +0000 UTC m=+22.634101303" watchObservedRunningTime="2026-09-14 12:58:41.971355458 +0000 UTC m=+22.634251089" Sep 14 12:58:41.971963 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:41.971918 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4v75b" podStartSLOduration=4.813618635 podStartE2EDuration="22.97191006s" totalImagesPullingTime="18.158291425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:41.953677763 +0000 UTC m=+22.616573393" watchObservedRunningTime="2026-09-14 12:58:41.97191006 +0000 UTC m=+22.634805689" Sep 14 12:58:42.007015 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:42.006973 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-slcjt" podStartSLOduration=4.912079179 podStartE2EDuration="23.006961457s" totalImagesPullingTime="18.094882278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:42.006484121 +0000 UTC m=+22.669379751" watchObservedRunningTime="2026-09-14 12:58:42.006961457 +0000 UTC m=+22.669857086" Sep 14 12:58:42.022864 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:42.022820 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v64ww" podStartSLOduration=4.844799476 podStartE2EDuration="23.022807847s" totalImagesPullingTime="18.178008371s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:42.022380019 +0000 UTC m=+22.685275648" watchObservedRunningTime="2026-09-14 12:58:42.022807847 +0000 UTC m=+22.685703517" Sep 14 12:58:42.309696 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:42.309548 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 14 12:58:42.451031 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:42.450952 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-slcjt" Sep 14 12:58:42.805945 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:42.805849 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:42.806429 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:42.806017 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kv576" podUID="ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63" Sep 14 12:58:42.819011 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:42.818882 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-14T12:58:42.309689463Z","UUID":"849c10bc-e4da-4be2-addf-a0ca62c68cc1","Handler":null,"Name":"","Endpoint":""} Sep 14 12:58:42.820766 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:42.820742 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 14 12:58:42.820904 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:42.820780 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 14 12:58:42.942079 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:42.942042 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rxz94" event={"ID":"d9567688-99e2-46d9-b2ce-007605405aea","Type":"ContainerStarted","Data":"18aae4ea7884a0d34b678e2da72a548955c2cd8ff9452226e32092143835bce3"} Sep 14 12:58:42.944542 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:42.944480 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" event={"ID":"dc3e8575-06c6-44d8-b486-ed4fca68c1cc","Type":"ContainerStarted","Data":"5c263742e7137984cf39f8767aec3bc7c8ad933a7b91c4eb232f84a1bb6f5f4c"} Sep 14 12:58:42.958149 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:42.958110 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rxz94" podStartSLOduration=5.7828475 podStartE2EDuration="23.958092881s" totalImagesPullingTime="18.175245381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:42.957968479 +0000 UTC m=+23.620864110" watchObservedRunningTime="2026-09-14 12:58:42.958092881 +0000 UTC m=+23.620988568" Sep 14 12:58:43.805803 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:43.805772 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:43.805993 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:43.805895 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7t274" podUID="746cbe95-b362-49b9-a280-e4c1faf73d32" Sep 14 12:58:43.806354 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:43.805996 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:43.806354 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:43.806104 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72fhn" podUID="d49b2943-be96-4d9a-ab55-965845ba03f9" Sep 14 12:58:43.950871 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:43.950829 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" event={"ID":"578bd075-5e22-4a15-bde8-57bd2c202ba0","Type":"ContainerStarted","Data":"ab379b75388ca6bd657a8f7792f4d455a6bbf8278fa8282236bb3337e766e32a"} Sep 14 12:58:43.953336 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:43.953300 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" event={"ID":"dc3e8575-06c6-44d8-b486-ed4fca68c1cc","Type":"ContainerStarted","Data":"4ce9eabf2844870265f68cb574fe498957a830ad0d2b10561b31dc3af618801b"} Sep 14 12:58:43.972417 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:43.972373 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5t92l" podStartSLOduration=3.690487874 podStartE2EDuration="23.972359184s" totalImagesPullingTime="20.28187131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:43.971766815 +0000 UTC m=+24.634662445" watchObservedRunningTime="2026-09-14 12:58:43.972359184 +0000 UTC m=+24.635254814" Sep 14 12:58:44.805962 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:44.805910 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:44.806145 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:44.806051 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kv576" podUID="ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63" Sep 14 12:58:45.808988 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:45.808958 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:45.809401 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:45.809068 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7t274" podUID="746cbe95-b362-49b9-a280-e4c1faf73d32" Sep 14 12:58:45.809491 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:45.809447 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:45.809565 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:45.809533 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72fhn" podUID="d49b2943-be96-4d9a-ab55-965845ba03f9" Sep 14 12:58:46.640060 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:46.639795 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-slcjt" Sep 14 12:58:46.640517 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:46.640490 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-slcjt" Sep 14 12:58:46.805957 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:46.805916 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:46.806053 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:46.806035 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kv576" podUID="ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63" Sep 14 12:58:46.963818 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:46.963732 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" event={"ID":"578bd075-5e22-4a15-bde8-57bd2c202ba0","Type":"ContainerStarted","Data":"1be3b95b441ded51a406c91928742e5aaa2bb20dbfff291172d0272258096c67"} Sep 14 12:58:46.964357 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:46.963971 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:46.965639 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:46.965613 2564 generic.go:356] "Generic (PLEG): container finished" podID="f44b8a44-25cf-4fec-a6ad-79aef4b55753" containerID="98e5382870e8ca54c0502bfb4113d99c86bbe6b581da30d6e1fd53a05407e126" exitCode=0 Sep 14 12:58:46.965759 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:46.965701 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6h4bn" event={"ID":"f44b8a44-25cf-4fec-a6ad-79aef4b55753","Type":"ContainerDied","Data":"98e5382870e8ca54c0502bfb4113d99c86bbe6b581da30d6e1fd53a05407e126"} Sep 14 12:58:46.966404 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:46.966386 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-slcjt" Sep 14 12:58:46.979838 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:46.979820 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:46.994068 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:46.994031 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" podStartSLOduration=8.778898931 podStartE2EDuration="26.994018411s" totalImagesPullingTime="18.21511948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:46.992472079 +0000 UTC m=+27.655367706" watchObservedRunningTime="2026-09-14 12:58:46.994018411 +0000 UTC m=+27.656914040" Sep 14 12:58:47.808866 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:47.808835 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:47.809141 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:47.808835 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:47.809141 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:47.808965 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7t274" podUID="746cbe95-b362-49b9-a280-e4c1faf73d32" Sep 14 12:58:47.809141 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:47.809015 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72fhn" podUID="d49b2943-be96-4d9a-ab55-965845ba03f9" Sep 14 12:58:47.968410 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:47.968379 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:47.968917 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:47.968468 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:47.983264 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:47.983236 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:58:48.480124 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:48.480091 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-72fhn"] Sep 14 12:58:48.480281 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:48.480241 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:48.480355 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:48.480336 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72fhn" podUID="d49b2943-be96-4d9a-ab55-965845ba03f9" Sep 14 12:58:48.483974 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:48.483949 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7t274"] Sep 14 12:58:48.484105 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:48.484047 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:48.484175 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:48.484139 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7t274" podUID="746cbe95-b362-49b9-a280-e4c1faf73d32" Sep 14 12:58:48.484653 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:48.484627 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kv576"] Sep 14 12:58:48.484763 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:48.484728 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:48.484822 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:48.484804 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kv576" podUID="ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63" Sep 14 12:58:48.972110 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:48.972080 2564 generic.go:356] "Generic (PLEG): container finished" podID="f44b8a44-25cf-4fec-a6ad-79aef4b55753" containerID="0cc1215d689292bca1e24309744b633e597bf2a8f47f1d09a5577d3d018e6004" exitCode=0 Sep 14 12:58:48.972691 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:48.972169 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6h4bn" event={"ID":"f44b8a44-25cf-4fec-a6ad-79aef4b55753","Type":"ContainerDied","Data":"0cc1215d689292bca1e24309744b633e597bf2a8f47f1d09a5577d3d018e6004"} Sep 14 12:58:49.805845 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:49.805656 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:49.806166 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:49.805946 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kv576" podUID="ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63" Sep 14 12:58:49.806166 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:49.805716 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:49.806166 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:49.806066 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72fhn" podUID="d49b2943-be96-4d9a-ab55-965845ba03f9" Sep 14 12:58:50.805612 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:50.805537 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:50.805973 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:50.805636 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7t274" podUID="746cbe95-b362-49b9-a280-e4c1faf73d32" Sep 14 12:58:50.979109 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:50.979077 2564 generic.go:356] "Generic (PLEG): container finished" podID="f44b8a44-25cf-4fec-a6ad-79aef4b55753" containerID="47d7086b4c1c5947a1b66a4045b421d0f2a839eba65e0eb1acdae902491bc8cc" exitCode=0 Sep 14 12:58:50.979258 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:50.979125 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6h4bn" event={"ID":"f44b8a44-25cf-4fec-a6ad-79aef4b55753","Type":"ContainerDied","Data":"47d7086b4c1c5947a1b66a4045b421d0f2a839eba65e0eb1acdae902491bc8cc"} Sep 14 12:58:51.808488 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:51.808456 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:51.809044 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:51.808470 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:51.809044 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:51.808578 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-72fhn" podUID="d49b2943-be96-4d9a-ab55-965845ba03f9" Sep 14 12:58:51.809044 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:51.808655 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kv576" podUID="ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63" Sep 14 12:58:52.804990 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:52.804950 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:52.805192 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:52.805084 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7t274" podUID="746cbe95-b362-49b9-a280-e4c1faf73d32" Sep 14 12:58:53.586575 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.586534 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs\") pod \"network-metrics-daemon-kv576\" (UID: \"ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63\") " pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:53.587088 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.586594 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret\") pod \"global-pull-secret-syncer-72fhn\" (UID: \"d49b2943-be96-4d9a-ab55-965845ba03f9\") " pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:53.587088 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:53.586708 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 14 12:58:53.587088 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:53.586713 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 12:58:53.587088 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:53.586770 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret podName:d49b2943-be96-4d9a-ab55-965845ba03f9 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:25.586753144 +0000 UTC m=+66.249648776 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret") pod "global-pull-secret-syncer-72fhn" (UID: "d49b2943-be96-4d9a-ab55-965845ba03f9") : object "kube-system"/"original-pull-secret" not registered Sep 14 12:58:53.587088 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:53.586788 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs podName:ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:25.586781358 +0000 UTC m=+66.249676966 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs") pod "network-metrics-daemon-kv576" (UID: "ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 14 12:58:53.615242 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.615212 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-96.ec2.internal" event="NodeReady" Sep 14 12:58:53.615400 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.615358 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 14 12:58:53.658433 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.658397 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qr6v6"] Sep 14 12:58:53.661331 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.661305 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dskx8"] Sep 14 12:58:53.661331 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.661325 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qr6v6" Sep 14 12:58:53.663127 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.663107 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dskx8" Sep 14 12:58:53.664562 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.664542 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 14 12:58:53.664680 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.664574 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xjstd\"" Sep 14 12:58:53.664680 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.664668 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 14 12:58:53.667405 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.665577 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 14 12:58:53.667405 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.665859 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 14 12:58:53.667852 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.667831 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 14 12:58:53.667852 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.667842 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5hqfk\"" Sep 14 12:58:53.672897 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.672879 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dskx8"] Sep 14 12:58:53.675411 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.675391 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qr6v6"] Sep 14 12:58:53.688130 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.688073 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v2x2\" (UniqueName: \"kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2\") pod \"network-check-target-7t274\" (UID: \"746cbe95-b362-49b9-a280-e4c1faf73d32\") " pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:53.688244 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:53.688221 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 14 12:58:53.688344 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:53.688251 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 14 12:58:53.688344 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:53.688261 2564 projected.go:196] Error preparing data for projected volume kube-api-access-6v2x2 for pod openshift-network-diagnostics/network-check-target-7t274: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 12:58:53.688344 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:53.688333 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2 podName:746cbe95-b362-49b9-a280-e4c1faf73d32 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:25.688315022 +0000 UTC m=+66.351210649 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6v2x2" (UniqueName: "kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2") pod "network-check-target-7t274" (UID: "746cbe95-b362-49b9-a280-e4c1faf73d32") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 14 12:58:53.789149 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.789117 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls\") pod \"dns-default-qr6v6\" (UID: \"dbe829bc-1888-48fe-8416-3ae70353c8d7\") " pod="openshift-dns/dns-default-qr6v6" Sep 14 12:58:53.789334 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.789173 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe829bc-1888-48fe-8416-3ae70353c8d7-tmp-dir\") pod \"dns-default-qr6v6\" (UID: \"dbe829bc-1888-48fe-8416-3ae70353c8d7\") " pod="openshift-dns/dns-default-qr6v6" Sep 14 12:58:53.789334 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.789278 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc512c34-085d-4934-9c57-95f05e3db013-cert\") pod \"ingress-canary-dskx8\" (UID: \"fc512c34-085d-4934-9c57-95f05e3db013\") " pod="openshift-ingress-canary/ingress-canary-dskx8" Sep 14 12:58:53.789334 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.789313 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld2dx\" (UniqueName: \"kubernetes.io/projected/fc512c34-085d-4934-9c57-95f05e3db013-kube-api-access-ld2dx\") pod \"ingress-canary-dskx8\" (UID: \"fc512c34-085d-4934-9c57-95f05e3db013\") " pod="openshift-ingress-canary/ingress-canary-dskx8" Sep 14 12:58:53.789486 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.789335 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe829bc-1888-48fe-8416-3ae70353c8d7-config-volume\") pod \"dns-default-qr6v6\" (UID: \"dbe829bc-1888-48fe-8416-3ae70353c8d7\") " pod="openshift-dns/dns-default-qr6v6" Sep 14 12:58:53.789486 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.789360 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc6qm\" (UniqueName: \"kubernetes.io/projected/dbe829bc-1888-48fe-8416-3ae70353c8d7-kube-api-access-bc6qm\") pod \"dns-default-qr6v6\" (UID: \"dbe829bc-1888-48fe-8416-3ae70353c8d7\") " pod="openshift-dns/dns-default-qr6v6" Sep 14 12:58:53.808704 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.808678 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:58:53.808891 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.808864 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:58:53.811983 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.811842 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 14 12:58:53.811983 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.811853 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 14 12:58:53.811983 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.811850 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9l5s5\"" Sep 14 12:58:53.890749 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.890351 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls\") pod \"dns-default-qr6v6\" (UID: \"dbe829bc-1888-48fe-8416-3ae70353c8d7\") " pod="openshift-dns/dns-default-qr6v6" Sep 14 12:58:53.890749 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.890424 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe829bc-1888-48fe-8416-3ae70353c8d7-tmp-dir\") pod \"dns-default-qr6v6\" (UID: \"dbe829bc-1888-48fe-8416-3ae70353c8d7\") " pod="openshift-dns/dns-default-qr6v6" Sep 14 12:58:53.890749 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.890479 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc512c34-085d-4934-9c57-95f05e3db013-cert\") pod \"ingress-canary-dskx8\" (UID: \"fc512c34-085d-4934-9c57-95f05e3db013\") " pod="openshift-ingress-canary/ingress-canary-dskx8" Sep 14 12:58:53.890749 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.890496 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld2dx\" (UniqueName: \"kubernetes.io/projected/fc512c34-085d-4934-9c57-95f05e3db013-kube-api-access-ld2dx\") pod \"ingress-canary-dskx8\" (UID: \"fc512c34-085d-4934-9c57-95f05e3db013\") " pod="openshift-ingress-canary/ingress-canary-dskx8" Sep 14 12:58:53.890749 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.890516 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe829bc-1888-48fe-8416-3ae70353c8d7-config-volume\") pod \"dns-default-qr6v6\" (UID: \"dbe829bc-1888-48fe-8416-3ae70353c8d7\") " pod="openshift-dns/dns-default-qr6v6" Sep 14 12:58:53.890749 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:53.890523 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 12:58:53.890749 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.890567 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc6qm\" (UniqueName: \"kubernetes.io/projected/dbe829bc-1888-48fe-8416-3ae70353c8d7-kube-api-access-bc6qm\") pod \"dns-default-qr6v6\" (UID: \"dbe829bc-1888-48fe-8416-3ae70353c8d7\") " pod="openshift-dns/dns-default-qr6v6" Sep 14 12:58:53.890749 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:53.890591 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls podName:dbe829bc-1888-48fe-8416-3ae70353c8d7 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:54.390571401 +0000 UTC m=+35.053467025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls") pod "dns-default-qr6v6" (UID: "dbe829bc-1888-48fe-8416-3ae70353c8d7") : secret "dns-default-metrics-tls" not found Sep 14 12:58:53.891307 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.890850 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe829bc-1888-48fe-8416-3ae70353c8d7-tmp-dir\") pod \"dns-default-qr6v6\" (UID: \"dbe829bc-1888-48fe-8416-3ae70353c8d7\") " pod="openshift-dns/dns-default-qr6v6" Sep 14 12:58:53.891307 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.891114 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe829bc-1888-48fe-8416-3ae70353c8d7-config-volume\") pod \"dns-default-qr6v6\" (UID: \"dbe829bc-1888-48fe-8416-3ae70353c8d7\") " pod="openshift-dns/dns-default-qr6v6" Sep 14 12:58:53.895216 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.895193 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc512c34-085d-4934-9c57-95f05e3db013-cert\") pod \"ingress-canary-dskx8\" (UID: \"fc512c34-085d-4934-9c57-95f05e3db013\") " pod="openshift-ingress-canary/ingress-canary-dskx8" Sep 14 12:58:53.901376 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.901351 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld2dx\" (UniqueName: \"kubernetes.io/projected/fc512c34-085d-4934-9c57-95f05e3db013-kube-api-access-ld2dx\") pod \"ingress-canary-dskx8\" (UID: \"fc512c34-085d-4934-9c57-95f05e3db013\") " pod="openshift-ingress-canary/ingress-canary-dskx8" Sep 14 12:58:53.901469 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.901351 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc6qm\" (UniqueName: \"kubernetes.io/projected/dbe829bc-1888-48fe-8416-3ae70353c8d7-kube-api-access-bc6qm\") pod \"dns-default-qr6v6\" (UID: \"dbe829bc-1888-48fe-8416-3ae70353c8d7\") " pod="openshift-dns/dns-default-qr6v6" Sep 14 12:58:53.982202 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:53.982129 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dskx8" Sep 14 12:58:54.393673 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:54.393639 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls\") pod \"dns-default-qr6v6\" (UID: \"dbe829bc-1888-48fe-8416-3ae70353c8d7\") " pod="openshift-dns/dns-default-qr6v6" Sep 14 12:58:54.393820 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:54.393786 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 12:58:54.393888 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:54.393858 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls podName:dbe829bc-1888-48fe-8416-3ae70353c8d7 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:55.39383861 +0000 UTC m=+36.056734238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls") pod "dns-default-qr6v6" (UID: "dbe829bc-1888-48fe-8416-3ae70353c8d7") : secret "dns-default-metrics-tls" not found Sep 14 12:58:54.805781 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:54.805747 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:58:54.808874 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:54.808845 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 14 12:58:54.809039 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:54.808878 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tlghq\"" Sep 14 12:58:54.809039 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:54.808847 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 14 12:58:55.207389 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:55.207183 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dskx8"] Sep 14 12:58:55.213755 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:55.213724 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc512c34_085d_4934_9c57_95f05e3db013.slice/crio-508049c8238f8a653862f10d5cb25e771e035982f9a193de97e6e903cf7eb48a WatchSource:0}: Error finding container 508049c8238f8a653862f10d5cb25e771e035982f9a193de97e6e903cf7eb48a: Status 404 returned error can't find the container with id 508049c8238f8a653862f10d5cb25e771e035982f9a193de97e6e903cf7eb48a Sep 14 12:58:55.400360 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:55.400330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls\") pod \"dns-default-qr6v6\" (UID: \"dbe829bc-1888-48fe-8416-3ae70353c8d7\") " pod="openshift-dns/dns-default-qr6v6" Sep 14 12:58:55.400518 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:55.400498 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 12:58:55.400585 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:55.400573 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls podName:dbe829bc-1888-48fe-8416-3ae70353c8d7 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:57.400552549 +0000 UTC m=+38.063448157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls") pod "dns-default-qr6v6" (UID: "dbe829bc-1888-48fe-8416-3ae70353c8d7") : secret "dns-default-metrics-tls" not found Sep 14 12:58:55.998249 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:55.998209 2564 generic.go:356] "Generic (PLEG): container finished" podID="f44b8a44-25cf-4fec-a6ad-79aef4b55753" containerID="08082aad8cae04e057b7b7e505ce276b5261f7b279a105546e1de901d28b6ce7" exitCode=0 Sep 14 12:58:55.998901 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:55.998299 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6h4bn" event={"ID":"f44b8a44-25cf-4fec-a6ad-79aef4b55753","Type":"ContainerDied","Data":"08082aad8cae04e057b7b7e505ce276b5261f7b279a105546e1de901d28b6ce7"} Sep 14 12:58:55.999503 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:55.999483 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dskx8" event={"ID":"fc512c34-085d-4934-9c57-95f05e3db013","Type":"ContainerStarted","Data":"508049c8238f8a653862f10d5cb25e771e035982f9a193de97e6e903cf7eb48a"} Sep 14 12:58:57.005806 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:57.005770 2564 generic.go:356] "Generic (PLEG): container finished" podID="f44b8a44-25cf-4fec-a6ad-79aef4b55753" containerID="1cbd564c0e3909fb2e803076742cf685742bc9f2599c83f991d8f3ebbcf9c02f" exitCode=0 Sep 14 12:58:57.006492 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:57.005831 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6h4bn" event={"ID":"f44b8a44-25cf-4fec-a6ad-79aef4b55753","Type":"ContainerDied","Data":"1cbd564c0e3909fb2e803076742cf685742bc9f2599c83f991d8f3ebbcf9c02f"} Sep 14 12:58:57.415046 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:57.415023 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls\") pod \"dns-default-qr6v6\" (UID: \"dbe829bc-1888-48fe-8416-3ae70353c8d7\") " pod="openshift-dns/dns-default-qr6v6" Sep 14 12:58:57.415182 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:57.415163 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 12:58:57.415233 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:57.415227 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls podName:dbe829bc-1888-48fe-8416-3ae70353c8d7 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:01.415211836 +0000 UTC m=+42.078107448 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls") pod "dns-default-qr6v6" (UID: "dbe829bc-1888-48fe-8416-3ae70353c8d7") : secret "dns-default-metrics-tls" not found Sep 14 12:58:58.009403 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.009218 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dskx8" event={"ID":"fc512c34-085d-4934-9c57-95f05e3db013","Type":"ContainerStarted","Data":"1b4e445789fe456f61e5dcd6f74325fc8dbb0751e04ac048851db0faa51d64a9"} Sep 14 12:58:58.012805 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.012783 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6h4bn" event={"ID":"f44b8a44-25cf-4fec-a6ad-79aef4b55753","Type":"ContainerStarted","Data":"2750b9da62ba3ccd454252d3235fbdddf24cdb2b040e2f310fb75f18d53bbffe"} Sep 14 12:58:58.048316 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.048189 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6h4bn" podStartSLOduration=10.031213029 podStartE2EDuration="39.048169973s" totalImagesPullingTime="29.016956944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:58.046767128 +0000 UTC m=+38.709662758" watchObservedRunningTime="2026-09-14 12:58:58.048169973 +0000 UTC m=+38.711065628" Sep 14 12:58:58.048465 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.048352 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dskx8" podStartSLOduration=2.873474538 podStartE2EDuration="5.048346153s" totalImagesPullingTime="2.174871615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:58:58.026332002 +0000 UTC m=+38.689227634" watchObservedRunningTime="2026-09-14 12:58:58.048346153 +0000 UTC m=+38.711241792" Sep 14 12:58:58.111713 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.111674 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-577p2"] Sep 14 12:58:58.150956 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.150906 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76"] Sep 14 12:58:58.151108 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.151021 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-577p2" Sep 14 12:58:58.153722 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.153700 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 14 12:58:58.153858 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.153744 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zfrjv\"" Sep 14 12:58:58.153858 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.153750 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 14 12:58:58.174938 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.174899 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2vff4"] Sep 14 12:58:58.175061 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.175033 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76" Sep 14 12:58:58.178089 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.178064 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 14 12:58:58.178231 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.178068 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 14 12:58:58.178231 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.178162 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-f65mr\"" Sep 14 12:58:58.178231 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.178210 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 14 12:58:58.199087 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.199060 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-577p2"] Sep 14 12:58:58.199087 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.199091 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-l78kn"] Sep 14 12:58:58.199237 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.199178 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2vff4" Sep 14 12:58:58.201666 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.201648 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 14 12:58:58.201827 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.201813 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 14 12:58:58.201887 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.201852 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 14 12:58:58.201952 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.201898 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 14 12:58:58.201952 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.201902 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jcqfh\"" Sep 14 12:58:58.220024 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.219996 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgc4b\" (UniqueName: \"kubernetes.io/projected/97519ad0-2873-4913-90da-1ba4dcfa7d9f-kube-api-access-zgc4b\") pod \"volume-data-source-validator-86695b7cc7-577p2\" (UID: \"97519ad0-2873-4913-90da-1ba4dcfa7d9f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-577p2" Sep 14 12:58:58.223096 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.223079 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2vff4"] Sep 14 12:58:58.223149 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.223107 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76"] Sep 14 12:58:58.223149 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.223132 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-l78kn"] Sep 14 12:58:58.223206 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.223161 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8"] Sep 14 12:58:58.223206 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.223201 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l78kn" Sep 14 12:58:58.227884 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.227865 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 14 12:58:58.227995 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.227895 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 14 12:58:58.228528 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.228354 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 14 12:58:58.228631 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.228537 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 14 12:58:58.228682 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.228669 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-7k9kz\"" Sep 14 12:58:58.247216 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.247193 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-wx768"] Sep 14 12:58:58.247339 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.247317 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8" Sep 14 12:58:58.250234 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.250205 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 14 12:58:58.251011 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.250912 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 14 12:58:58.251011 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.250973 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7cmvz\"" Sep 14 12:58:58.254031 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.253880 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 14 12:58:58.254122 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.254102 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 14 12:58:58.271191 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.271165 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-2xl57"] Sep 14 12:58:58.271300 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.271277 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-wx768" Sep 14 12:58:58.273715 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.273699 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-cszch\"" Sep 14 12:58:58.295291 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.295271 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f5cb7776d-tzhs8"] Sep 14 12:58:58.295398 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.295386 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-2xl57" Sep 14 12:58:58.297877 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.297862 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 14 12:58:58.297986 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.297967 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 14 12:58:58.298111 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.298098 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 14 12:58:58.298348 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.298300 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-tcz6b\"" Sep 14 12:58:58.298530 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.298513 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 14 12:58:58.303401 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.303377 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 14 12:58:58.315456 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.315415 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-64988bbc7d-tnjdw"] Sep 14 12:58:58.315539 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.315528 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.318749 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.318734 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 14 12:58:58.318994 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.318979 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 14 12:58:58.319196 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.319181 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 14 12:58:58.319273 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.319198 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8dx6q\"" Sep 14 12:58:58.320263 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.320245 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-b9nh8\" (UID: \"547938cd-bd0a-42d0-ac25-9406cc46fe95\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8" Sep 14 12:58:58.320398 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.320270 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnwlj\" (UniqueName: \"kubernetes.io/projected/547938cd-bd0a-42d0-ac25-9406cc46fe95-kube-api-access-hnwlj\") pod \"cluster-monitoring-operator-6cd954b897-b9nh8\" (UID: \"547938cd-bd0a-42d0-ac25-9406cc46fe95\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8" Sep 14 12:58:58.320398 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.320298 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-6kx76\" (UID: \"f8af22e7-1c34-404e-bc73-803803349989\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76" Sep 14 12:58:58.320398 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.320323 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/544ba1bf-58c6-428c-9830-a456e4940b63-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-2vff4\" (UID: \"544ba1bf-58c6-428c-9830-a456e4940b63\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2vff4" Sep 14 12:58:58.320398 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.320379 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgc4b\" (UniqueName: \"kubernetes.io/projected/97519ad0-2873-4913-90da-1ba4dcfa7d9f-kube-api-access-zgc4b\") pod \"volume-data-source-validator-86695b7cc7-577p2\" (UID: \"97519ad0-2873-4913-90da-1ba4dcfa7d9f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-577p2" Sep 14 12:58:58.320398 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.320397 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9nwl\" (UniqueName: \"kubernetes.io/projected/3be5d339-0117-4588-a03e-8562bc978355-kube-api-access-w9nwl\") pod \"service-ca-operator-cf7c776b7-l78kn\" (UID: \"3be5d339-0117-4588-a03e-8562bc978355\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l78kn" Sep 14 12:58:58.320650 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.320418 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/544ba1bf-58c6-428c-9830-a456e4940b63-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-2vff4\" (UID: \"544ba1bf-58c6-428c-9830-a456e4940b63\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2vff4" Sep 14 12:58:58.320650 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.320460 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf4pm\" (UniqueName: \"kubernetes.io/projected/f8af22e7-1c34-404e-bc73-803803349989-kube-api-access-gf4pm\") pod \"cluster-samples-operator-86499c89c4-6kx76\" (UID: \"f8af22e7-1c34-404e-bc73-803803349989\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76" Sep 14 12:58:58.320650 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.320502 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3be5d339-0117-4588-a03e-8562bc978355-serving-cert\") pod \"service-ca-operator-cf7c776b7-l78kn\" (UID: \"3be5d339-0117-4588-a03e-8562bc978355\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l78kn" Sep 14 12:58:58.320650 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.320535 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/547938cd-bd0a-42d0-ac25-9406cc46fe95-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-b9nh8\" (UID: \"547938cd-bd0a-42d0-ac25-9406cc46fe95\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8" Sep 14 12:58:58.320650 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.320550 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3be5d339-0117-4588-a03e-8562bc978355-config\") pod \"service-ca-operator-cf7c776b7-l78kn\" (UID: \"3be5d339-0117-4588-a03e-8562bc978355\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l78kn" Sep 14 12:58:58.320650 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.320581 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbg2w\" (UniqueName: \"kubernetes.io/projected/544ba1bf-58c6-428c-9830-a456e4940b63-kube-api-access-gbg2w\") pod \"kube-storage-version-migrator-operator-6c8f84f897-2vff4\" (UID: \"544ba1bf-58c6-428c-9830-a456e4940b63\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2vff4" Sep 14 12:58:58.323737 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.323720 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 14 12:58:58.333859 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.333842 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-wx768"] Sep 14 12:58:58.333984 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.333867 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-2xl57"] Sep 14 12:58:58.333984 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.333880 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8"] Sep 14 12:58:58.333984 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.333896 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f5cb7776d-tzhs8"] Sep 14 12:58:58.333984 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.333908 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64988bbc7d-tnjdw"] Sep 14 12:58:58.333984 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.333945 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-nn8d7"] Sep 14 12:58:58.333984 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.333968 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:58:58.336624 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.336608 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 14 12:58:58.336718 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.336609 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 14 12:58:58.336718 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.336651 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 14 12:58:58.336718 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.336659 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xm75b\"" Sep 14 12:58:58.336718 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.336612 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 14 12:58:58.336915 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.336873 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 14 12:58:58.337014 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.337001 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 14 12:58:58.344876 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.344851 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgc4b\" (UniqueName: \"kubernetes.io/projected/97519ad0-2873-4913-90da-1ba4dcfa7d9f-kube-api-access-zgc4b\") pod \"volume-data-source-validator-86695b7cc7-577p2\" (UID: \"97519ad0-2873-4913-90da-1ba4dcfa7d9f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-577p2" Sep 14 12:58:58.349745 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.349727 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-2995b"] Sep 14 12:58:58.349843 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.349827 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nn8d7" Sep 14 12:58:58.352832 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.352808 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-d464v\"" Sep 14 12:58:58.353112 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.353094 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 14 12:58:58.353345 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.353328 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 14 12:58:58.365898 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.365879 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-nn8d7"] Sep 14 12:58:58.366016 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.365914 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-2995b"] Sep 14 12:58:58.366090 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.366075 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-2995b" Sep 14 12:58:58.368807 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.368776 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 14 12:58:58.368904 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.368808 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 14 12:58:58.368904 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.368838 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 14 12:58:58.368904 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.368854 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-4bvzl\"" Sep 14 12:58:58.369135 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.369119 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 14 12:58:58.373628 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.373610 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 14 12:58:58.421496 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.421465 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhnkj\" (UniqueName: \"kubernetes.io/projected/8d33c9fb-e4e0-468c-b81d-a3955aec6267-kube-api-access-vhnkj\") pod \"network-check-source-6f8b768455-wx768\" (UID: \"8d33c9fb-e4e0-468c-b81d-a3955aec6267\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-wx768" Sep 14 12:58:58.421496 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.421498 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf4pm\" (UniqueName: \"kubernetes.io/projected/f8af22e7-1c34-404e-bc73-803803349989-kube-api-access-gf4pm\") pod \"cluster-samples-operator-86499c89c4-6kx76\" (UID: \"f8af22e7-1c34-404e-bc73-803803349989\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76" Sep 14 12:58:58.421668 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.421521 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-certificates\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.421668 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.421539 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3be5d339-0117-4588-a03e-8562bc978355-serving-cert\") pod \"service-ca-operator-cf7c776b7-l78kn\" (UID: \"3be5d339-0117-4588-a03e-8562bc978355\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l78kn" Sep 14 12:58:58.421668 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.421649 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1d79bb2f-3305-4485-a140-a2f5c5e1b009-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-nn8d7\" (UID: \"1d79bb2f-3305-4485-a140-a2f5c5e1b009\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nn8d7" Sep 14 12:58:58.421760 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.421715 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad5768be-fcc1-4c56-a485-2ae3f04504bc-service-ca-bundle\") pod \"insights-operator-5844b856db-2xl57\" (UID: \"ad5768be-fcc1-4c56-a485-2ae3f04504bc\") " pod="openshift-insights/insights-operator-5844b856db-2xl57" Sep 14 12:58:58.421760 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.421738 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccd8v\" (UniqueName: \"kubernetes.io/projected/0287d99a-ca2c-45bc-8f32-a2a273401e51-kube-api-access-ccd8v\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:58:58.421839 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.421773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3be5d339-0117-4588-a03e-8562bc978355-config\") pod \"service-ca-operator-cf7c776b7-l78kn\" (UID: \"3be5d339-0117-4588-a03e-8562bc978355\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l78kn" Sep 14 12:58:58.421839 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.421807 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/650e0502-50be-4f10-b3b3-7cf515fdb02d-image-registry-private-configuration\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.421958 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.421897 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-stats-auth\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:58:58.422011 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.421971 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbg2w\" (UniqueName: \"kubernetes.io/projected/544ba1bf-58c6-428c-9830-a456e4940b63-kube-api-access-gbg2w\") pod \"kube-storage-version-migrator-operator-6c8f84f897-2vff4\" (UID: \"544ba1bf-58c6-428c-9830-a456e4940b63\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2vff4" Sep 14 12:58:58.422069 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.422061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:58:58.422126 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.422084 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9nwl\" (UniqueName: \"kubernetes.io/projected/3be5d339-0117-4588-a03e-8562bc978355-kube-api-access-w9nwl\") pod \"service-ca-operator-cf7c776b7-l78kn\" (UID: \"3be5d339-0117-4588-a03e-8562bc978355\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l78kn" Sep 14 12:58:58.422126 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.422106 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad5768be-fcc1-4c56-a485-2ae3f04504bc-serving-cert\") pod \"insights-operator-5844b856db-2xl57\" (UID: \"ad5768be-fcc1-4c56-a485-2ae3f04504bc\") " pod="openshift-insights/insights-operator-5844b856db-2xl57" Sep 14 12:58:58.422223 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.422129 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-6kx76\" (UID: \"f8af22e7-1c34-404e-bc73-803803349989\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76" Sep 14 12:58:58.422223 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.422177 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8cgr\" (UniqueName: \"kubernetes.io/projected/ad5768be-fcc1-4c56-a485-2ae3f04504bc-kube-api-access-k8cgr\") pod \"insights-operator-5844b856db-2xl57\" (UID: \"ad5768be-fcc1-4c56-a485-2ae3f04504bc\") " pod="openshift-insights/insights-operator-5844b856db-2xl57" Sep 14 12:58:58.422223 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:58.422210 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 14 12:58:58.422362 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.422231 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fdc65821-255d-43be-9ccc-7dffc5a49791-serving-cert\") pod \"console-operator-696884fb9c-2995b\" (UID: \"fdc65821-255d-43be-9ccc-7dffc5a49791\") " pod="openshift-console-operator/console-operator-696884fb9c-2995b" Sep 14 12:58:58.422362 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:58.422266 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls podName:f8af22e7-1c34-404e-bc73-803803349989 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:58.922250268 +0000 UTC m=+39.585145875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-6kx76" (UID: "f8af22e7-1c34-404e-bc73-803803349989") : secret "samples-operator-tls" not found Sep 14 12:58:58.422362 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.422285 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/544ba1bf-58c6-428c-9830-a456e4940b63-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-2vff4\" (UID: \"544ba1bf-58c6-428c-9830-a456e4940b63\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2vff4" Sep 14 12:58:58.422362 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.422327 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.422362 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.422349 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/650e0502-50be-4f10-b3b3-7cf515fdb02d-trusted-ca\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.422578 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.422397 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad5768be-fcc1-4c56-a485-2ae3f04504bc-tmp\") pod \"insights-operator-5844b856db-2xl57\" (UID: \"ad5768be-fcc1-4c56-a485-2ae3f04504bc\") " pod="openshift-insights/insights-operator-5844b856db-2xl57" Sep 14 12:58:58.422578 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.422421 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/650e0502-50be-4f10-b3b3-7cf515fdb02d-installation-pull-secrets\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.422578 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.422420 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3be5d339-0117-4588-a03e-8562bc978355-config\") pod \"service-ca-operator-cf7c776b7-l78kn\" (UID: \"3be5d339-0117-4588-a03e-8562bc978355\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l78kn" Sep 14 12:58:58.422578 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.422455 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/547938cd-bd0a-42d0-ac25-9406cc46fe95-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-b9nh8\" (UID: \"547938cd-bd0a-42d0-ac25-9406cc46fe95\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8" Sep 14 12:58:58.422578 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.422475 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ad5768be-fcc1-4c56-a485-2ae3f04504bc-snapshots\") pod \"insights-operator-5844b856db-2xl57\" (UID: \"ad5768be-fcc1-4c56-a485-2ae3f04504bc\") " pod="openshift-insights/insights-operator-5844b856db-2xl57" Sep 14 12:58:58.422578 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.422561 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:58:58.422876 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.422601 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad5768be-fcc1-4c56-a485-2ae3f04504bc-trusted-ca-bundle\") pod \"insights-operator-5844b856db-2xl57\" (UID: \"ad5768be-fcc1-4c56-a485-2ae3f04504bc\") " pod="openshift-insights/insights-operator-5844b856db-2xl57" Sep 14 12:58:58.422876 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.422643 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fdc65821-255d-43be-9ccc-7dffc5a49791-config\") pod \"console-operator-696884fb9c-2995b\" (UID: \"fdc65821-255d-43be-9ccc-7dffc5a49791\") " pod="openshift-console-operator/console-operator-696884fb9c-2995b" Sep 14 12:58:58.422876 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.422714 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-b9nh8\" (UID: \"547938cd-bd0a-42d0-ac25-9406cc46fe95\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8" Sep 14 12:58:58.422876 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.422742 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwlj\" (UniqueName: \"kubernetes.io/projected/547938cd-bd0a-42d0-ac25-9406cc46fe95-kube-api-access-hnwlj\") pod \"cluster-monitoring-operator-6cd954b897-b9nh8\" (UID: \"547938cd-bd0a-42d0-ac25-9406cc46fe95\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8" Sep 14 12:58:58.423124 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:58.422894 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 14 12:58:58.423124 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.422955 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-default-certificate\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:58:58.423124 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:58.422965 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls podName:547938cd-bd0a-42d0-ac25-9406cc46fe95 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:58.922948023 +0000 UTC m=+39.585843635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-b9nh8" (UID: "547938cd-bd0a-42d0-ac25-9406cc46fe95") : secret "cluster-monitoring-operator-tls" not found Sep 14 12:58:58.423124 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.423004 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nn8d7\" (UID: \"1d79bb2f-3305-4485-a140-a2f5c5e1b009\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nn8d7" Sep 14 12:58:58.423124 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.423043 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fdc65821-255d-43be-9ccc-7dffc5a49791-trusted-ca\") pod \"console-operator-696884fb9c-2995b\" (UID: \"fdc65821-255d-43be-9ccc-7dffc5a49791\") " pod="openshift-console-operator/console-operator-696884fb9c-2995b" Sep 14 12:58:58.423124 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.423066 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsmmr\" (UniqueName: \"kubernetes.io/projected/fdc65821-255d-43be-9ccc-7dffc5a49791-kube-api-access-jsmmr\") pod \"console-operator-696884fb9c-2995b\" (UID: \"fdc65821-255d-43be-9ccc-7dffc5a49791\") " pod="openshift-console-operator/console-operator-696884fb9c-2995b" Sep 14 12:58:58.423330 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.423125 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/544ba1bf-58c6-428c-9830-a456e4940b63-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-2vff4\" (UID: \"544ba1bf-58c6-428c-9830-a456e4940b63\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2vff4" Sep 14 12:58:58.423330 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.423146 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/650e0502-50be-4f10-b3b3-7cf515fdb02d-ca-trust-extracted\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.423330 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.423166 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-bound-sa-token\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.423330 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.423180 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44dzv\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-kube-api-access-44dzv\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.423485 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.423457 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/547938cd-bd0a-42d0-ac25-9406cc46fe95-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-b9nh8\" (UID: \"547938cd-bd0a-42d0-ac25-9406cc46fe95\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8" Sep 14 12:58:58.423665 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.423646 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/544ba1bf-58c6-428c-9830-a456e4940b63-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-2vff4\" (UID: \"544ba1bf-58c6-428c-9830-a456e4940b63\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2vff4" Sep 14 12:58:58.424579 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.424560 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3be5d339-0117-4588-a03e-8562bc978355-serving-cert\") pod \"service-ca-operator-cf7c776b7-l78kn\" (UID: \"3be5d339-0117-4588-a03e-8562bc978355\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l78kn" Sep 14 12:58:58.424770 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.424754 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/544ba1bf-58c6-428c-9830-a456e4940b63-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-2vff4\" (UID: \"544ba1bf-58c6-428c-9830-a456e4940b63\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2vff4" Sep 14 12:58:58.432655 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.432597 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbg2w\" (UniqueName: \"kubernetes.io/projected/544ba1bf-58c6-428c-9830-a456e4940b63-kube-api-access-gbg2w\") pod \"kube-storage-version-migrator-operator-6c8f84f897-2vff4\" (UID: \"544ba1bf-58c6-428c-9830-a456e4940b63\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2vff4" Sep 14 12:58:58.433313 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.433286 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf4pm\" (UniqueName: \"kubernetes.io/projected/f8af22e7-1c34-404e-bc73-803803349989-kube-api-access-gf4pm\") pod \"cluster-samples-operator-86499c89c4-6kx76\" (UID: \"f8af22e7-1c34-404e-bc73-803803349989\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76" Sep 14 12:58:58.434042 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.434022 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnwlj\" (UniqueName: \"kubernetes.io/projected/547938cd-bd0a-42d0-ac25-9406cc46fe95-kube-api-access-hnwlj\") pod \"cluster-monitoring-operator-6cd954b897-b9nh8\" (UID: \"547938cd-bd0a-42d0-ac25-9406cc46fe95\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8" Sep 14 12:58:58.434146 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.434131 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9nwl\" (UniqueName: \"kubernetes.io/projected/3be5d339-0117-4588-a03e-8562bc978355-kube-api-access-w9nwl\") pod \"service-ca-operator-cf7c776b7-l78kn\" (UID: \"3be5d339-0117-4588-a03e-8562bc978355\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l78kn" Sep 14 12:58:58.460017 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.459996 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-577p2" Sep 14 12:58:58.508243 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.508222 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2vff4" Sep 14 12:58:58.523517 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.523493 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/650e0502-50be-4f10-b3b3-7cf515fdb02d-ca-trust-extracted\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.523620 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.523526 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-bound-sa-token\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.523620 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.523551 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44dzv\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-kube-api-access-44dzv\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.523620 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.523570 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhnkj\" (UniqueName: \"kubernetes.io/projected/8d33c9fb-e4e0-468c-b81d-a3955aec6267-kube-api-access-vhnkj\") pod \"network-check-source-6f8b768455-wx768\" (UID: \"8d33c9fb-e4e0-468c-b81d-a3955aec6267\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-wx768" Sep 14 12:58:58.523620 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.523591 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-certificates\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.523620 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.523617 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1d79bb2f-3305-4485-a140-a2f5c5e1b009-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-nn8d7\" (UID: \"1d79bb2f-3305-4485-a140-a2f5c5e1b009\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nn8d7" Sep 14 12:58:58.523849 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.523646 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad5768be-fcc1-4c56-a485-2ae3f04504bc-service-ca-bundle\") pod \"insights-operator-5844b856db-2xl57\" (UID: \"ad5768be-fcc1-4c56-a485-2ae3f04504bc\") " pod="openshift-insights/insights-operator-5844b856db-2xl57" Sep 14 12:58:58.523849 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.523670 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccd8v\" (UniqueName: \"kubernetes.io/projected/0287d99a-ca2c-45bc-8f32-a2a273401e51-kube-api-access-ccd8v\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:58:58.523849 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.523696 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/650e0502-50be-4f10-b3b3-7cf515fdb02d-image-registry-private-configuration\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.523849 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.523719 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-stats-auth\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:58:58.523849 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.523757 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:58:58.523849 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.523785 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad5768be-fcc1-4c56-a485-2ae3f04504bc-serving-cert\") pod \"insights-operator-5844b856db-2xl57\" (UID: \"ad5768be-fcc1-4c56-a485-2ae3f04504bc\") " pod="openshift-insights/insights-operator-5844b856db-2xl57" Sep 14 12:58:58.523849 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.523837 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8cgr\" (UniqueName: \"kubernetes.io/projected/ad5768be-fcc1-4c56-a485-2ae3f04504bc-kube-api-access-k8cgr\") pod \"insights-operator-5844b856db-2xl57\" (UID: \"ad5768be-fcc1-4c56-a485-2ae3f04504bc\") " pod="openshift-insights/insights-operator-5844b856db-2xl57" Sep 14 12:58:58.524210 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.523859 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fdc65821-255d-43be-9ccc-7dffc5a49791-serving-cert\") pod \"console-operator-696884fb9c-2995b\" (UID: \"fdc65821-255d-43be-9ccc-7dffc5a49791\") " pod="openshift-console-operator/console-operator-696884fb9c-2995b" Sep 14 12:58:58.524210 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.523948 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/650e0502-50be-4f10-b3b3-7cf515fdb02d-ca-trust-extracted\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.524313 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.524240 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.524313 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.524272 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/650e0502-50be-4f10-b3b3-7cf515fdb02d-trusted-ca\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.524417 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.524334 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad5768be-fcc1-4c56-a485-2ae3f04504bc-tmp\") pod \"insights-operator-5844b856db-2xl57\" (UID: \"ad5768be-fcc1-4c56-a485-2ae3f04504bc\") " pod="openshift-insights/insights-operator-5844b856db-2xl57" Sep 14 12:58:58.524417 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.524358 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-certificates\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.524417 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:58.524361 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 14 12:58:58.524417 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.524393 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1d79bb2f-3305-4485-a140-a2f5c5e1b009-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-nn8d7\" (UID: \"1d79bb2f-3305-4485-a140-a2f5c5e1b009\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nn8d7" Sep 14 12:58:58.524598 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:58.524433 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs podName:0287d99a-ca2c-45bc-8f32-a2a273401e51 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:59.024413881 +0000 UTC m=+39.687309511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs") pod "router-default-64988bbc7d-tnjdw" (UID: "0287d99a-ca2c-45bc-8f32-a2a273401e51") : secret "router-metrics-certs-default" not found Sep 14 12:58:58.524598 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.524361 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/650e0502-50be-4f10-b3b3-7cf515fdb02d-installation-pull-secrets\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.524598 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.524476 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ad5768be-fcc1-4c56-a485-2ae3f04504bc-snapshots\") pod \"insights-operator-5844b856db-2xl57\" (UID: \"ad5768be-fcc1-4c56-a485-2ae3f04504bc\") " pod="openshift-insights/insights-operator-5844b856db-2xl57" Sep 14 12:58:58.524598 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.524488 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad5768be-fcc1-4c56-a485-2ae3f04504bc-service-ca-bundle\") pod \"insights-operator-5844b856db-2xl57\" (UID: \"ad5768be-fcc1-4c56-a485-2ae3f04504bc\") " pod="openshift-insights/insights-operator-5844b856db-2xl57" Sep 14 12:58:58.524598 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.524507 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:58:58.524598 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.524539 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad5768be-fcc1-4c56-a485-2ae3f04504bc-trusted-ca-bundle\") pod \"insights-operator-5844b856db-2xl57\" (UID: \"ad5768be-fcc1-4c56-a485-2ae3f04504bc\") " pod="openshift-insights/insights-operator-5844b856db-2xl57" Sep 14 12:58:58.524598 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.524565 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fdc65821-255d-43be-9ccc-7dffc5a49791-config\") pod \"console-operator-696884fb9c-2995b\" (UID: \"fdc65821-255d-43be-9ccc-7dffc5a49791\") " pod="openshift-console-operator/console-operator-696884fb9c-2995b" Sep 14 12:58:58.524962 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.524616 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-default-certificate\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:58:58.524962 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.524643 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nn8d7\" (UID: \"1d79bb2f-3305-4485-a140-a2f5c5e1b009\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nn8d7" Sep 14 12:58:58.524962 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.524674 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fdc65821-255d-43be-9ccc-7dffc5a49791-trusted-ca\") pod \"console-operator-696884fb9c-2995b\" (UID: \"fdc65821-255d-43be-9ccc-7dffc5a49791\") " pod="openshift-console-operator/console-operator-696884fb9c-2995b" Sep 14 12:58:58.524962 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.524702 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsmmr\" (UniqueName: \"kubernetes.io/projected/fdc65821-255d-43be-9ccc-7dffc5a49791-kube-api-access-jsmmr\") pod \"console-operator-696884fb9c-2995b\" (UID: \"fdc65821-255d-43be-9ccc-7dffc5a49791\") " pod="openshift-console-operator/console-operator-696884fb9c-2995b" Sep 14 12:58:58.525531 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.525507 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/650e0502-50be-4f10-b3b3-7cf515fdb02d-trusted-ca\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.525900 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.525605 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ad5768be-fcc1-4c56-a485-2ae3f04504bc-snapshots\") pod \"insights-operator-5844b856db-2xl57\" (UID: \"ad5768be-fcc1-4c56-a485-2ae3f04504bc\") " pod="openshift-insights/insights-operator-5844b856db-2xl57" Sep 14 12:58:58.525900 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:58.525623 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle podName:0287d99a-ca2c-45bc-8f32-a2a273401e51 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:59.025606415 +0000 UTC m=+39.688502028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle") pod "router-default-64988bbc7d-tnjdw" (UID: "0287d99a-ca2c-45bc-8f32-a2a273401e51") : configmap references non-existent config key: service-ca.crt Sep 14 12:58:58.525900 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.525608 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad5768be-fcc1-4c56-a485-2ae3f04504bc-tmp\") pod \"insights-operator-5844b856db-2xl57\" (UID: \"ad5768be-fcc1-4c56-a485-2ae3f04504bc\") " pod="openshift-insights/insights-operator-5844b856db-2xl57" Sep 14 12:58:58.525900 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:58.525711 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 14 12:58:58.525900 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:58.525758 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert podName:1d79bb2f-3305-4485-a140-a2f5c5e1b009 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:59.025742146 +0000 UTC m=+39.688637771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nn8d7" (UID: "1d79bb2f-3305-4485-a140-a2f5c5e1b009") : secret "networking-console-plugin-cert" not found Sep 14 12:58:58.525900 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:58.525822 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 12:58:58.525900 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:58.525833 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f5cb7776d-tzhs8: secret "image-registry-tls" not found Sep 14 12:58:58.525900 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:58.525869 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls podName:650e0502-50be-4f10-b3b3-7cf515fdb02d nodeName:}" failed. No retries permitted until 2026-09-14 12:58:59.025856085 +0000 UTC m=+39.688751729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls") pod "image-registry-6f5cb7776d-tzhs8" (UID: "650e0502-50be-4f10-b3b3-7cf515fdb02d") : secret "image-registry-tls" not found Sep 14 12:58:58.526357 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.526234 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fdc65821-255d-43be-9ccc-7dffc5a49791-config\") pod \"console-operator-696884fb9c-2995b\" (UID: \"fdc65821-255d-43be-9ccc-7dffc5a49791\") " pod="openshift-console-operator/console-operator-696884fb9c-2995b" Sep 14 12:58:58.526357 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.526333 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad5768be-fcc1-4c56-a485-2ae3f04504bc-trusted-ca-bundle\") pod \"insights-operator-5844b856db-2xl57\" (UID: \"ad5768be-fcc1-4c56-a485-2ae3f04504bc\") " pod="openshift-insights/insights-operator-5844b856db-2xl57" Sep 14 12:58:58.526452 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.526390 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fdc65821-255d-43be-9ccc-7dffc5a49791-trusted-ca\") pod \"console-operator-696884fb9c-2995b\" (UID: \"fdc65821-255d-43be-9ccc-7dffc5a49791\") " pod="openshift-console-operator/console-operator-696884fb9c-2995b" Sep 14 12:58:58.527313 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.527292 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ad5768be-fcc1-4c56-a485-2ae3f04504bc-serving-cert\") pod \"insights-operator-5844b856db-2xl57\" (UID: \"ad5768be-fcc1-4c56-a485-2ae3f04504bc\") " pod="openshift-insights/insights-operator-5844b856db-2xl57" Sep 14 12:58:58.527450 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.527429 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-stats-auth\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:58:58.528131 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.528112 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fdc65821-255d-43be-9ccc-7dffc5a49791-serving-cert\") pod \"console-operator-696884fb9c-2995b\" (UID: \"fdc65821-255d-43be-9ccc-7dffc5a49791\") " pod="openshift-console-operator/console-operator-696884fb9c-2995b" Sep 14 12:58:58.528221 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.528198 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/650e0502-50be-4f10-b3b3-7cf515fdb02d-installation-pull-secrets\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.529147 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.529125 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-default-certificate\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:58:58.529516 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.529497 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/650e0502-50be-4f10-b3b3-7cf515fdb02d-image-registry-private-configuration\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.531454 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.531434 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l78kn" Sep 14 12:58:58.535334 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.535245 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-bound-sa-token\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.536106 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.536058 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8cgr\" (UniqueName: \"kubernetes.io/projected/ad5768be-fcc1-4c56-a485-2ae3f04504bc-kube-api-access-k8cgr\") pod \"insights-operator-5844b856db-2xl57\" (UID: \"ad5768be-fcc1-4c56-a485-2ae3f04504bc\") " pod="openshift-insights/insights-operator-5844b856db-2xl57" Sep 14 12:58:58.536200 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.536138 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhnkj\" (UniqueName: \"kubernetes.io/projected/8d33c9fb-e4e0-468c-b81d-a3955aec6267-kube-api-access-vhnkj\") pod \"network-check-source-6f8b768455-wx768\" (UID: \"8d33c9fb-e4e0-468c-b81d-a3955aec6267\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-wx768" Sep 14 12:58:58.536200 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.536162 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccd8v\" (UniqueName: \"kubernetes.io/projected/0287d99a-ca2c-45bc-8f32-a2a273401e51-kube-api-access-ccd8v\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:58:58.536200 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.536195 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44dzv\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-kube-api-access-44dzv\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:58.538178 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.538159 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsmmr\" (UniqueName: \"kubernetes.io/projected/fdc65821-255d-43be-9ccc-7dffc5a49791-kube-api-access-jsmmr\") pod \"console-operator-696884fb9c-2995b\" (UID: \"fdc65821-255d-43be-9ccc-7dffc5a49791\") " pod="openshift-console-operator/console-operator-696884fb9c-2995b" Sep 14 12:58:58.579260 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.579236 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-wx768" Sep 14 12:58:58.603134 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.602968 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-2xl57" Sep 14 12:58:58.674940 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.674883 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-2995b" Sep 14 12:58:58.769826 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.769761 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-577p2"] Sep 14 12:58:58.773531 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:58.773507 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97519ad0_2873_4913_90da_1ba4dcfa7d9f.slice/crio-07005a404f877597669e41e2f1f4f93edb9d44527770b73d99dbc4af4d7dafb3 WatchSource:0}: Error finding container 07005a404f877597669e41e2f1f4f93edb9d44527770b73d99dbc4af4d7dafb3: Status 404 returned error can't find the container with id 07005a404f877597669e41e2f1f4f93edb9d44527770b73d99dbc4af4d7dafb3 Sep 14 12:58:58.805211 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.805132 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2vff4"] Sep 14 12:58:58.808490 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:58.808462 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544ba1bf_58c6_428c_9830_a456e4940b63.slice/crio-0c16568df7264b5fb49ed4a9f24b578749e2bced21b5c782ad9b21a750270af1 WatchSource:0}: Error finding container 0c16568df7264b5fb49ed4a9f24b578749e2bced21b5c782ad9b21a750270af1: Status 404 returned error can't find the container with id 0c16568df7264b5fb49ed4a9f24b578749e2bced21b5c782ad9b21a750270af1 Sep 14 12:58:58.835347 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.835323 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-l78kn"] Sep 14 12:58:58.838008 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:58.837983 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be5d339_0117_4588_a03e_8562bc978355.slice/crio-bfcadd6f6cc33f806dfe290098c2625b2d61b610a2b5fb0f3f301f0d9dc674f5 WatchSource:0}: Error finding container bfcadd6f6cc33f806dfe290098c2625b2d61b610a2b5fb0f3f301f0d9dc674f5: Status 404 returned error can't find the container with id bfcadd6f6cc33f806dfe290098c2625b2d61b610a2b5fb0f3f301f0d9dc674f5 Sep 14 12:58:58.871742 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.871683 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-wx768"] Sep 14 12:58:58.873906 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:58.873884 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d33c9fb_e4e0_468c_b81d_a3955aec6267.slice/crio-af9194903a444e389fdd14aef6c534d5b7e587394c6013e23af53c6a67732600 WatchSource:0}: Error finding container af9194903a444e389fdd14aef6c534d5b7e587394c6013e23af53c6a67732600: Status 404 returned error can't find the container with id af9194903a444e389fdd14aef6c534d5b7e587394c6013e23af53c6a67732600 Sep 14 12:58:58.904678 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.904651 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-2xl57"] Sep 14 12:58:58.906981 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:58.906961 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5768be_fcc1_4c56_a485_2ae3f04504bc.slice/crio-f3b37b3d9d893a75a1c7f6da13790bb87761384111772a379a91a41efb29eed2 WatchSource:0}: Error finding container f3b37b3d9d893a75a1c7f6da13790bb87761384111772a379a91a41efb29eed2: Status 404 returned error can't find the container with id f3b37b3d9d893a75a1c7f6da13790bb87761384111772a379a91a41efb29eed2 Sep 14 12:58:58.928189 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.928167 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-b9nh8\" (UID: \"547938cd-bd0a-42d0-ac25-9406cc46fe95\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8" Sep 14 12:58:58.928281 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.928250 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-6kx76\" (UID: \"f8af22e7-1c34-404e-bc73-803803349989\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76" Sep 14 12:58:58.928329 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:58.928311 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 14 12:58:58.928365 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:58.928341 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 14 12:58:58.928394 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:58.928370 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls podName:547938cd-bd0a-42d0-ac25-9406cc46fe95 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:59.928354451 +0000 UTC m=+40.591250063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-b9nh8" (UID: "547938cd-bd0a-42d0-ac25-9406cc46fe95") : secret "cluster-monitoring-operator-tls" not found Sep 14 12:58:58.928394 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:58.928386 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls podName:f8af22e7-1c34-404e-bc73-803803349989 nodeName:}" failed. No retries permitted until 2026-09-14 12:58:59.928379278 +0000 UTC m=+40.591274886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-6kx76" (UID: "f8af22e7-1c34-404e-bc73-803803349989") : secret "samples-operator-tls" not found Sep 14 12:58:58.963242 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:58.963217 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-2995b"] Sep 14 12:58:58.966211 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:58:58.966183 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc65821_255d_43be_9ccc_7dffc5a49791.slice/crio-1aed35234aa5fbd9e14ca8134ebed62f070b4fdd3377bc8cbda0cdb00de2c035 WatchSource:0}: Error finding container 1aed35234aa5fbd9e14ca8134ebed62f070b4fdd3377bc8cbda0cdb00de2c035: Status 404 returned error can't find the container with id 1aed35234aa5fbd9e14ca8134ebed62f070b4fdd3377bc8cbda0cdb00de2c035 Sep 14 12:58:59.016964 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:59.016939 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-2995b" event={"ID":"fdc65821-255d-43be-9ccc-7dffc5a49791","Type":"ContainerStarted","Data":"1aed35234aa5fbd9e14ca8134ebed62f070b4fdd3377bc8cbda0cdb00de2c035"} Sep 14 12:58:59.018024 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:59.018006 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l78kn" event={"ID":"3be5d339-0117-4588-a03e-8562bc978355","Type":"ContainerStarted","Data":"bfcadd6f6cc33f806dfe290098c2625b2d61b610a2b5fb0f3f301f0d9dc674f5"} Sep 14 12:58:59.019136 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:59.019114 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2vff4" event={"ID":"544ba1bf-58c6-428c-9830-a456e4940b63","Type":"ContainerStarted","Data":"0c16568df7264b5fb49ed4a9f24b578749e2bced21b5c782ad9b21a750270af1"} Sep 14 12:58:59.020201 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:59.020180 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-577p2" event={"ID":"97519ad0-2873-4913-90da-1ba4dcfa7d9f","Type":"ContainerStarted","Data":"07005a404f877597669e41e2f1f4f93edb9d44527770b73d99dbc4af4d7dafb3"} Sep 14 12:58:59.021279 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:59.021261 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-2xl57" event={"ID":"ad5768be-fcc1-4c56-a485-2ae3f04504bc","Type":"ContainerStarted","Data":"f3b37b3d9d893a75a1c7f6da13790bb87761384111772a379a91a41efb29eed2"} Sep 14 12:58:59.022322 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:59.022304 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-wx768" event={"ID":"8d33c9fb-e4e0-468c-b81d-a3955aec6267","Type":"ContainerStarted","Data":"af9194903a444e389fdd14aef6c534d5b7e587394c6013e23af53c6a67732600"} Sep 14 12:58:59.029120 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:59.029097 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:58:59.029196 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:59.029147 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:58:59.029275 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:59.029265 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle podName:0287d99a-ca2c-45bc-8f32-a2a273401e51 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:00.029241959 +0000 UTC m=+40.692137580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle") pod "router-default-64988bbc7d-tnjdw" (UID: "0287d99a-ca2c-45bc-8f32-a2a273401e51") : configmap references non-existent config key: service-ca.crt Sep 14 12:58:59.029324 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:59.029271 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 12:58:59.029324 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:59.029287 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f5cb7776d-tzhs8: secret "image-registry-tls" not found Sep 14 12:58:59.029324 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:59.029310 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nn8d7\" (UID: \"1d79bb2f-3305-4485-a140-a2f5c5e1b009\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nn8d7" Sep 14 12:58:59.029411 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:59.029330 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls podName:650e0502-50be-4f10-b3b3-7cf515fdb02d nodeName:}" failed. No retries permitted until 2026-09-14 12:59:00.029317398 +0000 UTC m=+40.692213019 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls") pod "image-registry-6f5cb7776d-tzhs8" (UID: "650e0502-50be-4f10-b3b3-7cf515fdb02d") : secret "image-registry-tls" not found Sep 14 12:58:59.029411 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:59.029396 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 14 12:58:59.029474 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:59.029412 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:58:59.029474 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:59.029428 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert podName:1d79bb2f-3305-4485-a140-a2f5c5e1b009 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:00.029418205 +0000 UTC m=+40.692313832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nn8d7" (UID: "1d79bb2f-3305-4485-a140-a2f5c5e1b009") : secret "networking-console-plugin-cert" not found Sep 14 12:58:59.029538 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:59.029509 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 14 12:58:59.029567 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:59.029556 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs podName:0287d99a-ca2c-45bc-8f32-a2a273401e51 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:00.02954061 +0000 UTC m=+40.692436217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs") pod "router-default-64988bbc7d-tnjdw" (UID: "0287d99a-ca2c-45bc-8f32-a2a273401e51") : secret "router-metrics-certs-default" not found Sep 14 12:58:59.938555 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:59.937107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-6kx76\" (UID: \"f8af22e7-1c34-404e-bc73-803803349989\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76" Sep 14 12:58:59.938555 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:58:59.937441 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-b9nh8\" (UID: \"547938cd-bd0a-42d0-ac25-9406cc46fe95\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8" Sep 14 12:58:59.938555 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:59.937594 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 14 12:58:59.938555 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:59.937657 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls podName:547938cd-bd0a-42d0-ac25-9406cc46fe95 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:01.937637721 +0000 UTC m=+42.600533344 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-b9nh8" (UID: "547938cd-bd0a-42d0-ac25-9406cc46fe95") : secret "cluster-monitoring-operator-tls" not found Sep 14 12:58:59.938555 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:59.938101 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 14 12:58:59.938555 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:58:59.938151 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls podName:f8af22e7-1c34-404e-bc73-803803349989 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:01.938134596 +0000 UTC m=+42.601030220 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-6kx76" (UID: "f8af22e7-1c34-404e-bc73-803803349989") : secret "samples-operator-tls" not found Sep 14 12:59:00.039707 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:00.038658 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:59:00.039707 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:00.038747 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:59:00.039707 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:00.038800 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:59:00.039707 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:00.038852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nn8d7\" (UID: \"1d79bb2f-3305-4485-a140-a2f5c5e1b009\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nn8d7" Sep 14 12:59:00.039707 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:00.039039 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 14 12:59:00.039707 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:00.039103 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert podName:1d79bb2f-3305-4485-a140-a2f5c5e1b009 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:02.039084756 +0000 UTC m=+42.701980368 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nn8d7" (UID: "1d79bb2f-3305-4485-a140-a2f5c5e1b009") : secret "networking-console-plugin-cert" not found Sep 14 12:59:00.039707 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:00.039168 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 14 12:59:00.039707 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:00.039201 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs podName:0287d99a-ca2c-45bc-8f32-a2a273401e51 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:02.039188787 +0000 UTC m=+42.702084415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs") pod "router-default-64988bbc7d-tnjdw" (UID: "0287d99a-ca2c-45bc-8f32-a2a273401e51") : secret "router-metrics-certs-default" not found Sep 14 12:59:00.039707 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:00.039265 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 12:59:00.039707 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:00.039277 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f5cb7776d-tzhs8: secret "image-registry-tls" not found Sep 14 12:59:00.039707 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:00.039306 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls podName:650e0502-50be-4f10-b3b3-7cf515fdb02d nodeName:}" failed. No retries permitted until 2026-09-14 12:59:02.0392966 +0000 UTC m=+42.702192221 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls") pod "image-registry-6f5cb7776d-tzhs8" (UID: "650e0502-50be-4f10-b3b3-7cf515fdb02d") : secret "image-registry-tls" not found Sep 14 12:59:00.039707 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:00.039373 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle podName:0287d99a-ca2c-45bc-8f32-a2a273401e51 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:02.039363486 +0000 UTC m=+42.702259111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle") pod "router-default-64988bbc7d-tnjdw" (UID: "0287d99a-ca2c-45bc-8f32-a2a273401e51") : configmap references non-existent config key: service-ca.crt Sep 14 12:59:01.454836 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:01.454801 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls\") pod \"dns-default-qr6v6\" (UID: \"dbe829bc-1888-48fe-8416-3ae70353c8d7\") " pod="openshift-dns/dns-default-qr6v6" Sep 14 12:59:01.455235 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:01.454999 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 12:59:01.455235 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:01.455073 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls podName:dbe829bc-1888-48fe-8416-3ae70353c8d7 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:09.455053199 +0000 UTC m=+50.117948812 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls") pod "dns-default-qr6v6" (UID: "dbe829bc-1888-48fe-8416-3ae70353c8d7") : secret "dns-default-metrics-tls" not found Sep 14 12:59:01.960034 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:01.959994 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-6kx76\" (UID: \"f8af22e7-1c34-404e-bc73-803803349989\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76" Sep 14 12:59:01.960201 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:01.960110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-b9nh8\" (UID: \"547938cd-bd0a-42d0-ac25-9406cc46fe95\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8" Sep 14 12:59:01.960201 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:01.960155 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 14 12:59:01.960266 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:01.960222 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls podName:f8af22e7-1c34-404e-bc73-803803349989 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:05.960208332 +0000 UTC m=+46.623103953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-6kx76" (UID: "f8af22e7-1c34-404e-bc73-803803349989") : secret "samples-operator-tls" not found Sep 14 12:59:01.960266 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:01.960255 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 14 12:59:01.960332 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:01.960313 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls podName:547938cd-bd0a-42d0-ac25-9406cc46fe95 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:05.960298986 +0000 UTC m=+46.623194611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-b9nh8" (UID: "547938cd-bd0a-42d0-ac25-9406cc46fe95") : secret "cluster-monitoring-operator-tls" not found Sep 14 12:59:02.061147 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:02.061117 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:59:02.061294 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:02.061212 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:59:02.061294 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:02.061251 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nn8d7\" (UID: \"1d79bb2f-3305-4485-a140-a2f5c5e1b009\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nn8d7" Sep 14 12:59:02.061294 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:02.061270 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 12:59:02.061294 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:02.061286 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f5cb7776d-tzhs8: secret "image-registry-tls" not found Sep 14 12:59:02.061446 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:02.061330 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls podName:650e0502-50be-4f10-b3b3-7cf515fdb02d nodeName:}" failed. No retries permitted until 2026-09-14 12:59:06.061316492 +0000 UTC m=+46.724212099 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls") pod "image-registry-6f5cb7776d-tzhs8" (UID: "650e0502-50be-4f10-b3b3-7cf515fdb02d") : secret "image-registry-tls" not found Sep 14 12:59:02.061446 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:02.061347 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:59:02.061446 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:02.061383 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 14 12:59:02.061446 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:02.061432 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle podName:0287d99a-ca2c-45bc-8f32-a2a273401e51 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:06.061420962 +0000 UTC m=+46.724316570 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle") pod "router-default-64988bbc7d-tnjdw" (UID: "0287d99a-ca2c-45bc-8f32-a2a273401e51") : configmap references non-existent config key: service-ca.crt Sep 14 12:59:02.061574 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:02.061449 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert podName:1d79bb2f-3305-4485-a140-a2f5c5e1b009 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:06.061439708 +0000 UTC m=+46.724335317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nn8d7" (UID: "1d79bb2f-3305-4485-a140-a2f5c5e1b009") : secret "networking-console-plugin-cert" not found Sep 14 12:59:02.061574 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:02.061472 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 14 12:59:02.061574 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:02.061510 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs podName:0287d99a-ca2c-45bc-8f32-a2a273401e51 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:06.061501946 +0000 UTC m=+46.724397566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs") pod "router-default-64988bbc7d-tnjdw" (UID: "0287d99a-ca2c-45bc-8f32-a2a273401e51") : secret "router-metrics-certs-default" not found Sep 14 12:59:05.996346 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:05.996303 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-6kx76\" (UID: \"f8af22e7-1c34-404e-bc73-803803349989\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76" Sep 14 12:59:05.996832 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:05.996429 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-b9nh8\" (UID: \"547938cd-bd0a-42d0-ac25-9406cc46fe95\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8" Sep 14 12:59:05.996832 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:05.996488 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 14 12:59:05.996832 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:05.996555 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 14 12:59:05.996832 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:05.996571 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls podName:f8af22e7-1c34-404e-bc73-803803349989 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:13.996550208 +0000 UTC m=+54.659445822 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-6kx76" (UID: "f8af22e7-1c34-404e-bc73-803803349989") : secret "samples-operator-tls" not found Sep 14 12:59:05.996832 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:05.996608 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls podName:547938cd-bd0a-42d0-ac25-9406cc46fe95 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:13.996592831 +0000 UTC m=+54.659488459 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-b9nh8" (UID: "547938cd-bd0a-42d0-ac25-9406cc46fe95") : secret "cluster-monitoring-operator-tls" not found Sep 14 12:59:06.047676 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:06.047592 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/0.log" Sep 14 12:59:06.047676 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:06.047641 2564 generic.go:356] "Generic (PLEG): container finished" podID="fdc65821-255d-43be-9ccc-7dffc5a49791" containerID="ef2c063cd62f42511f1b52afd221b8aa6fddef786e0e81abf3364925945aaa79" exitCode=255 Sep 14 12:59:06.047884 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:06.047747 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-2995b" event={"ID":"fdc65821-255d-43be-9ccc-7dffc5a49791","Type":"ContainerDied","Data":"ef2c063cd62f42511f1b52afd221b8aa6fddef786e0e81abf3364925945aaa79"} Sep 14 12:59:06.048174 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:06.048122 2564 scope.go:117] "RemoveContainer" containerID="ef2c063cd62f42511f1b52afd221b8aa6fddef786e0e81abf3364925945aaa79" Sep 14 12:59:06.049799 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:06.049368 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l78kn" event={"ID":"3be5d339-0117-4588-a03e-8562bc978355","Type":"ContainerStarted","Data":"ef2b2e547c65c663cf3e925ccef1b6f51a994726250779e7701aae45e0d39d10"} Sep 14 12:59:06.052147 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:06.052110 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2vff4" event={"ID":"544ba1bf-58c6-428c-9830-a456e4940b63","Type":"ContainerStarted","Data":"52fcbeca0ed5db1cf730868619bbffd60066ffff1ab2235425207b4a36690ac1"} Sep 14 12:59:06.053971 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:06.053902 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-577p2" event={"ID":"97519ad0-2873-4913-90da-1ba4dcfa7d9f","Type":"ContainerStarted","Data":"5f32704b71b31626bcaad06e7aed7b10b3b818887fe92a052c2e84a5764ecb27"} Sep 14 12:59:06.055769 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:06.055747 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-2xl57" event={"ID":"ad5768be-fcc1-4c56-a485-2ae3f04504bc","Type":"ContainerStarted","Data":"da5b6818f60a8701739cfcc7ae7a4a9a08935923d068e0df50bf7bbc966d8730"} Sep 14 12:59:06.057539 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:06.057518 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-wx768" event={"ID":"8d33c9fb-e4e0-468c-b81d-a3955aec6267","Type":"ContainerStarted","Data":"ad48e8ecc8906f4a09c35a2ba6145b8d6a51eabefb68ea79e87eec806df47cb2"} Sep 14 12:59:06.082393 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:06.082339 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l78kn" podStartSLOduration=1.4248358429999999 podStartE2EDuration="8.082319623s" totalImagesPullingTime="6.65748378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:06.08141425 +0000 UTC m=+46.744309880" watchObservedRunningTime="2026-09-14 12:59:06.082319623 +0000 UTC m=+46.745215255" Sep 14 12:59:06.097280 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:06.097252 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:59:06.097394 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:06.097330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:59:06.097394 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:06.097378 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:59:06.097503 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:06.097424 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nn8d7\" (UID: \"1d79bb2f-3305-4485-a140-a2f5c5e1b009\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nn8d7" Sep 14 12:59:06.097563 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:06.097546 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 14 12:59:06.099207 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:06.099112 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-wx768" podStartSLOduration=1.466796 podStartE2EDuration="8.099094139s" totalImagesPullingTime="6.632298139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:06.098208129 +0000 UTC m=+46.761103765" watchObservedRunningTime="2026-09-14 12:59:06.099094139 +0000 UTC m=+46.761989766" Sep 14 12:59:06.100228 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:06.097768 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 12:59:06.100305 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:06.100237 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f5cb7776d-tzhs8: secret "image-registry-tls" not found Sep 14 12:59:06.100305 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:06.097865 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 14 12:59:06.100305 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:06.098020 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle podName:0287d99a-ca2c-45bc-8f32-a2a273401e51 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:14.098001877 +0000 UTC m=+54.760897501 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle") pod "router-default-64988bbc7d-tnjdw" (UID: "0287d99a-ca2c-45bc-8f32-a2a273401e51") : configmap references non-existent config key: service-ca.crt Sep 14 12:59:06.100600 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:06.100314 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert podName:1d79bb2f-3305-4485-a140-a2f5c5e1b009 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:14.100298808 +0000 UTC m=+54.763194432 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nn8d7" (UID: "1d79bb2f-3305-4485-a140-a2f5c5e1b009") : secret "networking-console-plugin-cert" not found Sep 14 12:59:06.100600 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:06.100328 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls podName:650e0502-50be-4f10-b3b3-7cf515fdb02d nodeName:}" failed. No retries permitted until 2026-09-14 12:59:14.10031927 +0000 UTC m=+54.763214884 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls") pod "image-registry-6f5cb7776d-tzhs8" (UID: "650e0502-50be-4f10-b3b3-7cf515fdb02d") : secret "image-registry-tls" not found Sep 14 12:59:06.100600 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:06.100343 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs podName:0287d99a-ca2c-45bc-8f32-a2a273401e51 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:14.100332917 +0000 UTC m=+54.763228531 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs") pod "router-default-64988bbc7d-tnjdw" (UID: "0287d99a-ca2c-45bc-8f32-a2a273401e51") : secret "router-metrics-certs-default" not found Sep 14 12:59:06.114947 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:06.114862 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-577p2" podStartSLOduration=1.392707188 podStartE2EDuration="8.114842463s" totalImagesPullingTime="6.722135275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:06.113768484 +0000 UTC m=+46.776664115" watchObservedRunningTime="2026-09-14 12:59:06.114842463 +0000 UTC m=+46.777738094" Sep 14 12:59:06.131761 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:06.131709 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2vff4" podStartSLOduration=1.444482463 podStartE2EDuration="8.131693945s" totalImagesPullingTime="6.687211482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:06.131553259 +0000 UTC m=+46.794448889" watchObservedRunningTime="2026-09-14 12:59:06.131693945 +0000 UTC m=+46.794589576" Sep 14 12:59:06.151830 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:06.151783 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-2xl57" podStartSLOduration=1.562651814 podStartE2EDuration="8.15176272s" totalImagesPullingTime="6.589110906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:06.149178156 +0000 UTC m=+46.812073767" watchObservedRunningTime="2026-09-14 12:59:06.15176272 +0000 UTC m=+46.814658351" Sep 14 12:59:07.064333 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.064305 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/1.log" Sep 14 12:59:07.064821 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.064803 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/0.log" Sep 14 12:59:07.064876 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.064845 2564 generic.go:356] "Generic (PLEG): container finished" podID="fdc65821-255d-43be-9ccc-7dffc5a49791" containerID="ef6c97a225509583277895134c7e78c899d5413eabde259cb34aedf318a55d1e" exitCode=255 Sep 14 12:59:07.064997 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.064972 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-2995b" event={"ID":"fdc65821-255d-43be-9ccc-7dffc5a49791","Type":"ContainerDied","Data":"ef6c97a225509583277895134c7e78c899d5413eabde259cb34aedf318a55d1e"} Sep 14 12:59:07.065098 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.065080 2564 scope.go:117] "RemoveContainer" containerID="ef2c063cd62f42511f1b52afd221b8aa6fddef786e0e81abf3364925945aaa79" Sep 14 12:59:07.065312 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.065296 2564 scope.go:117] "RemoveContainer" containerID="ef6c97a225509583277895134c7e78c899d5413eabde259cb34aedf318a55d1e" Sep 14 12:59:07.065495 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:07.065473 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-2995b_openshift-console-operator(fdc65821-255d-43be-9ccc-7dffc5a49791)\"" pod="openshift-console-operator/console-operator-696884fb9c-2995b" podUID="fdc65821-255d-43be-9ccc-7dffc5a49791" Sep 14 12:59:07.637644 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.637612 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2d9bv"] Sep 14 12:59:07.668424 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.668397 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2d9bv"] Sep 14 12:59:07.668564 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.668522 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:07.670984 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.670963 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 14 12:59:07.671208 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.671193 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 14 12:59:07.671296 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.671280 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-26vf9\"" Sep 14 12:59:07.819187 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.819160 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e80a9489-e6d6-4afb-b04e-970dc9741e6e-data-volume\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:07.819366 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.819192 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:07.819366 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.819235 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpncj\" (UniqueName: \"kubernetes.io/projected/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-api-access-fpncj\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:07.819366 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.819286 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e80a9489-e6d6-4afb-b04e-970dc9741e6e-crio-socket\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:07.819477 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.819369 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:07.919857 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.919770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:07.920042 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.919863 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e80a9489-e6d6-4afb-b04e-970dc9741e6e-data-volume\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:07.920042 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.919893 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:07.920042 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:07.919917 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 12:59:07.920042 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.920000 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpncj\" (UniqueName: \"kubernetes.io/projected/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-api-access-fpncj\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:07.920042 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:07.920025 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 12:59:07.920301 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:07.920076 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-rbac-proxy-cm podName:e80a9489-e6d6-4afb-b04e-970dc9741e6e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:08.420061311 +0000 UTC m=+49.082956953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2d9bv" (UID: "e80a9489-e6d6-4afb-b04e-970dc9741e6e") : configmap "kube-rbac-proxy" not found Sep 14 12:59:07.920301 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:07.920123 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls podName:e80a9489-e6d6-4afb-b04e-970dc9741e6e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:08.42010404 +0000 UTC m=+49.082999650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2d9bv" (UID: "e80a9489-e6d6-4afb-b04e-970dc9741e6e") : secret "insights-runtime-extractor-tls" not found Sep 14 12:59:07.920301 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.920167 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e80a9489-e6d6-4afb-b04e-970dc9741e6e-crio-socket\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:07.920416 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.920349 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e80a9489-e6d6-4afb-b04e-970dc9741e6e-data-volume\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:07.920416 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.920376 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e80a9489-e6d6-4afb-b04e-970dc9741e6e-crio-socket\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:07.931817 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:07.931794 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpncj\" (UniqueName: \"kubernetes.io/projected/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-api-access-fpncj\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:08.069157 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:08.069128 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/1.log" Sep 14 12:59:08.069519 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:08.069463 2564 scope.go:117] "RemoveContainer" containerID="ef6c97a225509583277895134c7e78c899d5413eabde259cb34aedf318a55d1e" Sep 14 12:59:08.069639 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:08.069621 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-2995b_openshift-console-operator(fdc65821-255d-43be-9ccc-7dffc5a49791)\"" pod="openshift-console-operator/console-operator-696884fb9c-2995b" podUID="fdc65821-255d-43be-9ccc-7dffc5a49791" Sep 14 12:59:08.425123 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:08.425088 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:08.425296 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:08.425146 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:08.425296 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:08.425241 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 12:59:08.425370 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:08.425296 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 12:59:08.425370 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:08.425314 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls podName:e80a9489-e6d6-4afb-b04e-970dc9741e6e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:09.425296929 +0000 UTC m=+50.088192551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2d9bv" (UID: "e80a9489-e6d6-4afb-b04e-970dc9741e6e") : secret "insights-runtime-extractor-tls" not found Sep 14 12:59:08.425370 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:08.425346 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-rbac-proxy-cm podName:e80a9489-e6d6-4afb-b04e-970dc9741e6e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:09.425334108 +0000 UTC m=+50.088229747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2d9bv" (UID: "e80a9489-e6d6-4afb-b04e-970dc9741e6e") : configmap "kube-rbac-proxy" not found Sep 14 12:59:08.675588 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:08.675509 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-2995b" Sep 14 12:59:08.675588 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:08.675570 2564 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-2995b" Sep 14 12:59:09.073189 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:09.073107 2564 scope.go:117] "RemoveContainer" containerID="ef6c97a225509583277895134c7e78c899d5413eabde259cb34aedf318a55d1e" Sep 14 12:59:09.073521 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:09.073273 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-2995b_openshift-console-operator(fdc65821-255d-43be-9ccc-7dffc5a49791)\"" pod="openshift-console-operator/console-operator-696884fb9c-2995b" podUID="fdc65821-255d-43be-9ccc-7dffc5a49791" Sep 14 12:59:09.434108 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:09.434062 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:09.434285 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:09.434132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:09.434285 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:09.434209 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 12:59:09.434285 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:09.434219 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 12:59:09.434285 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:09.434278 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-rbac-proxy-cm podName:e80a9489-e6d6-4afb-b04e-970dc9741e6e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:11.434260942 +0000 UTC m=+52.097156550 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2d9bv" (UID: "e80a9489-e6d6-4afb-b04e-970dc9741e6e") : configmap "kube-rbac-proxy" not found Sep 14 12:59:09.434418 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:09.434295 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls podName:e80a9489-e6d6-4afb-b04e-970dc9741e6e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:11.43428667 +0000 UTC m=+52.097182278 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2d9bv" (UID: "e80a9489-e6d6-4afb-b04e-970dc9741e6e") : secret "insights-runtime-extractor-tls" not found Sep 14 12:59:09.538547 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:09.535466 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls\") pod \"dns-default-qr6v6\" (UID: \"dbe829bc-1888-48fe-8416-3ae70353c8d7\") " pod="openshift-dns/dns-default-qr6v6" Sep 14 12:59:09.538547 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:09.535569 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 14 12:59:09.538547 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:09.535647 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls podName:dbe829bc-1888-48fe-8416-3ae70353c8d7 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:25.53562137 +0000 UTC m=+66.198516992 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls") pod "dns-default-qr6v6" (UID: "dbe829bc-1888-48fe-8416-3ae70353c8d7") : secret "dns-default-metrics-tls" not found Sep 14 12:59:10.201271 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:10.201246 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g2k4m_0d6daf95-87b4-4f08-8058-6ad1d9ea9dec/dns-node-resolver/0.log" Sep 14 12:59:11.196636 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:11.196613 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4v75b_570dacc2-2bf1-4b1a-8f03-3abb696a2ecf/node-ca/0.log" Sep 14 12:59:11.451289 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:11.451211 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:11.451649 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:11.451341 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 12:59:11.451649 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:11.451401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:11.451649 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:11.451422 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-rbac-proxy-cm podName:e80a9489-e6d6-4afb-b04e-970dc9741e6e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:15.451402691 +0000 UTC m=+56.114298317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2d9bv" (UID: "e80a9489-e6d6-4afb-b04e-970dc9741e6e") : configmap "kube-rbac-proxy" not found Sep 14 12:59:11.451649 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:11.451487 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 12:59:11.451649 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:11.451533 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls podName:e80a9489-e6d6-4afb-b04e-970dc9741e6e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:15.451523332 +0000 UTC m=+56.114418953 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2d9bv" (UID: "e80a9489-e6d6-4afb-b04e-970dc9741e6e") : secret "insights-runtime-extractor-tls" not found Sep 14 12:59:11.996895 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:11.996862 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dskx8_fc512c34-085d-4934-9c57-95f05e3db013/serve-healthcheck-canary/0.log" Sep 14 12:59:12.998092 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:12.998050 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-2vff4_544ba1bf-58c6-428c-9830-a456e4940b63/kube-storage-version-migrator-operator/0.log" Sep 14 12:59:14.074601 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:14.074562 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-b9nh8\" (UID: \"547938cd-bd0a-42d0-ac25-9406cc46fe95\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8" Sep 14 12:59:14.075046 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:14.074720 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 14 12:59:14.075046 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:14.074719 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-6kx76\" (UID: \"f8af22e7-1c34-404e-bc73-803803349989\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76" Sep 14 12:59:14.075046 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:14.074788 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls podName:547938cd-bd0a-42d0-ac25-9406cc46fe95 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:30.074769226 +0000 UTC m=+70.737664833 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-b9nh8" (UID: "547938cd-bd0a-42d0-ac25-9406cc46fe95") : secret "cluster-monitoring-operator-tls" not found Sep 14 12:59:14.075046 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:14.074805 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 14 12:59:14.075046 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:14.074881 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls podName:f8af22e7-1c34-404e-bc73-803803349989 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:30.074863013 +0000 UTC m=+70.737758629 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-6kx76" (UID: "f8af22e7-1c34-404e-bc73-803803349989") : secret "samples-operator-tls" not found Sep 14 12:59:14.175716 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:14.175686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:59:14.175886 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:14.175730 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:59:14.175886 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:14.175846 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle podName:0287d99a-ca2c-45bc-8f32-a2a273401e51 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:30.175832735 +0000 UTC m=+70.838728356 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle") pod "router-default-64988bbc7d-tnjdw" (UID: "0287d99a-ca2c-45bc-8f32-a2a273401e51") : configmap references non-existent config key: service-ca.crt Sep 14 12:59:14.175886 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:14.175844 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 14 12:59:14.175886 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:14.175862 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nn8d7\" (UID: \"1d79bb2f-3305-4485-a140-a2f5c5e1b009\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nn8d7" Sep 14 12:59:14.175886 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:14.175868 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f5cb7776d-tzhs8: secret "image-registry-tls" not found Sep 14 12:59:14.176215 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:14.175911 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls podName:650e0502-50be-4f10-b3b3-7cf515fdb02d nodeName:}" failed. No retries permitted until 2026-09-14 12:59:30.175894946 +0000 UTC m=+70.838790553 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls") pod "image-registry-6f5cb7776d-tzhs8" (UID: "650e0502-50be-4f10-b3b3-7cf515fdb02d") : secret "image-registry-tls" not found Sep 14 12:59:14.176215 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:14.176007 2564 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 14 12:59:14.176215 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:14.176015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:59:14.176215 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:14.176045 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert podName:1d79bb2f-3305-4485-a140-a2f5c5e1b009 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:30.176032053 +0000 UTC m=+70.838927660 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nn8d7" (UID: "1d79bb2f-3305-4485-a140-a2f5c5e1b009") : secret "networking-console-plugin-cert" not found Sep 14 12:59:14.176215 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:14.176157 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 14 12:59:14.176215 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:14.176219 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs podName:0287d99a-ca2c-45bc-8f32-a2a273401e51 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:30.176199522 +0000 UTC m=+70.839095130 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs") pod "router-default-64988bbc7d-tnjdw" (UID: "0287d99a-ca2c-45bc-8f32-a2a273401e51") : secret "router-metrics-certs-default" not found Sep 14 12:59:15.487283 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:15.487233 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:15.487758 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:15.487359 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 14 12:59:15.487758 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:15.487428 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-rbac-proxy-cm podName:e80a9489-e6d6-4afb-b04e-970dc9741e6e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:23.487409405 +0000 UTC m=+64.150305041 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2d9bv" (UID: "e80a9489-e6d6-4afb-b04e-970dc9741e6e") : configmap "kube-rbac-proxy" not found Sep 14 12:59:15.487758 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:15.487448 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:15.487758 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:15.487536 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 12:59:15.487758 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:15.487575 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls podName:e80a9489-e6d6-4afb-b04e-970dc9741e6e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:23.487562372 +0000 UTC m=+64.150457981 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2d9bv" (UID: "e80a9489-e6d6-4afb-b04e-970dc9741e6e") : secret "insights-runtime-extractor-tls" not found Sep 14 12:59:19.985351 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:19.985324 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b62sc" Sep 14 12:59:22.805282 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:22.805249 2564 scope.go:117] "RemoveContainer" containerID="ef6c97a225509583277895134c7e78c899d5413eabde259cb34aedf318a55d1e" Sep 14 12:59:23.117214 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:23.117191 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/1.log" Sep 14 12:59:23.117377 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:23.117243 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-2995b" event={"ID":"fdc65821-255d-43be-9ccc-7dffc5a49791","Type":"ContainerStarted","Data":"05261a1eb5ec3731f3a9b6e98bd58e916eddb06c3983940259327e041aa2fc28"} Sep 14 12:59:23.117539 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:23.117520 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-2995b" Sep 14 12:59:23.134589 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:23.134544 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-2995b" podStartSLOduration=18.605297755 podStartE2EDuration="25.134530238s" totalImagesPullingTime="6.529232483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:23.133885031 +0000 UTC m=+63.796780675" watchObservedRunningTime="2026-09-14 12:59:23.134530238 +0000 UTC m=+63.797425868" Sep 14 12:59:23.507083 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:23.507004 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-2995b" Sep 14 12:59:23.556101 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:23.556054 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:23.556284 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:23.556133 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 14 12:59:23.556284 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:23.556149 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:23.556284 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:23.556199 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls podName:e80a9489-e6d6-4afb-b04e-970dc9741e6e nodeName:}" failed. No retries permitted until 2026-09-14 12:59:39.556179815 +0000 UTC m=+80.219075443 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2d9bv" (UID: "e80a9489-e6d6-4afb-b04e-970dc9741e6e") : secret "insights-runtime-extractor-tls" not found Sep 14 12:59:23.556645 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:23.556626 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e80a9489-e6d6-4afb-b04e-970dc9741e6e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:25.573273 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:25.573237 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls\") pod \"dns-default-qr6v6\" (UID: \"dbe829bc-1888-48fe-8416-3ae70353c8d7\") " pod="openshift-dns/dns-default-qr6v6" Sep 14 12:59:25.575581 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:25.575563 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbe829bc-1888-48fe-8416-3ae70353c8d7-metrics-tls\") pod \"dns-default-qr6v6\" (UID: \"dbe829bc-1888-48fe-8416-3ae70353c8d7\") " pod="openshift-dns/dns-default-qr6v6" Sep 14 12:59:25.674502 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:25.674466 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs\") pod \"network-metrics-daemon-kv576\" (UID: \"ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63\") " pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:59:25.674662 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:25.674520 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret\") pod \"global-pull-secret-syncer-72fhn\" (UID: \"d49b2943-be96-4d9a-ab55-965845ba03f9\") " pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:59:25.677220 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:25.677197 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 14 12:59:25.677220 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:25.677207 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 14 12:59:25.687572 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:25.687549 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63-metrics-certs\") pod \"network-metrics-daemon-kv576\" (UID: \"ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63\") " pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:59:25.687667 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:25.687553 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d49b2943-be96-4d9a-ab55-965845ba03f9-original-pull-secret\") pod \"global-pull-secret-syncer-72fhn\" (UID: \"d49b2943-be96-4d9a-ab55-965845ba03f9\") " pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:59:25.775080 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:25.775040 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v2x2\" (UniqueName: \"kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2\") pod \"network-check-target-7t274\" (UID: \"746cbe95-b362-49b9-a280-e4c1faf73d32\") " pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:59:25.777595 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:25.777571 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v2x2\" (UniqueName: \"kubernetes.io/projected/746cbe95-b362-49b9-a280-e4c1faf73d32-kube-api-access-6v2x2\") pod \"network-check-target-7t274\" (UID: \"746cbe95-b362-49b9-a280-e4c1faf73d32\") " pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:59:25.777790 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:25.777774 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xjstd\"" Sep 14 12:59:25.786191 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:25.786174 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qr6v6" Sep 14 12:59:25.919145 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:25.919113 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-72fhn" Sep 14 12:59:25.926638 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:25.926617 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9l5s5\"" Sep 14 12:59:25.934805 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:25.934791 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kv576" Sep 14 12:59:26.019096 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:26.019073 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tlghq\"" Sep 14 12:59:26.026589 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:26.026572 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:59:26.096598 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:26.096528 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qr6v6"] Sep 14 12:59:26.100842 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:59:26.100794 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe829bc_1888_48fe_8416_3ae70353c8d7.slice/crio-57e9da28f6cc9136481e23a4c25a19b65f6a0f87b578507e3251482825282323 WatchSource:0}: Error finding container 57e9da28f6cc9136481e23a4c25a19b65f6a0f87b578507e3251482825282323: Status 404 returned error can't find the container with id 57e9da28f6cc9136481e23a4c25a19b65f6a0f87b578507e3251482825282323 Sep 14 12:59:26.126799 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:26.126771 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qr6v6" event={"ID":"dbe829bc-1888-48fe-8416-3ae70353c8d7","Type":"ContainerStarted","Data":"57e9da28f6cc9136481e23a4c25a19b65f6a0f87b578507e3251482825282323"} Sep 14 12:59:26.202554 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:26.202524 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-72fhn"] Sep 14 12:59:26.205522 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:59:26.205500 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49b2943_be96_4d9a_ab55_965845ba03f9.slice/crio-ae6ec581ffe432d1623d2e34577e385941239d3d907f92684bf2c34c1ff58b95 WatchSource:0}: Error finding container ae6ec581ffe432d1623d2e34577e385941239d3d907f92684bf2c34c1ff58b95: Status 404 returned error can't find the container with id ae6ec581ffe432d1623d2e34577e385941239d3d907f92684bf2c34c1ff58b95 Sep 14 12:59:26.236753 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:26.236726 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kv576"] Sep 14 12:59:26.239414 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:59:26.239387 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff51d3d8_9089_41b9_b6fe_13a3d7a8fb63.slice/crio-b3a7e4bde8bc789eb18647103e0539c2a6f6c93460ea030dcd8b085999eb354a WatchSource:0}: Error finding container b3a7e4bde8bc789eb18647103e0539c2a6f6c93460ea030dcd8b085999eb354a: Status 404 returned error can't find the container with id b3a7e4bde8bc789eb18647103e0539c2a6f6c93460ea030dcd8b085999eb354a Sep 14 12:59:26.321084 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:26.321055 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7t274"] Sep 14 12:59:26.324212 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:59:26.324189 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746cbe95_b362_49b9_a280_e4c1faf73d32.slice/crio-32bbdaa7684cb8eb5120e81c01fa3da12fadd7ac2189751e825d00443d149d17 WatchSource:0}: Error finding container 32bbdaa7684cb8eb5120e81c01fa3da12fadd7ac2189751e825d00443d149d17: Status 404 returned error can't find the container with id 32bbdaa7684cb8eb5120e81c01fa3da12fadd7ac2189751e825d00443d149d17 Sep 14 12:59:27.132189 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:27.132145 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kv576" event={"ID":"ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63","Type":"ContainerStarted","Data":"b3a7e4bde8bc789eb18647103e0539c2a6f6c93460ea030dcd8b085999eb354a"} Sep 14 12:59:27.133549 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:27.133519 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-72fhn" event={"ID":"d49b2943-be96-4d9a-ab55-965845ba03f9","Type":"ContainerStarted","Data":"ae6ec581ffe432d1623d2e34577e385941239d3d907f92684bf2c34c1ff58b95"} Sep 14 12:59:27.135358 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:27.135332 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7t274" event={"ID":"746cbe95-b362-49b9-a280-e4c1faf73d32","Type":"ContainerStarted","Data":"27bf70073255755945a1bf0e278155e1e1d67e187a75911c116ace34ac09824c"} Sep 14 12:59:27.135440 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:27.135381 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7t274" event={"ID":"746cbe95-b362-49b9-a280-e4c1faf73d32","Type":"ContainerStarted","Data":"32bbdaa7684cb8eb5120e81c01fa3da12fadd7ac2189751e825d00443d149d17"} Sep 14 12:59:27.135614 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:27.135598 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:59:27.152780 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:27.152727 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7t274" podStartSLOduration=68.152693461 podStartE2EDuration="1m8.152693461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:27.151184119 +0000 UTC m=+67.814079751" watchObservedRunningTime="2026-09-14 12:59:27.152693461 +0000 UTC m=+67.815589094" Sep 14 12:59:29.144526 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:29.144494 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kv576" event={"ID":"ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63","Type":"ContainerStarted","Data":"77f980b1bad55f5c68de8f16791f7c414d304a21319a0bb1b773a9c1c7511967"} Sep 14 12:59:29.145017 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:29.144558 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kv576" event={"ID":"ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63","Type":"ContainerStarted","Data":"44b24166c081bbc0739541ea043ea01d9b10b70675cfd5bbefd7583a8a6eeb43"} Sep 14 12:59:29.146792 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:29.146760 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qr6v6" event={"ID":"dbe829bc-1888-48fe-8416-3ae70353c8d7","Type":"ContainerStarted","Data":"2560149475f1349b4163e340a1cb13e43ee7af4e58772b069a923596e8d72e98"} Sep 14 12:59:29.146911 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:29.146815 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qr6v6" event={"ID":"dbe829bc-1888-48fe-8416-3ae70353c8d7","Type":"ContainerStarted","Data":"1a8a01bb982e63931104d6088b787d6b3dd726147142f5bd24d17b537cd79206"} Sep 14 12:59:29.146911 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:29.146844 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qr6v6" Sep 14 12:59:29.160986 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:29.160937 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kv576" podStartSLOduration=68.259359468 podStartE2EDuration="1m10.160905447s" totalImagesPullingTime="1.901545979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:29.160151695 +0000 UTC m=+69.823047325" watchObservedRunningTime="2026-09-14 12:59:29.160905447 +0000 UTC m=+69.823801076" Sep 14 12:59:29.177848 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:29.177803 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qr6v6" podStartSLOduration=34.136189664 podStartE2EDuration="36.177787516s" totalImagesPullingTime="2.041597852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:29.177102842 +0000 UTC m=+69.839998474" watchObservedRunningTime="2026-09-14 12:59:29.177787516 +0000 UTC m=+69.840683146" Sep 14 12:59:30.112696 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.112513 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-6kx76\" (UID: \"f8af22e7-1c34-404e-bc73-803803349989\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76" Sep 14 12:59:30.112696 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.112617 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-b9nh8\" (UID: \"547938cd-bd0a-42d0-ac25-9406cc46fe95\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8" Sep 14 12:59:30.115502 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.115471 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/547938cd-bd0a-42d0-ac25-9406cc46fe95-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-b9nh8\" (UID: \"547938cd-bd0a-42d0-ac25-9406cc46fe95\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8" Sep 14 12:59:30.115635 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.115510 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f8af22e7-1c34-404e-bc73-803803349989-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-6kx76\" (UID: \"f8af22e7-1c34-404e-bc73-803803349989\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76" Sep 14 12:59:30.213271 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.213220 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:59:30.213741 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.213295 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nn8d7\" (UID: \"1d79bb2f-3305-4485-a140-a2f5c5e1b009\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nn8d7" Sep 14 12:59:30.213741 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.213400 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:59:30.213741 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.213436 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:59:30.215245 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.215215 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0287d99a-ca2c-45bc-8f32-a2a273401e51-service-ca-bundle\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:59:30.216612 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.216587 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls\") pod \"image-registry-6f5cb7776d-tzhs8\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:59:30.216706 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.216592 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0287d99a-ca2c-45bc-8f32-a2a273401e51-metrics-certs\") pod \"router-default-64988bbc7d-tnjdw\" (UID: \"0287d99a-ca2c-45bc-8f32-a2a273401e51\") " pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:59:30.216706 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.216592 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d79bb2f-3305-4485-a140-a2f5c5e1b009-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nn8d7\" (UID: \"1d79bb2f-3305-4485-a140-a2f5c5e1b009\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nn8d7" Sep 14 12:59:30.286153 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.286111 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-f65mr\"" Sep 14 12:59:30.293644 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.293621 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76" Sep 14 12:59:30.359548 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.359518 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-7cmvz\"" Sep 14 12:59:30.366958 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.366914 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8" Sep 14 12:59:30.426412 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.426181 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8dx6q\"" Sep 14 12:59:30.434346 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.434318 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:59:30.446153 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.446135 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xm75b\"" Sep 14 12:59:30.454520 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.454493 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:59:30.463175 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.463149 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-d464v\"" Sep 14 12:59:30.471412 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.471383 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nn8d7" Sep 14 12:59:30.887233 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.887198 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8"] Sep 14 12:59:30.890623 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:59:30.890598 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod547938cd_bd0a_42d0_ac25_9406cc46fe95.slice/crio-6029c799d4592aa0be054c493c0f710147e90dc36d53e0b82f43ad71d621fcf6 WatchSource:0}: Error finding container 6029c799d4592aa0be054c493c0f710147e90dc36d53e0b82f43ad71d621fcf6: Status 404 returned error can't find the container with id 6029c799d4592aa0be054c493c0f710147e90dc36d53e0b82f43ad71d621fcf6 Sep 14 12:59:30.892988 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.892890 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f5cb7776d-tzhs8"] Sep 14 12:59:30.895767 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.895718 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76"] Sep 14 12:59:30.906587 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.906562 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-nn8d7"] Sep 14 12:59:30.911414 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:30.911391 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64988bbc7d-tnjdw"] Sep 14 12:59:31.157228 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:31.157144 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-72fhn" event={"ID":"d49b2943-be96-4d9a-ab55-965845ba03f9","Type":"ContainerStarted","Data":"4d9cad9108d66aa94b6d44e0b6637b7a496ca7ddb1000ba65ba81e712776f104"} Sep 14 12:59:31.158752 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:31.158725 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64988bbc7d-tnjdw" event={"ID":"0287d99a-ca2c-45bc-8f32-a2a273401e51","Type":"ContainerStarted","Data":"8f1ad0ea2a77af32cf2b3e63196e130ebdc3ba462825264ec5a8c2f452585711"} Sep 14 12:59:31.158861 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:31.158765 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64988bbc7d-tnjdw" event={"ID":"0287d99a-ca2c-45bc-8f32-a2a273401e51","Type":"ContainerStarted","Data":"c8822b3a1f8dc4bfc22be8cae96e598dc19324b13e4859e8d0f51119b44b591f"} Sep 14 12:59:31.160052 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:31.160028 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76" event={"ID":"f8af22e7-1c34-404e-bc73-803803349989","Type":"ContainerStarted","Data":"2cb1378f4be0c8816932dea0f6cd7287a3e358aace86f7efbc291bbba541171c"} Sep 14 12:59:31.161176 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:31.161154 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nn8d7" event={"ID":"1d79bb2f-3305-4485-a140-a2f5c5e1b009","Type":"ContainerStarted","Data":"2d47ce32625d6fecf67c8d1857f15e5000d7867e21a368f19c22e4127a034c82"} Sep 14 12:59:31.162606 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:31.162580 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" event={"ID":"650e0502-50be-4f10-b3b3-7cf515fdb02d","Type":"ContainerStarted","Data":"9f479b5474cb6c2f5cdfa9b140a1dbc0ae162457c683c89644fb5955802e3e1b"} Sep 14 12:59:31.162697 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:31.162626 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:59:31.162697 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:31.162641 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" event={"ID":"650e0502-50be-4f10-b3b3-7cf515fdb02d","Type":"ContainerStarted","Data":"a8988cbb9b2174480b60dd05f049ebe2a3e05c750bad85cb4d9573daa90cec14"} Sep 14 12:59:31.163845 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:31.163818 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8" event={"ID":"547938cd-bd0a-42d0-ac25-9406cc46fe95","Type":"ContainerStarted","Data":"6029c799d4592aa0be054c493c0f710147e90dc36d53e0b82f43ad71d621fcf6"} Sep 14 12:59:31.172984 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:31.172944 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-72fhn" podStartSLOduration=66.722926335 podStartE2EDuration="1m11.172910602s" totalImagesPullingTime="4.449984267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:31.172085669 +0000 UTC m=+71.834981299" watchObservedRunningTime="2026-09-14 12:59:31.172910602 +0000 UTC m=+71.835806223" Sep 14 12:59:31.190072 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:31.190032 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-64988bbc7d-tnjdw" podStartSLOduration=33.190017552 podStartE2EDuration="33.190017552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:31.189276092 +0000 UTC m=+71.852171723" watchObservedRunningTime="2026-09-14 12:59:31.190017552 +0000 UTC m=+71.852913224" Sep 14 12:59:31.208718 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:31.208670 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" podStartSLOduration=33.208654908 podStartE2EDuration="33.208654908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:31.207547148 +0000 UTC m=+71.870442778" watchObservedRunningTime="2026-09-14 12:59:31.208654908 +0000 UTC m=+71.871550532" Sep 14 12:59:31.455235 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:31.455148 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:59:31.458266 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:31.458237 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:59:32.170122 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:32.170090 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:59:32.171194 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:32.171170 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-64988bbc7d-tnjdw" Sep 14 12:59:32.353489 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:32.353450 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f5cb7776d-tzhs8"] Sep 14 12:59:34.178390 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:34.178358 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76" event={"ID":"f8af22e7-1c34-404e-bc73-803803349989","Type":"ContainerStarted","Data":"336b911efd5850a15aaa19a27718330381bbfff9d8c3fb687cecce454a1e2be8"} Sep 14 12:59:34.178787 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:34.178413 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76" event={"ID":"f8af22e7-1c34-404e-bc73-803803349989","Type":"ContainerStarted","Data":"ac4b7bd842209a7d27d77b9730eda52673a0086ed094933f17575cce4b0555c3"} Sep 14 12:59:34.179980 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:34.179948 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nn8d7" event={"ID":"1d79bb2f-3305-4485-a140-a2f5c5e1b009","Type":"ContainerStarted","Data":"67c92c91e67885988eeaa0ae7bf3ebf208510e95654d4e7df4030e8bfd4368dd"} Sep 14 12:59:34.181438 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:34.181417 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8" event={"ID":"547938cd-bd0a-42d0-ac25-9406cc46fe95","Type":"ContainerStarted","Data":"502bb88359fe137f051e2347f08595f562b1419ae6f28214e1c61173d6293a86"} Sep 14 12:59:34.197272 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:34.197230 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-6kx76" podStartSLOduration=33.887552811 podStartE2EDuration="36.19721589s" totalImagesPullingTime="2.309663079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:34.19559063 +0000 UTC m=+74.858486260" watchObservedRunningTime="2026-09-14 12:59:34.19721589 +0000 UTC m=+74.860111575" Sep 14 12:59:34.211174 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:34.211135 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-b9nh8" podStartSLOduration=33.842412418 podStartE2EDuration="36.211121018s" totalImagesPullingTime="2.3687086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:34.210573767 +0000 UTC m=+74.873469396" watchObservedRunningTime="2026-09-14 12:59:34.211121018 +0000 UTC m=+74.874016648" Sep 14 12:59:34.225724 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:34.225685 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nn8d7" podStartSLOduration=33.880772445 podStartE2EDuration="36.225672333s" totalImagesPullingTime="2.344899888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:58:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:34.22506048 +0000 UTC m=+74.887956112" watchObservedRunningTime="2026-09-14 12:59:34.225672333 +0000 UTC m=+74.888567963" Sep 14 12:59:39.153288 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:39.153259 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qr6v6" Sep 14 12:59:39.595342 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:39.595258 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:39.597685 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:39.597658 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e80a9489-e6d6-4afb-b04e-970dc9741e6e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2d9bv\" (UID: \"e80a9489-e6d6-4afb-b04e-970dc9741e6e\") " pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:39.780047 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:39.780015 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-26vf9\"" Sep 14 12:59:39.788242 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:39.788225 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2d9bv" Sep 14 12:59:40.060244 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:40.060143 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2d9bv"] Sep 14 12:59:40.062809 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:59:40.062787 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode80a9489_e6d6_4afb_b04e_970dc9741e6e.slice/crio-82aad6f8b5cbb3a4da31b173c32058af5f6784330808fac661c8e3e1baa980cc WatchSource:0}: Error finding container 82aad6f8b5cbb3a4da31b173c32058af5f6784330808fac661c8e3e1baa980cc: Status 404 returned error can't find the container with id 82aad6f8b5cbb3a4da31b173c32058af5f6784330808fac661c8e3e1baa980cc Sep 14 12:59:40.206819 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:40.206531 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2d9bv" event={"ID":"e80a9489-e6d6-4afb-b04e-970dc9741e6e","Type":"ContainerStarted","Data":"ce6880703d92fad6b6fce4da5f0a99b5f0d4b4269e802b71d1bc8623f5ee446b"} Sep 14 12:59:40.206819 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:40.206649 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2d9bv" event={"ID":"e80a9489-e6d6-4afb-b04e-970dc9741e6e","Type":"ContainerStarted","Data":"82aad6f8b5cbb3a4da31b173c32058af5f6784330808fac661c8e3e1baa980cc"} Sep 14 12:59:41.212978 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:41.212943 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2d9bv" event={"ID":"e80a9489-e6d6-4afb-b04e-970dc9741e6e","Type":"ContainerStarted","Data":"7890077241df2d54f7186ddaa4e2618cf9f00c0b0c73dfe202e1bd9a5ee8b876"} Sep 14 12:59:42.218359 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:42.218317 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2d9bv" event={"ID":"e80a9489-e6d6-4afb-b04e-970dc9741e6e","Type":"ContainerStarted","Data":"a21763520cef81be4382ac369c9a26025f5eac68407e6b1c7aad542ebaaab166"} Sep 14 12:59:42.239128 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:42.239078 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2d9bv" podStartSLOduration=33.369604619 podStartE2EDuration="35.239060674s" totalImagesPullingTime="1.869456055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 12:59:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:42.237311912 +0000 UTC m=+82.900207542" watchObservedRunningTime="2026-09-14 12:59:42.239060674 +0000 UTC m=+82.901956521" Sep 14 12:59:53.179291 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:53.179261 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:59:54.186445 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.186411 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mgbcd"] Sep 14 12:59:54.188909 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.188895 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.191628 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.191600 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 14 12:59:54.191776 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.191638 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 14 12:59:54.191776 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.191649 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 14 12:59:54.191776 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.191677 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5rv2c\"" Sep 14 12:59:54.191776 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.191604 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 14 12:59:54.306599 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.306570 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-tls\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.306599 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.306604 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b68906a-2c7d-44a3-837c-d62d9a661c79-metrics-client-ca\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.306835 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.306632 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-textfile\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.306835 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.306671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvcxp\" (UniqueName: \"kubernetes.io/projected/4b68906a-2c7d-44a3-837c-d62d9a661c79-kube-api-access-gvcxp\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.306835 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.306689 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b68906a-2c7d-44a3-837c-d62d9a661c79-root\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.306835 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.306714 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.306835 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.306733 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-accelerators-collector-config\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.306835 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.306791 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b68906a-2c7d-44a3-837c-d62d9a661c79-sys\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.306835 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.306816 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-wtmp\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.407473 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.407441 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-accelerators-collector-config\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.407648 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.407478 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b68906a-2c7d-44a3-837c-d62d9a661c79-sys\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.407648 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.407505 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-wtmp\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.407648 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.407533 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-tls\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.407648 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.407553 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b68906a-2c7d-44a3-837c-d62d9a661c79-metrics-client-ca\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.407648 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.407572 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b68906a-2c7d-44a3-837c-d62d9a661c79-sys\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.407648 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.407595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-textfile\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.407648 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.407640 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvcxp\" (UniqueName: \"kubernetes.io/projected/4b68906a-2c7d-44a3-837c-d62d9a661c79-kube-api-access-gvcxp\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.408028 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.407672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b68906a-2c7d-44a3-837c-d62d9a661c79-root\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.408028 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:54.407697 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 14 12:59:54.408028 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.407719 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.408028 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.407750 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-wtmp\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.408028 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:54.407776 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-tls podName:4b68906a-2c7d-44a3-837c-d62d9a661c79 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:54.907754646 +0000 UTC m=+95.570650282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-tls") pod "node-exporter-mgbcd" (UID: "4b68906a-2c7d-44a3-837c-d62d9a661c79") : secret "node-exporter-tls" not found Sep 14 12:59:54.408028 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.407806 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4b68906a-2c7d-44a3-837c-d62d9a661c79-root\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.408028 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.407943 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-textfile\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.408310 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.408101 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-accelerators-collector-config\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.408310 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.408124 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4b68906a-2c7d-44a3-837c-d62d9a661c79-metrics-client-ca\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.410202 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.410181 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.420884 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.420854 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvcxp\" (UniqueName: \"kubernetes.io/projected/4b68906a-2c7d-44a3-837c-d62d9a661c79-kube-api-access-gvcxp\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.912421 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:54.912382 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-tls\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:54.912582 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:54.912551 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 14 12:59:54.912633 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:54.912625 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-tls podName:4b68906a-2c7d-44a3-837c-d62d9a661c79 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:55.912610334 +0000 UTC m=+96.575505954 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-tls") pod "node-exporter-mgbcd" (UID: "4b68906a-2c7d-44a3-837c-d62d9a661c79") : secret "node-exporter-tls" not found Sep 14 12:59:55.220634 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.220556 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 12:59:55.223672 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.223656 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.226202 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.226178 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 14 12:59:55.226351 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.226249 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 14 12:59:55.226351 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.226317 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 14 12:59:55.226478 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.226400 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 14 12:59:55.226478 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.226468 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 14 12:59:55.226630 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.226484 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-99ffz\"" Sep 14 12:59:55.226630 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.226538 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 14 12:59:55.226630 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.226583 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 14 12:59:55.226630 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.226623 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 14 12:59:55.226822 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.226632 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 14 12:59:55.240679 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.240659 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 12:59:55.316940 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.316887 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.317108 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.316950 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.317108 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.317004 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-config-volume\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.317108 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.317024 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.317108 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.317049 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.317108 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.317106 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.317318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.317131 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.317318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.317153 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.317318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.317168 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.317318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.317252 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-config-out\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.317318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.317292 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.317318 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.317310 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-web-config\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.317516 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.317324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk85j\" (UniqueName: \"kubernetes.io/projected/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-kube-api-access-tk85j\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.417696 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.417665 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.417696 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.417701 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.417972 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.417720 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.417972 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.417745 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-config-out\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.417972 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.417777 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.418107 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.417966 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-web-config\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.418107 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.418014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk85j\" (UniqueName: \"kubernetes.io/projected/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-kube-api-access-tk85j\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.418107 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.418050 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.418107 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.418084 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.418311 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.418130 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-config-volume\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.418311 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.418156 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.418311 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.418179 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.418311 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.418235 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.418523 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.418458 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.418583 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.418542 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.420329 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.419894 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.420329 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:55.420019 2564 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 14 12:59:55.420329 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:55.420077 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-main-tls podName:fde29993-cdbc-4f60-96f2-f2f6728bcb9a nodeName:}" failed. No retries permitted until 2026-09-14 12:59:55.920058294 +0000 UTC m=+96.582953908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "fde29993-cdbc-4f60-96f2-f2f6728bcb9a") : secret "alertmanager-main-tls" not found Sep 14 12:59:55.421546 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.421239 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.421546 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.421525 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-web-config\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.421730 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.421630 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.421790 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.421745 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.422049 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.421998 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.422321 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.422298 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.422969 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.422952 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-config-out\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.423139 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.423125 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-config-volume\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.427258 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.427235 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk85j\" (UniqueName: \"kubernetes.io/projected/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-kube-api-access-tk85j\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.921794 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.921744 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-tls\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:55.922068 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.921808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.924735 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.924703 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:55.924965 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.924919 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4b68906a-2c7d-44a3-837c-d62d9a661c79-node-exporter-tls\") pod \"node-exporter-mgbcd\" (UID: \"4b68906a-2c7d-44a3-837c-d62d9a661c79\") " pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:55.997734 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:55.997699 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mgbcd" Sep 14 12:59:56.009720 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:59:56.009692 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b68906a_2c7d_44a3_837c_d62d9a661c79.slice/crio-ef05c5c1aaecb36e7c84a738607bd3a2eb59fdcd058b17889ec9cfa1b275e409 WatchSource:0}: Error finding container ef05c5c1aaecb36e7c84a738607bd3a2eb59fdcd058b17889ec9cfa1b275e409: Status 404 returned error can't find the container with id ef05c5c1aaecb36e7c84a738607bd3a2eb59fdcd058b17889ec9cfa1b275e409 Sep 14 12:59:56.134396 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:56.134364 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 12:59:56.269889 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:56.269853 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgbcd" event={"ID":"4b68906a-2c7d-44a3-837c-d62d9a661c79","Type":"ContainerStarted","Data":"ef05c5c1aaecb36e7c84a738607bd3a2eb59fdcd058b17889ec9cfa1b275e409"} Sep 14 12:59:56.416113 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:56.416085 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 12:59:56.417396 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:59:56.417372 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde29993_cdbc_4f60_96f2_f2f6728bcb9a.slice/crio-7007b152c8095775fc0ae2b5a2e23b22680d2c1f26d86b9665ad9dd7fad79673 WatchSource:0}: Error finding container 7007b152c8095775fc0ae2b5a2e23b22680d2c1f26d86b9665ad9dd7fad79673: Status 404 returned error can't find the container with id 7007b152c8095775fc0ae2b5a2e23b22680d2c1f26d86b9665ad9dd7fad79673 Sep 14 12:59:57.276123 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:57.276084 2564 generic.go:356] "Generic (PLEG): container finished" podID="4b68906a-2c7d-44a3-837c-d62d9a661c79" containerID="673b0e998555155a562b166b4ae3bc28949928d626e2cc270abf84149495eb3e" exitCode=0 Sep 14 12:59:57.276596 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:57.276141 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgbcd" event={"ID":"4b68906a-2c7d-44a3-837c-d62d9a661c79","Type":"ContainerDied","Data":"673b0e998555155a562b166b4ae3bc28949928d626e2cc270abf84149495eb3e"} Sep 14 12:59:57.277753 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:57.277719 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fde29993-cdbc-4f60-96f2-f2f6728bcb9a","Type":"ContainerStarted","Data":"7007b152c8095775fc0ae2b5a2e23b22680d2c1f26d86b9665ad9dd7fad79673"} Sep 14 12:59:58.141785 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.141756 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7t274" Sep 14 12:59:58.193837 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.193796 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" podUID="650e0502-50be-4f10-b3b3-7cf515fdb02d" containerName="registry" containerID="cri-o://9f479b5474cb6c2f5cdfa9b140a1dbc0ae162457c683c89644fb5955802e3e1b" gracePeriod=30 Sep 14 12:59:58.284080 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.284042 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgbcd" event={"ID":"4b68906a-2c7d-44a3-837c-d62d9a661c79","Type":"ContainerStarted","Data":"6ed70ac9fc3d285a2143fe00b2fd22069b55e665377b854a73605bcfad381057"} Sep 14 12:59:58.284477 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.284104 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mgbcd" event={"ID":"4b68906a-2c7d-44a3-837c-d62d9a661c79","Type":"ContainerStarted","Data":"eb71a1bba8190de89018b3b71e81aa7bea13c66417ca45eb3f5efd0fecad02c3"} Sep 14 12:59:58.285663 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.285641 2564 generic.go:356] "Generic (PLEG): container finished" podID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerID="f74ec29cd3742edd5ab28a7036c3e18296548c62c77015d84f274f112e88d877" exitCode=0 Sep 14 12:59:58.285779 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.285694 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fde29993-cdbc-4f60-96f2-f2f6728bcb9a","Type":"ContainerDied","Data":"f74ec29cd3742edd5ab28a7036c3e18296548c62c77015d84f274f112e88d877"} Sep 14 12:59:58.303344 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.303271 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mgbcd" podStartSLOduration=3.395049546 podStartE2EDuration="4.303254376s" totalImagesPullingTime="834.880521ms" totalInitContainerRuntime="73.324309ms" isStatefulPod=true podCreationTimestamp="2026-09-14 12:59:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 12:59:58.302378183 +0000 UTC m=+98.965273851" watchObservedRunningTime="2026-09-14 12:59:58.303254376 +0000 UTC m=+98.966150006" Sep 14 12:59:58.427619 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.427593 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:59:58.457111 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.457080 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-88bd8b6b4-bf4dh"] Sep 14 12:59:58.457510 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.457492 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="650e0502-50be-4f10-b3b3-7cf515fdb02d" containerName="registry" Sep 14 12:59:58.457564 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.457515 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="650e0502-50be-4f10-b3b3-7cf515fdb02d" containerName="registry" Sep 14 12:59:58.457608 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.457597 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="650e0502-50be-4f10-b3b3-7cf515fdb02d" containerName="registry" Sep 14 12:59:58.459831 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.459807 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.462546 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.462519 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 14 12:59:58.462709 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.462525 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-aqdb0mrvednsk\"" Sep 14 12:59:58.462709 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.462592 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jlss8\"" Sep 14 12:59:58.462874 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.462804 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 14 12:59:58.462874 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.462821 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 14 12:59:58.463007 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.462893 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 14 12:59:58.470777 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.470711 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-88bd8b6b4-bf4dh"] Sep 14 12:59:58.547028 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.547003 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/650e0502-50be-4f10-b3b3-7cf515fdb02d-trusted-ca\") pod \"650e0502-50be-4f10-b3b3-7cf515fdb02d\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " Sep 14 12:59:58.547194 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.547035 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/650e0502-50be-4f10-b3b3-7cf515fdb02d-ca-trust-extracted\") pod \"650e0502-50be-4f10-b3b3-7cf515fdb02d\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " Sep 14 12:59:58.547194 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.547070 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/650e0502-50be-4f10-b3b3-7cf515fdb02d-image-registry-private-configuration\") pod \"650e0502-50be-4f10-b3b3-7cf515fdb02d\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " Sep 14 12:59:58.547194 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.547101 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-bound-sa-token\") pod \"650e0502-50be-4f10-b3b3-7cf515fdb02d\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " Sep 14 12:59:58.547194 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.547119 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls\") pod \"650e0502-50be-4f10-b3b3-7cf515fdb02d\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " Sep 14 12:59:58.547194 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.547138 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-certificates\") pod \"650e0502-50be-4f10-b3b3-7cf515fdb02d\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " Sep 14 12:59:58.547194 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.547156 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/650e0502-50be-4f10-b3b3-7cf515fdb02d-installation-pull-secrets\") pod \"650e0502-50be-4f10-b3b3-7cf515fdb02d\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " Sep 14 12:59:58.547466 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.547199 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44dzv\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-kube-api-access-44dzv\") pod \"650e0502-50be-4f10-b3b3-7cf515fdb02d\" (UID: \"650e0502-50be-4f10-b3b3-7cf515fdb02d\") " Sep 14 12:59:58.547871 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.547832 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "650e0502-50be-4f10-b3b3-7cf515fdb02d" (UID: "650e0502-50be-4f10-b3b3-7cf515fdb02d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 12:59:58.547871 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.547839 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/650e0502-50be-4f10-b3b3-7cf515fdb02d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "650e0502-50be-4f10-b3b3-7cf515fdb02d" (UID: "650e0502-50be-4f10-b3b3-7cf515fdb02d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 12:59:58.549860 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.549836 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "650e0502-50be-4f10-b3b3-7cf515fdb02d" (UID: "650e0502-50be-4f10-b3b3-7cf515fdb02d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 12:59:58.550046 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.549996 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-kube-api-access-44dzv" (OuterVolumeSpecName: "kube-api-access-44dzv") pod "650e0502-50be-4f10-b3b3-7cf515fdb02d" (UID: "650e0502-50be-4f10-b3b3-7cf515fdb02d"). InnerVolumeSpecName "kube-api-access-44dzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 12:59:58.550112 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.550086 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "650e0502-50be-4f10-b3b3-7cf515fdb02d" (UID: "650e0502-50be-4f10-b3b3-7cf515fdb02d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 12:59:58.550303 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.550283 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/650e0502-50be-4f10-b3b3-7cf515fdb02d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "650e0502-50be-4f10-b3b3-7cf515fdb02d" (UID: "650e0502-50be-4f10-b3b3-7cf515fdb02d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 12:59:58.550415 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.550371 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/650e0502-50be-4f10-b3b3-7cf515fdb02d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "650e0502-50be-4f10-b3b3-7cf515fdb02d" (UID: "650e0502-50be-4f10-b3b3-7cf515fdb02d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 12:59:58.556226 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.556201 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/650e0502-50be-4f10-b3b3-7cf515fdb02d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "650e0502-50be-4f10-b3b3-7cf515fdb02d" (UID: "650e0502-50be-4f10-b3b3-7cf515fdb02d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 12:59:58.648674 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.648637 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aea629de-fae3-4027-9931-197ffd9e91dc-secret-metrics-server-tls\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.648909 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.648678 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aea629de-fae3-4027-9931-197ffd9e91dc-audit-log\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.648909 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.648716 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aea629de-fae3-4027-9931-197ffd9e91dc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.648909 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.648763 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x45zj\" (UniqueName: \"kubernetes.io/projected/aea629de-fae3-4027-9931-197ffd9e91dc-kube-api-access-x45zj\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.648909 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.648798 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aea629de-fae3-4027-9931-197ffd9e91dc-secret-metrics-server-client-certs\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.648909 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.648821 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aea629de-fae3-4027-9931-197ffd9e91dc-metrics-server-audit-profiles\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.648909 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.648835 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aea629de-fae3-4027-9931-197ffd9e91dc-client-ca-bundle\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.649201 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.648959 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/650e0502-50be-4f10-b3b3-7cf515fdb02d-trusted-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 12:59:58.649201 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.648978 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/650e0502-50be-4f10-b3b3-7cf515fdb02d-ca-trust-extracted\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 12:59:58.649201 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.648993 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/650e0502-50be-4f10-b3b3-7cf515fdb02d-image-registry-private-configuration\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 12:59:58.649201 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.649007 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-bound-sa-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 12:59:58.649201 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.649020 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 12:59:58.649201 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.649032 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/650e0502-50be-4f10-b3b3-7cf515fdb02d-registry-certificates\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 12:59:58.649201 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.649046 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/650e0502-50be-4f10-b3b3-7cf515fdb02d-installation-pull-secrets\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 12:59:58.649201 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.649062 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44dzv\" (UniqueName: \"kubernetes.io/projected/650e0502-50be-4f10-b3b3-7cf515fdb02d-kube-api-access-44dzv\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 12:59:58.749982 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.749942 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aea629de-fae3-4027-9931-197ffd9e91dc-secret-metrics-server-tls\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.749982 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.749989 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aea629de-fae3-4027-9931-197ffd9e91dc-audit-log\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.750229 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.750024 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aea629de-fae3-4027-9931-197ffd9e91dc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.750229 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.750056 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x45zj\" (UniqueName: \"kubernetes.io/projected/aea629de-fae3-4027-9931-197ffd9e91dc-kube-api-access-x45zj\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.750229 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.750089 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aea629de-fae3-4027-9931-197ffd9e91dc-secret-metrics-server-client-certs\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.750229 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.750110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aea629de-fae3-4027-9931-197ffd9e91dc-metrics-server-audit-profiles\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.750229 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.750124 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aea629de-fae3-4027-9931-197ffd9e91dc-client-ca-bundle\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.750518 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.750489 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/aea629de-fae3-4027-9931-197ffd9e91dc-audit-log\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.750788 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.750761 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aea629de-fae3-4027-9931-197ffd9e91dc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.751598 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.751556 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/aea629de-fae3-4027-9931-197ffd9e91dc-metrics-server-audit-profiles\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.753183 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.753158 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/aea629de-fae3-4027-9931-197ffd9e91dc-secret-metrics-server-tls\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.753298 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.753271 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/aea629de-fae3-4027-9931-197ffd9e91dc-secret-metrics-server-client-certs\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.753365 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.753317 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/aea629de-fae3-4027-9931-197ffd9e91dc-client-ca-bundle\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.758387 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.758365 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x45zj\" (UniqueName: \"kubernetes.io/projected/aea629de-fae3-4027-9931-197ffd9e91dc-kube-api-access-x45zj\") pod \"metrics-server-88bd8b6b4-bf4dh\" (UID: \"aea629de-fae3-4027-9931-197ffd9e91dc\") " pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.778864 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.778836 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 12:59:58.988508 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.988475 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85c7f8f5b-r4ljw"] Sep 14 12:59:58.992611 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.991677 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:58.995807 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.995781 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 14 12:59:58.995807 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.995803 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 14 12:59:58.996047 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.996021 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 14 12:59:58.996125 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.996104 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 14 12:59:58.996784 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.996249 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 14 12:59:58.996784 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.996504 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pmz8m\"" Sep 14 12:59:58.996784 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.996021 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 14 12:59:58.997545 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:58.997378 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 14 12:59:59.001074 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.001052 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-j4x9t"] Sep 14 12:59:59.001457 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.001438 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 14 12:59:59.004133 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.004116 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4x9t" Sep 14 12:59:59.006893 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.006771 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-2x88v\"" Sep 14 12:59:59.006893 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.006862 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 14 12:59:59.010655 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.010629 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c7f8f5b-r4ljw"] Sep 14 12:59:59.013901 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.013877 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-j4x9t"] Sep 14 12:59:59.058092 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.058063 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-88bd8b6b4-bf4dh"] Sep 14 12:59:59.060941 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:59:59.060900 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea629de_fae3_4027_9931_197ffd9e91dc.slice/crio-e816678724541e0817fe143baeb471fdfcbdb8e9337a0e52c90483098e21a210 WatchSource:0}: Error finding container e816678724541e0817fe143baeb471fdfcbdb8e9337a0e52c90483098e21a210: Status 404 returned error can't find the container with id e816678724541e0817fe143baeb471fdfcbdb8e9337a0e52c90483098e21a210 Sep 14 12:59:59.155274 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.155244 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-console-config\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.155274 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.155284 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-oauth-serving-cert\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.155552 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.155326 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69d15a6f-cd51-456e-a72c-5b62e13720c8-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-j4x9t\" (UID: \"69d15a6f-cd51-456e-a72c-5b62e13720c8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4x9t" Sep 14 12:59:59.155552 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.155352 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-trusted-ca-bundle\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.155552 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.155382 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-service-ca\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.155552 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.155406 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb30aede-458d-4b29-9a2d-71868e936370-console-serving-cert\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.155552 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.155449 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb30aede-458d-4b29-9a2d-71868e936370-console-oauth-config\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.155552 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.155502 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78p57\" (UniqueName: \"kubernetes.io/projected/69d15a6f-cd51-456e-a72c-5b62e13720c8-kube-api-access-78p57\") pod \"monitoring-plugin-5d85778b9d-j4x9t\" (UID: \"69d15a6f-cd51-456e-a72c-5b62e13720c8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4x9t" Sep 14 12:59:59.155552 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.155531 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgt5f\" (UniqueName: \"kubernetes.io/projected/fb30aede-458d-4b29-9a2d-71868e936370-kube-api-access-kgt5f\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.256812 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.256757 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgt5f\" (UniqueName: \"kubernetes.io/projected/fb30aede-458d-4b29-9a2d-71868e936370-kube-api-access-kgt5f\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.257014 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.256867 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-console-config\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.257014 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.256891 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-oauth-serving-cert\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.257014 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.256915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69d15a6f-cd51-456e-a72c-5b62e13720c8-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-j4x9t\" (UID: \"69d15a6f-cd51-456e-a72c-5b62e13720c8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4x9t" Sep 14 12:59:59.257014 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.256980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-trusted-ca-bundle\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.257014 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.257012 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-service-ca\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.257323 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.257034 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb30aede-458d-4b29-9a2d-71868e936370-console-serving-cert\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.257323 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.257080 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb30aede-458d-4b29-9a2d-71868e936370-console-oauth-config\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.257323 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.257104 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78p57\" (UniqueName: \"kubernetes.io/projected/69d15a6f-cd51-456e-a72c-5b62e13720c8-kube-api-access-78p57\") pod \"monitoring-plugin-5d85778b9d-j4x9t\" (UID: \"69d15a6f-cd51-456e-a72c-5b62e13720c8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4x9t" Sep 14 12:59:59.257323 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:59.257111 2564 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 14 12:59:59.257323 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:59.257203 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69d15a6f-cd51-456e-a72c-5b62e13720c8-monitoring-plugin-cert podName:69d15a6f-cd51-456e-a72c-5b62e13720c8 nodeName:}" failed. No retries permitted until 2026-09-14 12:59:59.757187937 +0000 UTC m=+100.420083558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/69d15a6f-cd51-456e-a72c-5b62e13720c8-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-j4x9t" (UID: "69d15a6f-cd51-456e-a72c-5b62e13720c8") : secret "monitoring-plugin-cert" not found Sep 14 12:59:59.257737 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.257675 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-console-config\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.257737 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.257723 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-oauth-serving-cert\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.258306 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.258261 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-trusted-ca-bundle\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.258306 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.258281 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-service-ca\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.260052 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.260031 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb30aede-458d-4b29-9a2d-71868e936370-console-oauth-config\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.260206 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.260190 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb30aede-458d-4b29-9a2d-71868e936370-console-serving-cert\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.265539 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.265522 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgt5f\" (UniqueName: \"kubernetes.io/projected/fb30aede-458d-4b29-9a2d-71868e936370-kube-api-access-kgt5f\") pod \"console-85c7f8f5b-r4ljw\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.265644 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.265599 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78p57\" (UniqueName: \"kubernetes.io/projected/69d15a6f-cd51-456e-a72c-5b62e13720c8-kube-api-access-78p57\") pod \"monitoring-plugin-5d85778b9d-j4x9t\" (UID: \"69d15a6f-cd51-456e-a72c-5b62e13720c8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4x9t" Sep 14 12:59:59.292963 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.292909 2564 generic.go:356] "Generic (PLEG): container finished" podID="650e0502-50be-4f10-b3b3-7cf515fdb02d" containerID="9f479b5474cb6c2f5cdfa9b140a1dbc0ae162457c683c89644fb5955802e3e1b" exitCode=0 Sep 14 12:59:59.293386 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.293004 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" Sep 14 12:59:59.293386 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.293013 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" event={"ID":"650e0502-50be-4f10-b3b3-7cf515fdb02d","Type":"ContainerDied","Data":"9f479b5474cb6c2f5cdfa9b140a1dbc0ae162457c683c89644fb5955802e3e1b"} Sep 14 12:59:59.293386 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.293074 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f5cb7776d-tzhs8" event={"ID":"650e0502-50be-4f10-b3b3-7cf515fdb02d","Type":"ContainerDied","Data":"a8988cbb9b2174480b60dd05f049ebe2a3e05c750bad85cb4d9573daa90cec14"} Sep 14 12:59:59.293386 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.293109 2564 scope.go:117] "RemoveContainer" containerID="9f479b5474cb6c2f5cdfa9b140a1dbc0ae162457c683c89644fb5955802e3e1b" Sep 14 12:59:59.295208 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.295187 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" event={"ID":"aea629de-fae3-4027-9931-197ffd9e91dc","Type":"ContainerStarted","Data":"e816678724541e0817fe143baeb471fdfcbdb8e9337a0e52c90483098e21a210"} Sep 14 12:59:59.306766 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.306721 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 12:59:59.321949 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.321914 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f5cb7776d-tzhs8"] Sep 14 12:59:59.325811 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.325793 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f5cb7776d-tzhs8"] Sep 14 12:59:59.552547 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.552524 2564 scope.go:117] "RemoveContainer" containerID="9f479b5474cb6c2f5cdfa9b140a1dbc0ae162457c683c89644fb5955802e3e1b" Sep 14 12:59:59.552969 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:59.552909 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f479b5474cb6c2f5cdfa9b140a1dbc0ae162457c683c89644fb5955802e3e1b\": container with ID starting with 9f479b5474cb6c2f5cdfa9b140a1dbc0ae162457c683c89644fb5955802e3e1b not found: ID does not exist" containerID="9f479b5474cb6c2f5cdfa9b140a1dbc0ae162457c683c89644fb5955802e3e1b" Sep 14 12:59:59.553085 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.552986 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f479b5474cb6c2f5cdfa9b140a1dbc0ae162457c683c89644fb5955802e3e1b"} err="failed to get container status \"9f479b5474cb6c2f5cdfa9b140a1dbc0ae162457c683c89644fb5955802e3e1b\": rpc error: code = NotFound desc = could not find container \"9f479b5474cb6c2f5cdfa9b140a1dbc0ae162457c683c89644fb5955802e3e1b\": container with ID starting with 9f479b5474cb6c2f5cdfa9b140a1dbc0ae162457c683c89644fb5955802e3e1b not found: ID does not exist" Sep 14 12:59:59.761002 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.760969 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69d15a6f-cd51-456e-a72c-5b62e13720c8-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-j4x9t\" (UID: \"69d15a6f-cd51-456e-a72c-5b62e13720c8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4x9t" Sep 14 12:59:59.761154 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:59.761082 2564 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 14 12:59:59.761154 ip-10-0-136-96 kubenswrapper[2564]: E0914 12:59:59.761141 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69d15a6f-cd51-456e-a72c-5b62e13720c8-monitoring-plugin-cert podName:69d15a6f-cd51-456e-a72c-5b62e13720c8 nodeName:}" failed. No retries permitted until 2026-09-14 13:00:00.761125242 +0000 UTC m=+101.424020869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/69d15a6f-cd51-456e-a72c-5b62e13720c8-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-j4x9t" (UID: "69d15a6f-cd51-456e-a72c-5b62e13720c8") : secret "monitoring-plugin-cert" not found Sep 14 12:59:59.810499 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.810413 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="650e0502-50be-4f10-b3b3-7cf515fdb02d" path="/var/lib/kubelet/pods/650e0502-50be-4f10-b3b3-7cf515fdb02d/volumes" Sep 14 12:59:59.820820 ip-10-0-136-96 kubenswrapper[2564]: I0914 12:59:59.820686 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c7f8f5b-r4ljw"] Sep 14 12:59:59.899812 ip-10-0-136-96 kubenswrapper[2564]: W0914 12:59:59.899777 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb30aede_458d_4b29_9a2d_71868e936370.slice/crio-0c8b8ae1b37daeb734565e422f265e477b92e52e0e69c3a1792a2f38e0fb75f6 WatchSource:0}: Error finding container 0c8b8ae1b37daeb734565e422f265e477b92e52e0e69c3a1792a2f38e0fb75f6: Status 404 returned error can't find the container with id 0c8b8ae1b37daeb734565e422f265e477b92e52e0e69c3a1792a2f38e0fb75f6 Sep 14 13:00:00.301747 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:00.301711 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c7f8f5b-r4ljw" event={"ID":"fb30aede-458d-4b29-9a2d-71868e936370","Type":"ContainerStarted","Data":"0c8b8ae1b37daeb734565e422f265e477b92e52e0e69c3a1792a2f38e0fb75f6"} Sep 14 13:00:00.304804 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:00.304776 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fde29993-cdbc-4f60-96f2-f2f6728bcb9a","Type":"ContainerStarted","Data":"4d3c02aa5156ef6cb96cdc9fd291c6d955e445848c9a77e48d62040abee3a2bf"} Sep 14 13:00:00.304948 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:00.304839 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fde29993-cdbc-4f60-96f2-f2f6728bcb9a","Type":"ContainerStarted","Data":"5d767fe650764561a02f8094623649d75738096139b978256a2ce03de4e01b8e"} Sep 14 13:00:00.769810 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:00.769772 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69d15a6f-cd51-456e-a72c-5b62e13720c8-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-j4x9t\" (UID: \"69d15a6f-cd51-456e-a72c-5b62e13720c8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4x9t" Sep 14 13:00:00.772860 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:00.772828 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/69d15a6f-cd51-456e-a72c-5b62e13720c8-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-j4x9t\" (UID: \"69d15a6f-cd51-456e-a72c-5b62e13720c8\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4x9t" Sep 14 13:00:00.816824 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:00.816786 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4x9t" Sep 14 13:00:01.313566 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:01.313490 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fde29993-cdbc-4f60-96f2-f2f6728bcb9a","Type":"ContainerStarted","Data":"e3b4df28d718b3c846b966550492a341cbdcaf06fcebacce6152fa27f3880348"} Sep 14 13:00:01.314028 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:01.313567 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fde29993-cdbc-4f60-96f2-f2f6728bcb9a","Type":"ContainerStarted","Data":"70295f9cdb122f74a65229332900022f87ec0cc0a78d1b3c235083d361fbb340"} Sep 14 13:00:01.314028 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:01.313613 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fde29993-cdbc-4f60-96f2-f2f6728bcb9a","Type":"ContainerStarted","Data":"d3ca7374424fa84f1df3017e4e5f22ef4ea5c9b99d69b88023ccd55e9dd8d81e"} Sep 14 13:00:01.339123 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:01.339095 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-j4x9t"] Sep 14 13:00:01.370122 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:00:01.370087 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d15a6f_cd51_456e_a72c_5b62e13720c8.slice/crio-eaf748c14bf414bef7d837ce519d64b83de0a932c3088c21ef898edfeae908e2 WatchSource:0}: Error finding container eaf748c14bf414bef7d837ce519d64b83de0a932c3088c21ef898edfeae908e2: Status 404 returned error can't find the container with id eaf748c14bf414bef7d837ce519d64b83de0a932c3088c21ef898edfeae908e2 Sep 14 13:00:02.320609 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:02.320574 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" event={"ID":"aea629de-fae3-4027-9931-197ffd9e91dc","Type":"ContainerStarted","Data":"74209d59ce79547bd5b48754ad04e1384e44504cb7d06198d455ac7ecdfe8888"} Sep 14 13:00:02.322397 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:02.322368 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4x9t" event={"ID":"69d15a6f-cd51-456e-a72c-5b62e13720c8","Type":"ContainerStarted","Data":"eaf748c14bf414bef7d837ce519d64b83de0a932c3088c21ef898edfeae908e2"} Sep 14 13:00:02.339285 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:02.339228 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" podStartSLOduration=2.008257208 podStartE2EDuration="4.339206805s" totalImagesPullingTime="2.330949597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:00:02.337406941 +0000 UTC m=+103.000302571" watchObservedRunningTime="2026-09-14 13:00:02.339206805 +0000 UTC m=+103.002102438" Sep 14 13:00:04.332869 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:04.332834 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c7f8f5b-r4ljw" event={"ID":"fb30aede-458d-4b29-9a2d-71868e936370","Type":"ContainerStarted","Data":"29d96ba280609a2db785b0a07a87a2eb27aa089399b815bbbfe6c5f5ef9ec7de"} Sep 14 13:00:04.336441 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:04.336413 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fde29993-cdbc-4f60-96f2-f2f6728bcb9a","Type":"ContainerStarted","Data":"da104b6dc706486554d44ad0083ba3d1efd82c3a4408a578c59cb401acadabfc"} Sep 14 13:00:04.355517 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:04.355465 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85c7f8f5b-r4ljw" podStartSLOduration=2.916367708 podStartE2EDuration="6.355448314s" totalImagesPullingTime="3.439080606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:00:04.354316139 +0000 UTC m=+105.017211801" watchObservedRunningTime="2026-09-14 13:00:04.355448314 +0000 UTC m=+105.018343944" Sep 14 13:00:04.382824 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:04.382766 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.898564566 podStartE2EDuration="9.382747921s" totalImagesPullingTime="5.463044352s" totalInitContainerRuntime="21.139003ms" isStatefulPod=false podCreationTimestamp="2026-09-14 12:59:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:00:04.381653346 +0000 UTC m=+105.044548987" watchObservedRunningTime="2026-09-14 13:00:04.382747921 +0000 UTC m=+105.045643553" Sep 14 13:00:05.341604 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:05.341557 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4x9t" event={"ID":"69d15a6f-cd51-456e-a72c-5b62e13720c8","Type":"ContainerStarted","Data":"d2dd78f75ad5fafee200d5c9002d98a789c4ab053da30391a8e3d11691c20d09"} Sep 14 13:00:05.342149 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:05.342129 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4x9t" Sep 14 13:00:05.346804 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:05.346779 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4x9t" Sep 14 13:00:05.359362 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:05.359320 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4x9t" podStartSLOduration=4.370556254 podStartE2EDuration="7.359306078s" totalImagesPullingTime="2.988749824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 12:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:00:05.35741281 +0000 UTC m=+106.020308440" watchObservedRunningTime="2026-09-14 13:00:05.359306078 +0000 UTC m=+106.022201707" Sep 14 13:00:09.306869 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:09.306828 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 13:00:09.307352 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:09.306886 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 13:00:09.311684 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:09.311665 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 13:00:09.362162 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:09.362138 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 13:00:12.370336 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:12.370305 2564 generic.go:356] "Generic (PLEG): container finished" podID="544ba1bf-58c6-428c-9830-a456e4940b63" containerID="52fcbeca0ed5db1cf730868619bbffd60066ffff1ab2235425207b4a36690ac1" exitCode=0 Sep 14 13:00:12.370722 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:12.370381 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2vff4" event={"ID":"544ba1bf-58c6-428c-9830-a456e4940b63","Type":"ContainerDied","Data":"52fcbeca0ed5db1cf730868619bbffd60066ffff1ab2235425207b4a36690ac1"} Sep 14 13:00:12.370765 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:12.370749 2564 scope.go:117] "RemoveContainer" containerID="52fcbeca0ed5db1cf730868619bbffd60066ffff1ab2235425207b4a36690ac1" Sep 14 13:00:13.376811 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:13.376775 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-2vff4" event={"ID":"544ba1bf-58c6-428c-9830-a456e4940b63","Type":"ContainerStarted","Data":"f3330a20c4e6671f56fb3cab6644c24a35b2c7161d416f88649a86a3470464ab"} Sep 14 13:00:18.779557 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:18.779517 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 13:00:18.779973 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:18.779583 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 13:00:22.413513 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:22.413480 2564 generic.go:356] "Generic (PLEG): container finished" podID="ad5768be-fcc1-4c56-a485-2ae3f04504bc" containerID="da5b6818f60a8701739cfcc7ae7a4a9a08935923d068e0df50bf7bbc966d8730" exitCode=0 Sep 14 13:00:22.413513 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:22.413515 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-2xl57" event={"ID":"ad5768be-fcc1-4c56-a485-2ae3f04504bc","Type":"ContainerDied","Data":"da5b6818f60a8701739cfcc7ae7a4a9a08935923d068e0df50bf7bbc966d8730"} Sep 14 13:00:22.413990 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:22.413831 2564 scope.go:117] "RemoveContainer" containerID="da5b6818f60a8701739cfcc7ae7a4a9a08935923d068e0df50bf7bbc966d8730" Sep 14 13:00:23.234216 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:23.234185 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64988bbc7d-tnjdw_0287d99a-ca2c-45bc-8f32-a2a273401e51/router/0.log" Sep 14 13:00:23.245174 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:23.245150 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dskx8_fc512c34-085d-4934-9c57-95f05e3db013/serve-healthcheck-canary/0.log" Sep 14 13:00:23.418976 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:23.418912 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-2xl57" event={"ID":"ad5768be-fcc1-4c56-a485-2ae3f04504bc","Type":"ContainerStarted","Data":"4528f22e638c27a83a2568e3b7c8bce4ae1d44bf6ee059f165f83ce4a945ca6e"} Sep 14 13:00:36.480733 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:36.480680 2564 generic.go:356] "Generic (PLEG): container finished" podID="3be5d339-0117-4588-a03e-8562bc978355" containerID="ef2b2e547c65c663cf3e925ccef1b6f51a994726250779e7701aae45e0d39d10" exitCode=0 Sep 14 13:00:36.481268 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:36.480796 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l78kn" event={"ID":"3be5d339-0117-4588-a03e-8562bc978355","Type":"ContainerDied","Data":"ef2b2e547c65c663cf3e925ccef1b6f51a994726250779e7701aae45e0d39d10"} Sep 14 13:00:36.481268 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:36.481195 2564 scope.go:117] "RemoveContainer" containerID="ef2b2e547c65c663cf3e925ccef1b6f51a994726250779e7701aae45e0d39d10" Sep 14 13:00:37.486174 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:37.486142 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-l78kn" event={"ID":"3be5d339-0117-4588-a03e-8562bc978355","Type":"ContainerStarted","Data":"fd9f77ef483c769aadd8ba873809e8f4f7bbe0b0ca2050a232c52fdd88bb85ab"} Sep 14 13:00:38.784820 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:38.784791 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 13:00:38.788540 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:00:38.788517 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-88bd8b6b4-bf4dh" Sep 14 13:01:14.382770 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:14.382686 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 13:01:14.383349 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:14.383064 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="alertmanager" containerID="cri-o://5d767fe650764561a02f8094623649d75738096139b978256a2ce03de4e01b8e" gracePeriod=120 Sep 14 13:01:14.383349 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:14.383116 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="kube-rbac-proxy-metric" containerID="cri-o://e3b4df28d718b3c846b966550492a341cbdcaf06fcebacce6152fa27f3880348" gracePeriod=120 Sep 14 13:01:14.383349 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:14.383162 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="kube-rbac-proxy-web" containerID="cri-o://d3ca7374424fa84f1df3017e4e5f22ef4ea5c9b99d69b88023ccd55e9dd8d81e" gracePeriod=120 Sep 14 13:01:14.383349 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:14.383133 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="prom-label-proxy" containerID="cri-o://da104b6dc706486554d44ad0083ba3d1efd82c3a4408a578c59cb401acadabfc" gracePeriod=120 Sep 14 13:01:14.383349 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:14.383172 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="kube-rbac-proxy" containerID="cri-o://70295f9cdb122f74a65229332900022f87ec0cc0a78d1b3c235083d361fbb340" gracePeriod=120 Sep 14 13:01:14.383349 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:14.383187 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="config-reloader" containerID="cri-o://4d3c02aa5156ef6cb96cdc9fd291c6d955e445848c9a77e48d62040abee3a2bf" gracePeriod=120 Sep 14 13:01:14.634621 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:14.634543 2564 generic.go:356] "Generic (PLEG): container finished" podID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerID="da104b6dc706486554d44ad0083ba3d1efd82c3a4408a578c59cb401acadabfc" exitCode=0 Sep 14 13:01:14.634621 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:14.634566 2564 generic.go:356] "Generic (PLEG): container finished" podID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerID="70295f9cdb122f74a65229332900022f87ec0cc0a78d1b3c235083d361fbb340" exitCode=0 Sep 14 13:01:14.634621 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:14.634572 2564 generic.go:356] "Generic (PLEG): container finished" podID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerID="4d3c02aa5156ef6cb96cdc9fd291c6d955e445848c9a77e48d62040abee3a2bf" exitCode=0 Sep 14 13:01:14.634621 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:14.634580 2564 generic.go:356] "Generic (PLEG): container finished" podID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerID="5d767fe650764561a02f8094623649d75738096139b978256a2ce03de4e01b8e" exitCode=0 Sep 14 13:01:14.634836 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:14.634619 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fde29993-cdbc-4f60-96f2-f2f6728bcb9a","Type":"ContainerDied","Data":"da104b6dc706486554d44ad0083ba3d1efd82c3a4408a578c59cb401acadabfc"} Sep 14 13:01:14.634836 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:14.634680 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fde29993-cdbc-4f60-96f2-f2f6728bcb9a","Type":"ContainerDied","Data":"70295f9cdb122f74a65229332900022f87ec0cc0a78d1b3c235083d361fbb340"} Sep 14 13:01:14.634836 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:14.634711 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fde29993-cdbc-4f60-96f2-f2f6728bcb9a","Type":"ContainerDied","Data":"4d3c02aa5156ef6cb96cdc9fd291c6d955e445848c9a77e48d62040abee3a2bf"} Sep 14 13:01:14.634836 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:14.634735 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fde29993-cdbc-4f60-96f2-f2f6728bcb9a","Type":"ContainerDied","Data":"5d767fe650764561a02f8094623649d75738096139b978256a2ce03de4e01b8e"} Sep 14 13:01:15.643373 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.643345 2564 generic.go:356] "Generic (PLEG): container finished" podID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerID="e3b4df28d718b3c846b966550492a341cbdcaf06fcebacce6152fa27f3880348" exitCode=0 Sep 14 13:01:15.643373 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.643370 2564 generic.go:356] "Generic (PLEG): container finished" podID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerID="d3ca7374424fa84f1df3017e4e5f22ef4ea5c9b99d69b88023ccd55e9dd8d81e" exitCode=0 Sep 14 13:01:15.643749 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.643394 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fde29993-cdbc-4f60-96f2-f2f6728bcb9a","Type":"ContainerDied","Data":"e3b4df28d718b3c846b966550492a341cbdcaf06fcebacce6152fa27f3880348"} Sep 14 13:01:15.643749 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.643449 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fde29993-cdbc-4f60-96f2-f2f6728bcb9a","Type":"ContainerDied","Data":"d3ca7374424fa84f1df3017e4e5f22ef4ea5c9b99d69b88023ccd55e9dd8d81e"} Sep 14 13:01:15.722652 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.722630 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:15.788866 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.788830 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-cluster-tls-config\") pod \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " Sep 14 13:01:15.789048 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.788896 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-alertmanager-main-db\") pod \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " Sep 14 13:01:15.789048 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.788917 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-kube-rbac-proxy\") pod \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " Sep 14 13:01:15.789048 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.788964 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-config-volume\") pod \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " Sep 14 13:01:15.789048 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.789001 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-alertmanager-trusted-ca-bundle\") pod \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " Sep 14 13:01:15.789048 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.789031 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-kube-rbac-proxy-web\") pod \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " Sep 14 13:01:15.789295 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.789080 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-config-out\") pod \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " Sep 14 13:01:15.789295 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.789103 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tk85j\" (UniqueName: \"kubernetes.io/projected/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-kube-api-access-tk85j\") pod \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " Sep 14 13:01:15.789295 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.789124 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-web-config\") pod \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " Sep 14 13:01:15.789295 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.789157 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-metrics-client-ca\") pod \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " Sep 14 13:01:15.789295 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.789198 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-tls-assets\") pod \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " Sep 14 13:01:15.789295 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.789232 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " Sep 14 13:01:15.789295 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.789265 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-main-tls\") pod \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\" (UID: \"fde29993-cdbc-4f60-96f2-f2f6728bcb9a\") " Sep 14 13:01:15.789295 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.789264 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "fde29993-cdbc-4f60-96f2-f2f6728bcb9a" (UID: "fde29993-cdbc-4f60-96f2-f2f6728bcb9a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:01:15.789687 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.789509 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-alertmanager-main-db\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:01:15.789774 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.789749 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "fde29993-cdbc-4f60-96f2-f2f6728bcb9a" (UID: "fde29993-cdbc-4f60-96f2-f2f6728bcb9a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:01:15.789841 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.789774 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "fde29993-cdbc-4f60-96f2-f2f6728bcb9a" (UID: "fde29993-cdbc-4f60-96f2-f2f6728bcb9a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:01:15.792631 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.792600 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "fde29993-cdbc-4f60-96f2-f2f6728bcb9a" (UID: "fde29993-cdbc-4f60-96f2-f2f6728bcb9a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:01:15.793051 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.793027 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "fde29993-cdbc-4f60-96f2-f2f6728bcb9a" (UID: "fde29993-cdbc-4f60-96f2-f2f6728bcb9a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:01:15.795124 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.795097 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-config-volume" (OuterVolumeSpecName: "config-volume") pod "fde29993-cdbc-4f60-96f2-f2f6728bcb9a" (UID: "fde29993-cdbc-4f60-96f2-f2f6728bcb9a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:01:15.795235 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.795192 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-config-out" (OuterVolumeSpecName: "config-out") pod "fde29993-cdbc-4f60-96f2-f2f6728bcb9a" (UID: "fde29993-cdbc-4f60-96f2-f2f6728bcb9a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:01:15.795292 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.795260 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "fde29993-cdbc-4f60-96f2-f2f6728bcb9a" (UID: "fde29993-cdbc-4f60-96f2-f2f6728bcb9a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:01:15.795666 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.795513 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-kube-api-access-tk85j" (OuterVolumeSpecName: "kube-api-access-tk85j") pod "fde29993-cdbc-4f60-96f2-f2f6728bcb9a" (UID: "fde29993-cdbc-4f60-96f2-f2f6728bcb9a"). InnerVolumeSpecName "kube-api-access-tk85j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:01:15.796214 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.796184 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "fde29993-cdbc-4f60-96f2-f2f6728bcb9a" (UID: "fde29993-cdbc-4f60-96f2-f2f6728bcb9a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:01:15.796615 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.796594 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "fde29993-cdbc-4f60-96f2-f2f6728bcb9a" (UID: "fde29993-cdbc-4f60-96f2-f2f6728bcb9a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:01:15.796791 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.796776 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "fde29993-cdbc-4f60-96f2-f2f6728bcb9a" (UID: "fde29993-cdbc-4f60-96f2-f2f6728bcb9a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:01:15.803456 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.803435 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-web-config" (OuterVolumeSpecName: "web-config") pod "fde29993-cdbc-4f60-96f2-f2f6728bcb9a" (UID: "fde29993-cdbc-4f60-96f2-f2f6728bcb9a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:01:15.890419 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.890383 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:01:15.890419 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.890416 2564 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-config-volume\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:01:15.890419 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.890427 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:01:15.890627 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.890438 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:01:15.890627 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.890446 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-config-out\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:01:15.890627 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.890456 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk85j\" (UniqueName: \"kubernetes.io/projected/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-kube-api-access-tk85j\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:01:15.890627 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.890464 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-web-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:01:15.890627 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.890472 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-metrics-client-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:01:15.890627 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.890479 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-tls-assets\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:01:15.890627 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.890487 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:01:15.890627 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.890495 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-secret-alertmanager-main-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:01:15.890627 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:15.890503 2564 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fde29993-cdbc-4f60-96f2-f2f6728bcb9a-cluster-tls-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:01:16.650635 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.650597 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fde29993-cdbc-4f60-96f2-f2f6728bcb9a","Type":"ContainerDied","Data":"7007b152c8095775fc0ae2b5a2e23b22680d2c1f26d86b9665ad9dd7fad79673"} Sep 14 13:01:16.651061 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.650654 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.651061 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.650675 2564 scope.go:117] "RemoveContainer" containerID="da104b6dc706486554d44ad0083ba3d1efd82c3a4408a578c59cb401acadabfc" Sep 14 13:01:16.659092 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.659073 2564 scope.go:117] "RemoveContainer" containerID="e3b4df28d718b3c846b966550492a341cbdcaf06fcebacce6152fa27f3880348" Sep 14 13:01:16.666025 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.666004 2564 scope.go:117] "RemoveContainer" containerID="70295f9cdb122f74a65229332900022f87ec0cc0a78d1b3c235083d361fbb340" Sep 14 13:01:16.673495 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.673479 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 13:01:16.673550 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.673534 2564 scope.go:117] "RemoveContainer" containerID="d3ca7374424fa84f1df3017e4e5f22ef4ea5c9b99d69b88023ccd55e9dd8d81e" Sep 14 13:01:16.677646 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.677628 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 13:01:16.680525 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.680511 2564 scope.go:117] "RemoveContainer" containerID="4d3c02aa5156ef6cb96cdc9fd291c6d955e445848c9a77e48d62040abee3a2bf" Sep 14 13:01:16.687262 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.687243 2564 scope.go:117] "RemoveContainer" containerID="5d767fe650764561a02f8094623649d75738096139b978256a2ce03de4e01b8e" Sep 14 13:01:16.694255 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.694237 2564 scope.go:117] "RemoveContainer" containerID="f74ec29cd3742edd5ab28a7036c3e18296548c62c77015d84f274f112e88d877" Sep 14 13:01:16.703569 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.703549 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 13:01:16.703831 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.703821 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="kube-rbac-proxy" Sep 14 13:01:16.703872 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.703833 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="kube-rbac-proxy" Sep 14 13:01:16.703872 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.703842 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="kube-rbac-proxy-metric" Sep 14 13:01:16.703872 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.703848 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="kube-rbac-proxy-metric" Sep 14 13:01:16.703872 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.703859 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="kube-rbac-proxy-web" Sep 14 13:01:16.703872 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.703864 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="kube-rbac-proxy-web" Sep 14 13:01:16.703872 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.703871 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="config-reloader" Sep 14 13:01:16.704079 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.703876 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="config-reloader" Sep 14 13:01:16.704079 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.703882 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="prom-label-proxy" Sep 14 13:01:16.704079 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.703887 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="prom-label-proxy" Sep 14 13:01:16.704079 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.703895 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="init-config-reloader" Sep 14 13:01:16.704079 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.703900 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="init-config-reloader" Sep 14 13:01:16.704079 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.703907 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="alertmanager" Sep 14 13:01:16.704079 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.703912 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="alertmanager" Sep 14 13:01:16.704079 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.703978 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="config-reloader" Sep 14 13:01:16.704079 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.703986 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="alertmanager" Sep 14 13:01:16.704079 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.703992 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="prom-label-proxy" Sep 14 13:01:16.704079 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.703999 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="kube-rbac-proxy-web" Sep 14 13:01:16.704079 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.704005 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="kube-rbac-proxy" Sep 14 13:01:16.704079 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.704010 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" containerName="kube-rbac-proxy-metric" Sep 14 13:01:16.707734 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.707719 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.710506 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.710492 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 14 13:01:16.710580 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.710494 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 14 13:01:16.710949 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.710919 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 14 13:01:16.711300 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.711276 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 14 13:01:16.711516 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.711498 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 14 13:01:16.711516 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.711513 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 14 13:01:16.711648 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.711573 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 14 13:01:16.711648 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.711590 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-99ffz\"" Sep 14 13:01:16.711820 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.711805 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 14 13:01:16.716138 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.716116 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 14 13:01:16.724349 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.724328 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 13:01:16.796311 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.796267 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.796311 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.796309 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-web-config\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.796507 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.796332 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.796507 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.796370 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-config-volume\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.796507 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.796385 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74b140d0-ffc6-4819-b176-2e0204ef7397-tls-assets\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.796507 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.796408 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.796507 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.796432 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74b140d0-ffc6-4819-b176-2e0204ef7397-config-out\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.796507 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.796449 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.796507 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.796462 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/74b140d0-ffc6-4819-b176-2e0204ef7397-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.796507 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.796477 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74b140d0-ffc6-4819-b176-2e0204ef7397-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.796507 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.796497 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b140d0-ffc6-4819-b176-2e0204ef7397-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.796794 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.796549 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.796794 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.796584 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwh8d\" (UniqueName: \"kubernetes.io/projected/74b140d0-ffc6-4819-b176-2e0204ef7397-kube-api-access-jwh8d\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.897837 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.897801 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.897837 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.897838 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-web-config\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.898081 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.897865 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.898081 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.898001 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-config-volume\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.898081 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.898039 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74b140d0-ffc6-4819-b176-2e0204ef7397-tls-assets\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.898250 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.898087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.898250 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.898114 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74b140d0-ffc6-4819-b176-2e0204ef7397-config-out\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.898250 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.898153 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.898250 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.898180 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/74b140d0-ffc6-4819-b176-2e0204ef7397-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.898250 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.898207 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74b140d0-ffc6-4819-b176-2e0204ef7397-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.898250 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.898231 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b140d0-ffc6-4819-b176-2e0204ef7397-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.898530 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.898276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.898530 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.898348 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwh8d\" (UniqueName: \"kubernetes.io/projected/74b140d0-ffc6-4819-b176-2e0204ef7397-kube-api-access-jwh8d\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.899425 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.899085 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/74b140d0-ffc6-4819-b176-2e0204ef7397-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.901087 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.900801 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/74b140d0-ffc6-4819-b176-2e0204ef7397-config-out\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.901192 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.901103 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-web-config\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.901458 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.901334 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.901458 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.901393 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-config-volume\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.901711 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.901686 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/74b140d0-ffc6-4819-b176-2e0204ef7397-tls-assets\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.901711 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.901700 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.901989 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.901969 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74b140d0-ffc6-4819-b176-2e0204ef7397-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.902056 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.902000 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.902056 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.901971 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.902213 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.902194 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74b140d0-ffc6-4819-b176-2e0204ef7397-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.903053 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.903035 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/74b140d0-ffc6-4819-b176-2e0204ef7397-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:16.907035 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:16.907016 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwh8d\" (UniqueName: \"kubernetes.io/projected/74b140d0-ffc6-4819-b176-2e0204ef7397-kube-api-access-jwh8d\") pod \"alertmanager-main-0\" (UID: \"74b140d0-ffc6-4819-b176-2e0204ef7397\") " pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:17.017277 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:17.017191 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 14 13:01:17.307001 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:17.306908 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 14 13:01:17.310686 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:01:17.310661 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b140d0_ffc6_4819_b176_2e0204ef7397.slice/crio-2ff838b5e8f226314f347c182151cd9952fbdb8d68d3f4ffc270b68a73b1ee70 WatchSource:0}: Error finding container 2ff838b5e8f226314f347c182151cd9952fbdb8d68d3f4ffc270b68a73b1ee70: Status 404 returned error can't find the container with id 2ff838b5e8f226314f347c182151cd9952fbdb8d68d3f4ffc270b68a73b1ee70 Sep 14 13:01:17.656645 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:17.656610 2564 generic.go:356] "Generic (PLEG): container finished" podID="74b140d0-ffc6-4819-b176-2e0204ef7397" containerID="57c31011bb1e6298e183f0bbd88ed97871494c4e0eaaddadd6177550225545a1" exitCode=0 Sep 14 13:01:17.657096 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:17.656702 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74b140d0-ffc6-4819-b176-2e0204ef7397","Type":"ContainerDied","Data":"57c31011bb1e6298e183f0bbd88ed97871494c4e0eaaddadd6177550225545a1"} Sep 14 13:01:17.657096 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:17.656773 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74b140d0-ffc6-4819-b176-2e0204ef7397","Type":"ContainerStarted","Data":"2ff838b5e8f226314f347c182151cd9952fbdb8d68d3f4ffc270b68a73b1ee70"} Sep 14 13:01:17.814854 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:17.814824 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fde29993-cdbc-4f60-96f2-f2f6728bcb9a" path="/var/lib/kubelet/pods/fde29993-cdbc-4f60-96f2-f2f6728bcb9a/volumes" Sep 14 13:01:18.403404 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.403366 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-76b5648bd9-6bgpj"] Sep 14 13:01:18.406152 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.406128 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.408663 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.408642 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 14 13:01:18.408776 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.408664 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 14 13:01:18.408953 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.408915 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 14 13:01:18.409629 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.409610 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 14 13:01:18.409821 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.409804 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 14 13:01:18.410033 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.410007 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lgxg5\"" Sep 14 13:01:18.414533 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.414511 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 14 13:01:18.419403 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.419381 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76b5648bd9-6bgpj"] Sep 14 13:01:18.513121 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.513095 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/15c77fd6-d40c-44e3-be94-2ec048db6d17-federate-client-tls\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.513259 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.513133 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/15c77fd6-d40c-44e3-be94-2ec048db6d17-secret-telemeter-client\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.513259 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.513152 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/15c77fd6-d40c-44e3-be94-2ec048db6d17-telemeter-client-tls\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.513259 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.513241 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15c77fd6-d40c-44e3-be94-2ec048db6d17-metrics-client-ca\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.513389 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.513296 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15c77fd6-d40c-44e3-be94-2ec048db6d17-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.513389 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.513328 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15c77fd6-d40c-44e3-be94-2ec048db6d17-serving-certs-ca-bundle\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.513389 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.513365 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6b6z\" (UniqueName: \"kubernetes.io/projected/15c77fd6-d40c-44e3-be94-2ec048db6d17-kube-api-access-q6b6z\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.513482 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.513416 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15c77fd6-d40c-44e3-be94-2ec048db6d17-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.614657 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.614609 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6b6z\" (UniqueName: \"kubernetes.io/projected/15c77fd6-d40c-44e3-be94-2ec048db6d17-kube-api-access-q6b6z\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.614657 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.614668 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15c77fd6-d40c-44e3-be94-2ec048db6d17-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.614850 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.614695 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/15c77fd6-d40c-44e3-be94-2ec048db6d17-federate-client-tls\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.614850 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.614724 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/15c77fd6-d40c-44e3-be94-2ec048db6d17-secret-telemeter-client\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.614850 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.614744 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/15c77fd6-d40c-44e3-be94-2ec048db6d17-telemeter-client-tls\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.614850 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.614788 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15c77fd6-d40c-44e3-be94-2ec048db6d17-metrics-client-ca\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.614850 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.614816 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15c77fd6-d40c-44e3-be94-2ec048db6d17-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.615139 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.614858 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15c77fd6-d40c-44e3-be94-2ec048db6d17-serving-certs-ca-bundle\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.615842 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.615802 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15c77fd6-d40c-44e3-be94-2ec048db6d17-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.615983 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.615860 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15c77fd6-d40c-44e3-be94-2ec048db6d17-serving-certs-ca-bundle\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.615983 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.615902 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15c77fd6-d40c-44e3-be94-2ec048db6d17-metrics-client-ca\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.617518 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.617490 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/15c77fd6-d40c-44e3-be94-2ec048db6d17-federate-client-tls\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.617688 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.617668 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/15c77fd6-d40c-44e3-be94-2ec048db6d17-secret-telemeter-client\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.617982 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.617958 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15c77fd6-d40c-44e3-be94-2ec048db6d17-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.618019 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.618006 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/15c77fd6-d40c-44e3-be94-2ec048db6d17-telemeter-client-tls\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.622669 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.622647 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6b6z\" (UniqueName: \"kubernetes.io/projected/15c77fd6-d40c-44e3-be94-2ec048db6d17-kube-api-access-q6b6z\") pod \"telemeter-client-76b5648bd9-6bgpj\" (UID: \"15c77fd6-d40c-44e3-be94-2ec048db6d17\") " pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:18.665006 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.664907 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74b140d0-ffc6-4819-b176-2e0204ef7397","Type":"ContainerStarted","Data":"944dd1440af41a20ccb1330a5e6ca662dbf65ded900d0967732be0dfbe2929a5"} Sep 14 13:01:18.665006 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.664994 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74b140d0-ffc6-4819-b176-2e0204ef7397","Type":"ContainerStarted","Data":"4edbf9fd9ff990bc2e13786b83eebfc95b94c49c3b1e809835efaf62858eea49"} Sep 14 13:01:18.665392 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.665025 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74b140d0-ffc6-4819-b176-2e0204ef7397","Type":"ContainerStarted","Data":"189dcd074ac4e3f0623d2566d95af22fc3dcfca16b8287ce109cff86c7a7c7ff"} Sep 14 13:01:18.665392 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.665053 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74b140d0-ffc6-4819-b176-2e0204ef7397","Type":"ContainerStarted","Data":"25eb5a86564a6c64eaacdba26f018372bfd52418200ed8f39449eb9af0736bf4"} Sep 14 13:01:18.665392 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.665076 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74b140d0-ffc6-4819-b176-2e0204ef7397","Type":"ContainerStarted","Data":"4dd3803fc65ea66f9bea510747a75f7223efdddc117541f1bd17453cafffd474"} Sep 14 13:01:18.665392 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.665100 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"74b140d0-ffc6-4819-b176-2e0204ef7397","Type":"ContainerStarted","Data":"41edafb1da1e227bbebdcde0ea94f3e194248533f6048293151140a06ba005f3"} Sep 14 13:01:18.692429 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.692383 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.677742803 podStartE2EDuration="2.692367878s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.625075ms" isStatefulPod=false podCreationTimestamp="2026-09-14 13:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:01:18.690252427 +0000 UTC m=+179.353148091" watchObservedRunningTime="2026-09-14 13:01:18.692367878 +0000 UTC m=+179.355263507" Sep 14 13:01:18.718051 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:18.718023 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" Sep 14 13:01:19.002831 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:19.002656 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76b5648bd9-6bgpj"] Sep 14 13:01:19.005250 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:01:19.005214 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c77fd6_d40c_44e3_be94_2ec048db6d17.slice/crio-7bfe36f4e134170b9030b6e189a1811912702f9de3a8646365c719c03b3a08c6 WatchSource:0}: Error finding container 7bfe36f4e134170b9030b6e189a1811912702f9de3a8646365c719c03b3a08c6: Status 404 returned error can't find the container with id 7bfe36f4e134170b9030b6e189a1811912702f9de3a8646365c719c03b3a08c6 Sep 14 13:01:19.672231 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:19.672196 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" event={"ID":"15c77fd6-d40c-44e3-be94-2ec048db6d17","Type":"ContainerStarted","Data":"7bfe36f4e134170b9030b6e189a1811912702f9de3a8646365c719c03b3a08c6"} Sep 14 13:01:21.683899 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.683860 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" event={"ID":"15c77fd6-d40c-44e3-be94-2ec048db6d17","Type":"ContainerStarted","Data":"983756a8247668d080cfc475c77afc034cfa21cb2cdcb2898008d0583da1b4c2"} Sep 14 13:01:21.684290 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.683950 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" event={"ID":"15c77fd6-d40c-44e3-be94-2ec048db6d17","Type":"ContainerStarted","Data":"f6aa6db256dc44c39c78fa2561a16733f33ea2f9a2706ea5120d63cc1a61c94e"} Sep 14 13:01:21.684290 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.683979 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" event={"ID":"15c77fd6-d40c-44e3-be94-2ec048db6d17","Type":"ContainerStarted","Data":"7427c847ebc8dd4336ed01b0a58a1560fbbeb35d1a88b4a80148c05223881f04"} Sep 14 13:01:21.708437 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.708393 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-76b5648bd9-6bgpj" podStartSLOduration=1.8544724540000002 podStartE2EDuration="3.708378166s" totalImagesPullingTime="1.853905712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:01:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:01:21.707809829 +0000 UTC m=+182.370705494" watchObservedRunningTime="2026-09-14 13:01:21.708378166 +0000 UTC m=+182.371273796" Sep 14 13:01:21.792485 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.792453 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fbbf9bc44-zrgcb"] Sep 14 13:01:21.795124 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.795097 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.811430 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.811406 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fbbf9bc44-zrgcb"] Sep 14 13:01:21.842412 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.842301 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dea212b3-cff1-462b-b62d-72dcaf969f65-console-oauth-config\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.842412 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.842337 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-oauth-serving-cert\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.842412 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.842382 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knrmr\" (UniqueName: \"kubernetes.io/projected/dea212b3-cff1-462b-b62d-72dcaf969f65-kube-api-access-knrmr\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.842412 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.842480 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-service-ca\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.842804 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.842516 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dea212b3-cff1-462b-b62d-72dcaf969f65-console-serving-cert\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.842804 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.842544 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-console-config\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.842804 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.842568 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-trusted-ca-bundle\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.944119 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.944037 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-service-ca\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.944119 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.944072 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dea212b3-cff1-462b-b62d-72dcaf969f65-console-serving-cert\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.944119 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.944093 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-console-config\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.944119 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.944116 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-trusted-ca-bundle\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.944413 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.944160 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dea212b3-cff1-462b-b62d-72dcaf969f65-console-oauth-config\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.944413 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.944186 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-oauth-serving-cert\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.944413 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.944222 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knrmr\" (UniqueName: \"kubernetes.io/projected/dea212b3-cff1-462b-b62d-72dcaf969f65-kube-api-access-knrmr\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.944972 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.944914 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-service-ca\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.945098 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.944914 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-console-config\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.945098 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.945006 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-oauth-serving-cert\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.945203 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.945185 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-trusted-ca-bundle\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.947379 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.947352 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dea212b3-cff1-462b-b62d-72dcaf969f65-console-oauth-config\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.947470 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.947440 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dea212b3-cff1-462b-b62d-72dcaf969f65-console-serving-cert\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:21.953257 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:21.953237 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knrmr\" (UniqueName: \"kubernetes.io/projected/dea212b3-cff1-462b-b62d-72dcaf969f65-kube-api-access-knrmr\") pod \"console-7fbbf9bc44-zrgcb\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:22.106110 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:22.106058 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:22.385421 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:22.385397 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fbbf9bc44-zrgcb"] Sep 14 13:01:22.387810 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:01:22.387772 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea212b3_cff1_462b_b62d_72dcaf969f65.slice/crio-590110b9235b0b9e3ec5f6e128cdb2405624a36cd84bdb10f5071333d4ddf511 WatchSource:0}: Error finding container 590110b9235b0b9e3ec5f6e128cdb2405624a36cd84bdb10f5071333d4ddf511: Status 404 returned error can't find the container with id 590110b9235b0b9e3ec5f6e128cdb2405624a36cd84bdb10f5071333d4ddf511 Sep 14 13:01:22.690442 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:22.690344 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbbf9bc44-zrgcb" event={"ID":"dea212b3-cff1-462b-b62d-72dcaf969f65","Type":"ContainerStarted","Data":"5fc049ea0976fda04c77248d7ab8494f986d2b2a2ee2c573aa354161673672ea"} Sep 14 13:01:22.690442 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:22.690405 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbbf9bc44-zrgcb" event={"ID":"dea212b3-cff1-462b-b62d-72dcaf969f65","Type":"ContainerStarted","Data":"590110b9235b0b9e3ec5f6e128cdb2405624a36cd84bdb10f5071333d4ddf511"} Sep 14 13:01:22.711017 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:22.710966 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fbbf9bc44-zrgcb" podStartSLOduration=1.710950565 podStartE2EDuration="1.710950565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:01:22.709561055 +0000 UTC m=+183.372456686" watchObservedRunningTime="2026-09-14 13:01:22.710950565 +0000 UTC m=+183.373846191" Sep 14 13:01:32.106915 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:32.106877 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:32.107336 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:32.106976 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:32.111792 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:32.111771 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:32.733279 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:32.733249 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:01:32.781914 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:32.781881 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85c7f8f5b-r4ljw"] Sep 14 13:01:57.805593 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:57.805527 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85c7f8f5b-r4ljw" podUID="fb30aede-458d-4b29-9a2d-71868e936370" containerName="console" containerID="cri-o://29d96ba280609a2db785b0a07a87a2eb27aa089399b815bbbfe6c5f5ef9ec7de" gracePeriod=15 Sep 14 13:01:58.047429 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.047405 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 13:01:58.151595 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.151563 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-trusted-ca-bundle\") pod \"fb30aede-458d-4b29-9a2d-71868e936370\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " Sep 14 13:01:58.151765 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.151616 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-console-config\") pod \"fb30aede-458d-4b29-9a2d-71868e936370\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " Sep 14 13:01:58.151765 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.151648 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-service-ca\") pod \"fb30aede-458d-4b29-9a2d-71868e936370\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " Sep 14 13:01:58.151765 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.151666 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgt5f\" (UniqueName: \"kubernetes.io/projected/fb30aede-458d-4b29-9a2d-71868e936370-kube-api-access-kgt5f\") pod \"fb30aede-458d-4b29-9a2d-71868e936370\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " Sep 14 13:01:58.151765 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.151699 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb30aede-458d-4b29-9a2d-71868e936370-console-oauth-config\") pod \"fb30aede-458d-4b29-9a2d-71868e936370\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " Sep 14 13:01:58.151765 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.151752 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb30aede-458d-4b29-9a2d-71868e936370-console-serving-cert\") pod \"fb30aede-458d-4b29-9a2d-71868e936370\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " Sep 14 13:01:58.152028 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.151886 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-oauth-serving-cert\") pod \"fb30aede-458d-4b29-9a2d-71868e936370\" (UID: \"fb30aede-458d-4b29-9a2d-71868e936370\") " Sep 14 13:01:58.152086 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.152006 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fb30aede-458d-4b29-9a2d-71868e936370" (UID: "fb30aede-458d-4b29-9a2d-71868e936370"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:01:58.152156 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.152135 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-console-config" (OuterVolumeSpecName: "console-config") pod "fb30aede-458d-4b29-9a2d-71868e936370" (UID: "fb30aede-458d-4b29-9a2d-71868e936370"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:01:58.152211 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.152190 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-service-ca" (OuterVolumeSpecName: "service-ca") pod "fb30aede-458d-4b29-9a2d-71868e936370" (UID: "fb30aede-458d-4b29-9a2d-71868e936370"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:01:58.152321 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.152289 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fb30aede-458d-4b29-9a2d-71868e936370" (UID: "fb30aede-458d-4b29-9a2d-71868e936370"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:01:58.152455 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.152437 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-oauth-serving-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:01:58.152513 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.152461 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-trusted-ca-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:01:58.152513 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.152476 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-console-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:01:58.152513 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.152490 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fb30aede-458d-4b29-9a2d-71868e936370-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:01:58.154396 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.154366 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb30aede-458d-4b29-9a2d-71868e936370-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fb30aede-458d-4b29-9a2d-71868e936370" (UID: "fb30aede-458d-4b29-9a2d-71868e936370"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:01:58.154503 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.154400 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb30aede-458d-4b29-9a2d-71868e936370-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fb30aede-458d-4b29-9a2d-71868e936370" (UID: "fb30aede-458d-4b29-9a2d-71868e936370"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:01:58.154503 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.154488 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb30aede-458d-4b29-9a2d-71868e936370-kube-api-access-kgt5f" (OuterVolumeSpecName: "kube-api-access-kgt5f") pod "fb30aede-458d-4b29-9a2d-71868e936370" (UID: "fb30aede-458d-4b29-9a2d-71868e936370"). InnerVolumeSpecName "kube-api-access-kgt5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:01:58.253168 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.253135 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fb30aede-458d-4b29-9a2d-71868e936370-console-serving-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:01:58.253168 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.253158 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgt5f\" (UniqueName: \"kubernetes.io/projected/fb30aede-458d-4b29-9a2d-71868e936370-kube-api-access-kgt5f\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:01:58.253168 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.253168 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fb30aede-458d-4b29-9a2d-71868e936370-console-oauth-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:01:58.836737 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.836705 2564 generic.go:356] "Generic (PLEG): container finished" podID="fb30aede-458d-4b29-9a2d-71868e936370" containerID="29d96ba280609a2db785b0a07a87a2eb27aa089399b815bbbfe6c5f5ef9ec7de" exitCode=0 Sep 14 13:01:58.837232 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.836751 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c7f8f5b-r4ljw" Sep 14 13:01:58.837232 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.836792 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c7f8f5b-r4ljw" event={"ID":"fb30aede-458d-4b29-9a2d-71868e936370","Type":"ContainerDied","Data":"29d96ba280609a2db785b0a07a87a2eb27aa089399b815bbbfe6c5f5ef9ec7de"} Sep 14 13:01:58.837232 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.836837 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c7f8f5b-r4ljw" event={"ID":"fb30aede-458d-4b29-9a2d-71868e936370","Type":"ContainerDied","Data":"0c8b8ae1b37daeb734565e422f265e477b92e52e0e69c3a1792a2f38e0fb75f6"} Sep 14 13:01:58.837232 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.836858 2564 scope.go:117] "RemoveContainer" containerID="29d96ba280609a2db785b0a07a87a2eb27aa089399b815bbbfe6c5f5ef9ec7de" Sep 14 13:01:58.845504 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.845487 2564 scope.go:117] "RemoveContainer" containerID="29d96ba280609a2db785b0a07a87a2eb27aa089399b815bbbfe6c5f5ef9ec7de" Sep 14 13:01:58.845744 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:01:58.845727 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29d96ba280609a2db785b0a07a87a2eb27aa089399b815bbbfe6c5f5ef9ec7de\": container with ID starting with 29d96ba280609a2db785b0a07a87a2eb27aa089399b815bbbfe6c5f5ef9ec7de not found: ID does not exist" containerID="29d96ba280609a2db785b0a07a87a2eb27aa089399b815bbbfe6c5f5ef9ec7de" Sep 14 13:01:58.845791 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.845752 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29d96ba280609a2db785b0a07a87a2eb27aa089399b815bbbfe6c5f5ef9ec7de"} err="failed to get container status \"29d96ba280609a2db785b0a07a87a2eb27aa089399b815bbbfe6c5f5ef9ec7de\": rpc error: code = NotFound desc = could not find container \"29d96ba280609a2db785b0a07a87a2eb27aa089399b815bbbfe6c5f5ef9ec7de\": container with ID starting with 29d96ba280609a2db785b0a07a87a2eb27aa089399b815bbbfe6c5f5ef9ec7de not found: ID does not exist" Sep 14 13:01:58.862438 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.862413 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85c7f8f5b-r4ljw"] Sep 14 13:01:58.866309 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:58.866290 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85c7f8f5b-r4ljw"] Sep 14 13:01:59.810828 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:01:59.810793 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb30aede-458d-4b29-9a2d-71868e936370" path="/var/lib/kubelet/pods/fb30aede-458d-4b29-9a2d-71868e936370/volumes" Sep 14 13:02:33.183274 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.183240 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b96f7cd45-bzhkq"] Sep 14 13:02:33.183734 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.183563 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb30aede-458d-4b29-9a2d-71868e936370" containerName="console" Sep 14 13:02:33.183734 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.183574 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb30aede-458d-4b29-9a2d-71868e936370" containerName="console" Sep 14 13:02:33.183734 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.183627 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb30aede-458d-4b29-9a2d-71868e936370" containerName="console" Sep 14 13:02:33.188270 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.188248 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.202006 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.201981 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b96f7cd45-bzhkq"] Sep 14 13:02:33.333953 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.333896 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-console-config\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.333953 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.333958 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-service-ca\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.334172 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.333983 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-oauth-serving-cert\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.334172 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.334004 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckc4k\" (UniqueName: \"kubernetes.io/projected/fc632355-6859-488f-9cbf-e4a17f602cd9-kube-api-access-ckc4k\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.334172 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.334032 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-trusted-ca-bundle\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.334172 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.334144 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc632355-6859-488f-9cbf-e4a17f602cd9-console-serving-cert\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.334295 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.334185 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc632355-6859-488f-9cbf-e4a17f602cd9-console-oauth-config\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.434890 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.434802 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-console-config\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.434890 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.434836 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-service-ca\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.434890 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.434852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-oauth-serving-cert\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.434890 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.434869 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckc4k\" (UniqueName: \"kubernetes.io/projected/fc632355-6859-488f-9cbf-e4a17f602cd9-kube-api-access-ckc4k\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.435211 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.434976 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-trusted-ca-bundle\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.435211 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.435087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc632355-6859-488f-9cbf-e4a17f602cd9-console-serving-cert\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.435211 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.435134 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc632355-6859-488f-9cbf-e4a17f602cd9-console-oauth-config\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.435630 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.435600 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-service-ca\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.435630 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.435619 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-console-config\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.436103 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.436082 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-oauth-serving-cert\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.436482 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.436459 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-trusted-ca-bundle\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.441957 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.441919 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc632355-6859-488f-9cbf-e4a17f602cd9-console-oauth-config\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.442077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.442057 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc632355-6859-488f-9cbf-e4a17f602cd9-console-serving-cert\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.445182 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.445154 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckc4k\" (UniqueName: \"kubernetes.io/projected/fc632355-6859-488f-9cbf-e4a17f602cd9-kube-api-access-ckc4k\") pod \"console-5b96f7cd45-bzhkq\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.499262 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.499236 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:33.773351 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.773267 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b96f7cd45-bzhkq"] Sep 14 13:02:33.776666 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:02:33.776633 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc632355_6859_488f_9cbf_e4a17f602cd9.slice/crio-5a0f3986f07e71a6dccfe32962b7053c1e8232a8a83658bb523d2c8c9ecfb3c6 WatchSource:0}: Error finding container 5a0f3986f07e71a6dccfe32962b7053c1e8232a8a83658bb523d2c8c9ecfb3c6: Status 404 returned error can't find the container with id 5a0f3986f07e71a6dccfe32962b7053c1e8232a8a83658bb523d2c8c9ecfb3c6 Sep 14 13:02:33.975678 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.975644 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b96f7cd45-bzhkq" event={"ID":"fc632355-6859-488f-9cbf-e4a17f602cd9","Type":"ContainerStarted","Data":"7b3966e889be08ba0f374e3b072de25409d04683d35a795894596aa8ea704861"} Sep 14 13:02:33.975834 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.975694 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b96f7cd45-bzhkq" event={"ID":"fc632355-6859-488f-9cbf-e4a17f602cd9","Type":"ContainerStarted","Data":"5a0f3986f07e71a6dccfe32962b7053c1e8232a8a83658bb523d2c8c9ecfb3c6"} Sep 14 13:02:33.994634 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:33.994589 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b96f7cd45-bzhkq" podStartSLOduration=0.994574451 podStartE2EDuration="994.574451ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:02:33.993733007 +0000 UTC m=+254.656628636" watchObservedRunningTime="2026-09-14 13:02:33.994574451 +0000 UTC m=+254.657470082" Sep 14 13:02:43.499799 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:43.499708 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:43.500279 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:43.499806 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:43.504295 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:43.504277 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:44.020752 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:44.020720 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:02:44.074763 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:02:44.074720 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fbbf9bc44-zrgcb"] Sep 14 13:03:09.099651 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.099586 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7fbbf9bc44-zrgcb" podUID="dea212b3-cff1-462b-b62d-72dcaf969f65" containerName="console" containerID="cri-o://5fc049ea0976fda04c77248d7ab8494f986d2b2a2ee2c573aa354161673672ea" gracePeriod=15 Sep 14 13:03:09.337815 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.337789 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:03:09.435515 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.435479 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-service-ca\") pod \"dea212b3-cff1-462b-b62d-72dcaf969f65\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " Sep 14 13:03:09.435515 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.435520 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dea212b3-cff1-462b-b62d-72dcaf969f65-console-serving-cert\") pod \"dea212b3-cff1-462b-b62d-72dcaf969f65\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " Sep 14 13:03:09.435730 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.435604 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-oauth-serving-cert\") pod \"dea212b3-cff1-462b-b62d-72dcaf969f65\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " Sep 14 13:03:09.435730 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.435635 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-console-config\") pod \"dea212b3-cff1-462b-b62d-72dcaf969f65\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " Sep 14 13:03:09.435730 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.435693 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knrmr\" (UniqueName: \"kubernetes.io/projected/dea212b3-cff1-462b-b62d-72dcaf969f65-kube-api-access-knrmr\") pod \"dea212b3-cff1-462b-b62d-72dcaf969f65\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " Sep 14 13:03:09.435854 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.435743 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dea212b3-cff1-462b-b62d-72dcaf969f65-console-oauth-config\") pod \"dea212b3-cff1-462b-b62d-72dcaf969f65\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " Sep 14 13:03:09.435854 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.435788 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-trusted-ca-bundle\") pod \"dea212b3-cff1-462b-b62d-72dcaf969f65\" (UID: \"dea212b3-cff1-462b-b62d-72dcaf969f65\") " Sep 14 13:03:09.436077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.436030 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-service-ca" (OuterVolumeSpecName: "service-ca") pod "dea212b3-cff1-462b-b62d-72dcaf969f65" (UID: "dea212b3-cff1-462b-b62d-72dcaf969f65"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:03:09.436322 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.436077 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dea212b3-cff1-462b-b62d-72dcaf969f65" (UID: "dea212b3-cff1-462b-b62d-72dcaf969f65"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:03:09.436322 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.436114 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-console-config" (OuterVolumeSpecName: "console-config") pod "dea212b3-cff1-462b-b62d-72dcaf969f65" (UID: "dea212b3-cff1-462b-b62d-72dcaf969f65"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:03:09.436322 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.436302 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dea212b3-cff1-462b-b62d-72dcaf969f65" (UID: "dea212b3-cff1-462b-b62d-72dcaf969f65"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:03:09.437897 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.437875 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dea212b3-cff1-462b-b62d-72dcaf969f65-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dea212b3-cff1-462b-b62d-72dcaf969f65" (UID: "dea212b3-cff1-462b-b62d-72dcaf969f65"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:03:09.438053 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.438034 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dea212b3-cff1-462b-b62d-72dcaf969f65-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dea212b3-cff1-462b-b62d-72dcaf969f65" (UID: "dea212b3-cff1-462b-b62d-72dcaf969f65"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:03:09.438100 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.438077 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea212b3-cff1-462b-b62d-72dcaf969f65-kube-api-access-knrmr" (OuterVolumeSpecName: "kube-api-access-knrmr") pod "dea212b3-cff1-462b-b62d-72dcaf969f65" (UID: "dea212b3-cff1-462b-b62d-72dcaf969f65"). InnerVolumeSpecName "kube-api-access-knrmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:03:09.536554 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.536524 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knrmr\" (UniqueName: \"kubernetes.io/projected/dea212b3-cff1-462b-b62d-72dcaf969f65-kube-api-access-knrmr\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:03:09.536554 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.536551 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dea212b3-cff1-462b-b62d-72dcaf969f65-console-oauth-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:03:09.536554 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.536562 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-trusted-ca-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:03:09.536775 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.536572 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:03:09.536775 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.536580 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dea212b3-cff1-462b-b62d-72dcaf969f65-console-serving-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:03:09.536775 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.536587 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-oauth-serving-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:03:09.536775 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:09.536595 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dea212b3-cff1-462b-b62d-72dcaf969f65-console-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:03:10.122112 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:10.122075 2564 generic.go:356] "Generic (PLEG): container finished" podID="dea212b3-cff1-462b-b62d-72dcaf969f65" containerID="5fc049ea0976fda04c77248d7ab8494f986d2b2a2ee2c573aa354161673672ea" exitCode=0 Sep 14 13:03:10.122525 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:10.122130 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbbf9bc44-zrgcb" Sep 14 13:03:10.122525 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:10.122143 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbbf9bc44-zrgcb" event={"ID":"dea212b3-cff1-462b-b62d-72dcaf969f65","Type":"ContainerDied","Data":"5fc049ea0976fda04c77248d7ab8494f986d2b2a2ee2c573aa354161673672ea"} Sep 14 13:03:10.122525 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:10.122191 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbbf9bc44-zrgcb" event={"ID":"dea212b3-cff1-462b-b62d-72dcaf969f65","Type":"ContainerDied","Data":"590110b9235b0b9e3ec5f6e128cdb2405624a36cd84bdb10f5071333d4ddf511"} Sep 14 13:03:10.122525 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:10.122214 2564 scope.go:117] "RemoveContainer" containerID="5fc049ea0976fda04c77248d7ab8494f986d2b2a2ee2c573aa354161673672ea" Sep 14 13:03:10.130472 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:10.130454 2564 scope.go:117] "RemoveContainer" containerID="5fc049ea0976fda04c77248d7ab8494f986d2b2a2ee2c573aa354161673672ea" Sep 14 13:03:10.130705 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:03:10.130689 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fc049ea0976fda04c77248d7ab8494f986d2b2a2ee2c573aa354161673672ea\": container with ID starting with 5fc049ea0976fda04c77248d7ab8494f986d2b2a2ee2c573aa354161673672ea not found: ID does not exist" containerID="5fc049ea0976fda04c77248d7ab8494f986d2b2a2ee2c573aa354161673672ea" Sep 14 13:03:10.130765 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:10.130711 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fc049ea0976fda04c77248d7ab8494f986d2b2a2ee2c573aa354161673672ea"} err="failed to get container status \"5fc049ea0976fda04c77248d7ab8494f986d2b2a2ee2c573aa354161673672ea\": rpc error: code = NotFound desc = could not find container \"5fc049ea0976fda04c77248d7ab8494f986d2b2a2ee2c573aa354161673672ea\": container with ID starting with 5fc049ea0976fda04c77248d7ab8494f986d2b2a2ee2c573aa354161673672ea not found: ID does not exist" Sep 14 13:03:10.139531 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:10.139507 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fbbf9bc44-zrgcb"] Sep 14 13:03:10.145840 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:10.145820 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fbbf9bc44-zrgcb"] Sep 14 13:03:11.809866 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:11.809825 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dea212b3-cff1-462b-b62d-72dcaf969f65" path="/var/lib/kubelet/pods/dea212b3-cff1-462b-b62d-72dcaf969f65/volumes" Sep 14 13:03:19.757643 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:19.757613 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/1.log" Sep 14 13:03:19.761447 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:19.761420 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/1.log" Sep 14 13:03:19.770153 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:19.770135 2564 kubelet.go:1648] "Image garbage collection succeeded" Sep 14 13:03:53.360060 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.360030 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg"] Sep 14 13:03:53.362164 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.360333 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dea212b3-cff1-462b-b62d-72dcaf969f65" containerName="console" Sep 14 13:03:53.362164 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.360344 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="dea212b3-cff1-462b-b62d-72dcaf969f65" containerName="console" Sep 14 13:03:53.362164 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.360402 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="dea212b3-cff1-462b-b62d-72dcaf969f65" containerName="console" Sep 14 13:03:53.363228 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.363212 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg" Sep 14 13:03:53.366348 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.366328 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cq4js\"" Sep 14 13:03:53.366465 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.366446 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 14 13:03:53.367509 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.367493 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 14 13:03:53.379174 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.379154 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg"] Sep 14 13:03:53.393876 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.393855 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab826936-17e6-4c36-a34b-69fd3307316d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg\" (UID: \"ab826936-17e6-4c36-a34b-69fd3307316d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg" Sep 14 13:03:53.394004 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.393887 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab826936-17e6-4c36-a34b-69fd3307316d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg\" (UID: \"ab826936-17e6-4c36-a34b-69fd3307316d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg" Sep 14 13:03:53.394004 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.393914 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrnc9\" (UniqueName: \"kubernetes.io/projected/ab826936-17e6-4c36-a34b-69fd3307316d-kube-api-access-hrnc9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg\" (UID: \"ab826936-17e6-4c36-a34b-69fd3307316d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg" Sep 14 13:03:53.494470 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.494438 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab826936-17e6-4c36-a34b-69fd3307316d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg\" (UID: \"ab826936-17e6-4c36-a34b-69fd3307316d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg" Sep 14 13:03:53.494470 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.494474 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrnc9\" (UniqueName: \"kubernetes.io/projected/ab826936-17e6-4c36-a34b-69fd3307316d-kube-api-access-hrnc9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg\" (UID: \"ab826936-17e6-4c36-a34b-69fd3307316d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg" Sep 14 13:03:53.494660 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.494536 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab826936-17e6-4c36-a34b-69fd3307316d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg\" (UID: \"ab826936-17e6-4c36-a34b-69fd3307316d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg" Sep 14 13:03:53.494838 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.494811 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab826936-17e6-4c36-a34b-69fd3307316d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg\" (UID: \"ab826936-17e6-4c36-a34b-69fd3307316d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg" Sep 14 13:03:53.494838 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.494828 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab826936-17e6-4c36-a34b-69fd3307316d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg\" (UID: \"ab826936-17e6-4c36-a34b-69fd3307316d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg" Sep 14 13:03:53.505802 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.505768 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrnc9\" (UniqueName: \"kubernetes.io/projected/ab826936-17e6-4c36-a34b-69fd3307316d-kube-api-access-hrnc9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg\" (UID: \"ab826936-17e6-4c36-a34b-69fd3307316d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg" Sep 14 13:03:53.672297 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.672264 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg" Sep 14 13:03:53.944894 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.944870 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg"] Sep 14 13:03:53.947388 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:03:53.947355 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab826936_17e6_4c36_a34b_69fd3307316d.slice/crio-63bb6adfbb19eda1b0888565d6a567008b846487e909639f33577172f941dd5d WatchSource:0}: Error finding container 63bb6adfbb19eda1b0888565d6a567008b846487e909639f33577172f941dd5d: Status 404 returned error can't find the container with id 63bb6adfbb19eda1b0888565d6a567008b846487e909639f33577172f941dd5d Sep 14 13:03:53.949190 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:53.949175 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 13:03:54.302532 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:03:54.302453 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg" event={"ID":"ab826936-17e6-4c36-a34b-69fd3307316d","Type":"ContainerStarted","Data":"63bb6adfbb19eda1b0888565d6a567008b846487e909639f33577172f941dd5d"} Sep 14 13:04:00.333613 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:00.333577 2564 generic.go:356] "Generic (PLEG): container finished" podID="ab826936-17e6-4c36-a34b-69fd3307316d" containerID="01aa01897d5865fed57b6ce2575e93ec063d090ed491f5066e4dca3f9eeb04ff" exitCode=0 Sep 14 13:04:00.334028 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:00.333649 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg" event={"ID":"ab826936-17e6-4c36-a34b-69fd3307316d","Type":"ContainerDied","Data":"01aa01897d5865fed57b6ce2575e93ec063d090ed491f5066e4dca3f9eeb04ff"} Sep 14 13:04:03.348743 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:03.348705 2564 generic.go:356] "Generic (PLEG): container finished" podID="ab826936-17e6-4c36-a34b-69fd3307316d" containerID="51d3f39375ad07664fbd50e174f0cc0c54f80ba90bcafa37ad26c0d6238b2778" exitCode=0 Sep 14 13:04:03.348743 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:03.348750 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg" event={"ID":"ab826936-17e6-4c36-a34b-69fd3307316d","Type":"ContainerDied","Data":"51d3f39375ad07664fbd50e174f0cc0c54f80ba90bcafa37ad26c0d6238b2778"} Sep 14 13:04:10.382678 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:10.382636 2564 generic.go:356] "Generic (PLEG): container finished" podID="ab826936-17e6-4c36-a34b-69fd3307316d" containerID="7d1c59a2bde14521c6224abeae95615ab463225a84e6cd0683181258309c22b5" exitCode=0 Sep 14 13:04:10.383060 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:10.382722 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg" event={"ID":"ab826936-17e6-4c36-a34b-69fd3307316d","Type":"ContainerDied","Data":"7d1c59a2bde14521c6224abeae95615ab463225a84e6cd0683181258309c22b5"} Sep 14 13:04:11.511434 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:11.511411 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg" Sep 14 13:04:11.556628 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:11.556596 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab826936-17e6-4c36-a34b-69fd3307316d-bundle\") pod \"ab826936-17e6-4c36-a34b-69fd3307316d\" (UID: \"ab826936-17e6-4c36-a34b-69fd3307316d\") " Sep 14 13:04:11.556802 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:11.556638 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrnc9\" (UniqueName: \"kubernetes.io/projected/ab826936-17e6-4c36-a34b-69fd3307316d-kube-api-access-hrnc9\") pod \"ab826936-17e6-4c36-a34b-69fd3307316d\" (UID: \"ab826936-17e6-4c36-a34b-69fd3307316d\") " Sep 14 13:04:11.556802 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:11.556670 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab826936-17e6-4c36-a34b-69fd3307316d-util\") pod \"ab826936-17e6-4c36-a34b-69fd3307316d\" (UID: \"ab826936-17e6-4c36-a34b-69fd3307316d\") " Sep 14 13:04:11.557328 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:11.557299 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab826936-17e6-4c36-a34b-69fd3307316d-bundle" (OuterVolumeSpecName: "bundle") pod "ab826936-17e6-4c36-a34b-69fd3307316d" (UID: "ab826936-17e6-4c36-a34b-69fd3307316d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:04:11.559085 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:11.559059 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab826936-17e6-4c36-a34b-69fd3307316d-kube-api-access-hrnc9" (OuterVolumeSpecName: "kube-api-access-hrnc9") pod "ab826936-17e6-4c36-a34b-69fd3307316d" (UID: "ab826936-17e6-4c36-a34b-69fd3307316d"). InnerVolumeSpecName "kube-api-access-hrnc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:04:11.561158 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:11.561135 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab826936-17e6-4c36-a34b-69fd3307316d-util" (OuterVolumeSpecName: "util") pod "ab826936-17e6-4c36-a34b-69fd3307316d" (UID: "ab826936-17e6-4c36-a34b-69fd3307316d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:04:11.657710 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:11.657636 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab826936-17e6-4c36-a34b-69fd3307316d-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:04:11.657710 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:11.657662 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrnc9\" (UniqueName: \"kubernetes.io/projected/ab826936-17e6-4c36-a34b-69fd3307316d-kube-api-access-hrnc9\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:04:11.657710 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:11.657672 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab826936-17e6-4c36-a34b-69fd3307316d-util\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:04:12.392518 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:12.392487 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg" Sep 14 13:04:12.392687 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:12.392493 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540485qg" event={"ID":"ab826936-17e6-4c36-a34b-69fd3307316d","Type":"ContainerDied","Data":"63bb6adfbb19eda1b0888565d6a567008b846487e909639f33577172f941dd5d"} Sep 14 13:04:12.392687 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:12.392602 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63bb6adfbb19eda1b0888565d6a567008b846487e909639f33577172f941dd5d" Sep 14 13:04:16.341217 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.341185 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m75ks"] Sep 14 13:04:16.341590 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.341491 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab826936-17e6-4c36-a34b-69fd3307316d" containerName="util" Sep 14 13:04:16.341590 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.341502 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab826936-17e6-4c36-a34b-69fd3307316d" containerName="util" Sep 14 13:04:16.341590 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.341521 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab826936-17e6-4c36-a34b-69fd3307316d" containerName="pull" Sep 14 13:04:16.341590 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.341527 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab826936-17e6-4c36-a34b-69fd3307316d" containerName="pull" Sep 14 13:04:16.341590 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.341536 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab826936-17e6-4c36-a34b-69fd3307316d" containerName="extract" Sep 14 13:04:16.341590 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.341541 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab826936-17e6-4c36-a34b-69fd3307316d" containerName="extract" Sep 14 13:04:16.341590 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.341592 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab826936-17e6-4c36-a34b-69fd3307316d" containerName="extract" Sep 14 13:04:16.375749 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.375721 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m75ks"] Sep 14 13:04:16.375898 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.375826 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m75ks" Sep 14 13:04:16.379025 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.379005 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 14 13:04:16.379283 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.379264 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-5j9qg\"" Sep 14 13:04:16.379369 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.379318 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 14 13:04:16.497768 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.497736 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blwpn\" (UniqueName: \"kubernetes.io/projected/3421e258-bde0-4a37-ae18-94d2f11ae78e-kube-api-access-blwpn\") pod \"cert-manager-operator-controller-manager-55c444bf86-m75ks\" (UID: \"3421e258-bde0-4a37-ae18-94d2f11ae78e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m75ks" Sep 14 13:04:16.497962 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.497782 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3421e258-bde0-4a37-ae18-94d2f11ae78e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-m75ks\" (UID: \"3421e258-bde0-4a37-ae18-94d2f11ae78e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m75ks" Sep 14 13:04:16.598296 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.598210 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blwpn\" (UniqueName: \"kubernetes.io/projected/3421e258-bde0-4a37-ae18-94d2f11ae78e-kube-api-access-blwpn\") pod \"cert-manager-operator-controller-manager-55c444bf86-m75ks\" (UID: \"3421e258-bde0-4a37-ae18-94d2f11ae78e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m75ks" Sep 14 13:04:16.598296 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.598266 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3421e258-bde0-4a37-ae18-94d2f11ae78e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-m75ks\" (UID: \"3421e258-bde0-4a37-ae18-94d2f11ae78e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m75ks" Sep 14 13:04:16.598600 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.598585 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3421e258-bde0-4a37-ae18-94d2f11ae78e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-m75ks\" (UID: \"3421e258-bde0-4a37-ae18-94d2f11ae78e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m75ks" Sep 14 13:04:16.610070 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.610041 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blwpn\" (UniqueName: \"kubernetes.io/projected/3421e258-bde0-4a37-ae18-94d2f11ae78e-kube-api-access-blwpn\") pod \"cert-manager-operator-controller-manager-55c444bf86-m75ks\" (UID: \"3421e258-bde0-4a37-ae18-94d2f11ae78e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m75ks" Sep 14 13:04:16.685298 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.685267 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m75ks" Sep 14 13:04:16.983195 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:16.983098 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m75ks"] Sep 14 13:04:16.985627 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:04:16.985596 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3421e258_bde0_4a37_ae18_94d2f11ae78e.slice/crio-4e7276427632d2f449ba1ba4b3f52408cf170fc2a0ac3eb03dde3abf0ff59681 WatchSource:0}: Error finding container 4e7276427632d2f449ba1ba4b3f52408cf170fc2a0ac3eb03dde3abf0ff59681: Status 404 returned error can't find the container with id 4e7276427632d2f449ba1ba4b3f52408cf170fc2a0ac3eb03dde3abf0ff59681 Sep 14 13:04:17.415484 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:17.415449 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m75ks" event={"ID":"3421e258-bde0-4a37-ae18-94d2f11ae78e","Type":"ContainerStarted","Data":"4e7276427632d2f449ba1ba4b3f52408cf170fc2a0ac3eb03dde3abf0ff59681"} Sep 14 13:04:19.426111 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:19.426078 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m75ks" event={"ID":"3421e258-bde0-4a37-ae18-94d2f11ae78e","Type":"ContainerStarted","Data":"16ca395ec469d2d62a432461002d43d96143b78573a9df9cb3a42f395ed4ead5"} Sep 14 13:04:19.453543 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:19.453475 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-m75ks" podStartSLOduration=1.172992845 podStartE2EDuration="3.453455267s" totalImagesPullingTime="2.280462422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:04:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:04:19.450142775 +0000 UTC m=+360.113038396" watchObservedRunningTime="2026-09-14 13:04:19.453455267 +0000 UTC m=+360.116350899" Sep 14 13:04:20.171799 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:20.171759 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm"] Sep 14 13:04:20.189668 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:20.189637 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm" Sep 14 13:04:20.192414 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:20.192330 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cq4js\"" Sep 14 13:04:20.192559 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:20.192503 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm"] Sep 14 13:04:20.192632 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:20.192561 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 14 13:04:20.193512 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:20.193494 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 14 13:04:20.327802 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:20.327769 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6a68aa5-3472-4d83-8787-605c8fd6e1e1-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm\" (UID: \"c6a68aa5-3472-4d83-8787-605c8fd6e1e1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm" Sep 14 13:04:20.328012 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:20.327825 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6a68aa5-3472-4d83-8787-605c8fd6e1e1-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm\" (UID: \"c6a68aa5-3472-4d83-8787-605c8fd6e1e1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm" Sep 14 13:04:20.328012 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:20.327846 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6tjm\" (UniqueName: \"kubernetes.io/projected/c6a68aa5-3472-4d83-8787-605c8fd6e1e1-kube-api-access-c6tjm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm\" (UID: \"c6a68aa5-3472-4d83-8787-605c8fd6e1e1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm" Sep 14 13:04:20.428406 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:20.428323 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6a68aa5-3472-4d83-8787-605c8fd6e1e1-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm\" (UID: \"c6a68aa5-3472-4d83-8787-605c8fd6e1e1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm" Sep 14 13:04:20.428406 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:20.428360 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6tjm\" (UniqueName: \"kubernetes.io/projected/c6a68aa5-3472-4d83-8787-605c8fd6e1e1-kube-api-access-c6tjm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm\" (UID: \"c6a68aa5-3472-4d83-8787-605c8fd6e1e1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm" Sep 14 13:04:20.428894 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:20.428433 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6a68aa5-3472-4d83-8787-605c8fd6e1e1-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm\" (UID: \"c6a68aa5-3472-4d83-8787-605c8fd6e1e1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm" Sep 14 13:04:20.428894 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:20.428762 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6a68aa5-3472-4d83-8787-605c8fd6e1e1-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm\" (UID: \"c6a68aa5-3472-4d83-8787-605c8fd6e1e1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm" Sep 14 13:04:20.428894 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:20.428811 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6a68aa5-3472-4d83-8787-605c8fd6e1e1-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm\" (UID: \"c6a68aa5-3472-4d83-8787-605c8fd6e1e1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm" Sep 14 13:04:20.447042 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:20.447012 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6tjm\" (UniqueName: \"kubernetes.io/projected/c6a68aa5-3472-4d83-8787-605c8fd6e1e1-kube-api-access-c6tjm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm\" (UID: \"c6a68aa5-3472-4d83-8787-605c8fd6e1e1\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm" Sep 14 13:04:20.501088 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:20.501057 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm" Sep 14 13:04:20.798119 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:20.798064 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm"] Sep 14 13:04:20.800274 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:04:20.800245 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a68aa5_3472_4d83_8787_605c8fd6e1e1.slice/crio-dd34e9adf7a436fd33659a511e31145e3e7ffab104b523dee358e92341f80695 WatchSource:0}: Error finding container dd34e9adf7a436fd33659a511e31145e3e7ffab104b523dee358e92341f80695: Status 404 returned error can't find the container with id dd34e9adf7a436fd33659a511e31145e3e7ffab104b523dee358e92341f80695 Sep 14 13:04:21.437344 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:21.437302 2564 generic.go:356] "Generic (PLEG): container finished" podID="c6a68aa5-3472-4d83-8787-605c8fd6e1e1" containerID="6fae77d0d6e9fb3fc18f04fc38a854968445eaaf12b385c58b96bd08a8819b99" exitCode=0 Sep 14 13:04:21.437732 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:21.437354 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm" event={"ID":"c6a68aa5-3472-4d83-8787-605c8fd6e1e1","Type":"ContainerDied","Data":"6fae77d0d6e9fb3fc18f04fc38a854968445eaaf12b385c58b96bd08a8819b99"} Sep 14 13:04:21.437732 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:21.437391 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm" event={"ID":"c6a68aa5-3472-4d83-8787-605c8fd6e1e1","Type":"ContainerStarted","Data":"dd34e9adf7a436fd33659a511e31145e3e7ffab104b523dee358e92341f80695"} Sep 14 13:04:23.448164 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:23.448128 2564 generic.go:356] "Generic (PLEG): container finished" podID="c6a68aa5-3472-4d83-8787-605c8fd6e1e1" containerID="b71f8834b0b0f1ca6751112270dc4c1ee91f27aea1fd9c69a407d4da07ac2018" exitCode=0 Sep 14 13:04:23.448632 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:23.448175 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm" event={"ID":"c6a68aa5-3472-4d83-8787-605c8fd6e1e1","Type":"ContainerDied","Data":"b71f8834b0b0f1ca6751112270dc4c1ee91f27aea1fd9c69a407d4da07ac2018"} Sep 14 13:04:24.454800 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:24.454769 2564 generic.go:356] "Generic (PLEG): container finished" podID="c6a68aa5-3472-4d83-8787-605c8fd6e1e1" containerID="339e4af36e412e99dab06bb6ec395e653ecee10cc831bdca9c4d868d061210ac" exitCode=0 Sep 14 13:04:24.455253 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:24.454857 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm" event={"ID":"c6a68aa5-3472-4d83-8787-605c8fd6e1e1","Type":"ContainerDied","Data":"339e4af36e412e99dab06bb6ec395e653ecee10cc831bdca9c4d868d061210ac"} Sep 14 13:04:25.582385 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:25.582356 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm" Sep 14 13:04:25.676643 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:25.676602 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6a68aa5-3472-4d83-8787-605c8fd6e1e1-util\") pod \"c6a68aa5-3472-4d83-8787-605c8fd6e1e1\" (UID: \"c6a68aa5-3472-4d83-8787-605c8fd6e1e1\") " Sep 14 13:04:25.676838 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:25.676709 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6a68aa5-3472-4d83-8787-605c8fd6e1e1-bundle\") pod \"c6a68aa5-3472-4d83-8787-605c8fd6e1e1\" (UID: \"c6a68aa5-3472-4d83-8787-605c8fd6e1e1\") " Sep 14 13:04:25.676838 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:25.676777 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6tjm\" (UniqueName: \"kubernetes.io/projected/c6a68aa5-3472-4d83-8787-605c8fd6e1e1-kube-api-access-c6tjm\") pod \"c6a68aa5-3472-4d83-8787-605c8fd6e1e1\" (UID: \"c6a68aa5-3472-4d83-8787-605c8fd6e1e1\") " Sep 14 13:04:25.677103 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:25.677073 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a68aa5-3472-4d83-8787-605c8fd6e1e1-bundle" (OuterVolumeSpecName: "bundle") pod "c6a68aa5-3472-4d83-8787-605c8fd6e1e1" (UID: "c6a68aa5-3472-4d83-8787-605c8fd6e1e1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:04:25.678961 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:25.678937 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a68aa5-3472-4d83-8787-605c8fd6e1e1-kube-api-access-c6tjm" (OuterVolumeSpecName: "kube-api-access-c6tjm") pod "c6a68aa5-3472-4d83-8787-605c8fd6e1e1" (UID: "c6a68aa5-3472-4d83-8787-605c8fd6e1e1"). InnerVolumeSpecName "kube-api-access-c6tjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:04:25.684734 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:25.684709 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a68aa5-3472-4d83-8787-605c8fd6e1e1-util" (OuterVolumeSpecName: "util") pod "c6a68aa5-3472-4d83-8787-605c8fd6e1e1" (UID: "c6a68aa5-3472-4d83-8787-605c8fd6e1e1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:04:25.778143 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:25.778058 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c6a68aa5-3472-4d83-8787-605c8fd6e1e1-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:04:25.778143 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:25.778087 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6tjm\" (UniqueName: \"kubernetes.io/projected/c6a68aa5-3472-4d83-8787-605c8fd6e1e1-kube-api-access-c6tjm\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:04:25.778143 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:25.778100 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c6a68aa5-3472-4d83-8787-605c8fd6e1e1-util\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:04:26.464854 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:26.464827 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm" Sep 14 13:04:26.464854 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:26.464833 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f6nsvm" event={"ID":"c6a68aa5-3472-4d83-8787-605c8fd6e1e1","Type":"ContainerDied","Data":"dd34e9adf7a436fd33659a511e31145e3e7ffab104b523dee358e92341f80695"} Sep 14 13:04:26.465082 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:26.464875 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd34e9adf7a436fd33659a511e31145e3e7ffab104b523dee358e92341f80695" Sep 14 13:04:27.450856 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:27.450826 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pjt6c"] Sep 14 13:04:27.451236 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:27.451178 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a68aa5-3472-4d83-8787-605c8fd6e1e1" containerName="pull" Sep 14 13:04:27.451236 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:27.451190 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a68aa5-3472-4d83-8787-605c8fd6e1e1" containerName="pull" Sep 14 13:04:27.451236 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:27.451201 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a68aa5-3472-4d83-8787-605c8fd6e1e1" containerName="util" Sep 14 13:04:27.451236 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:27.451206 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a68aa5-3472-4d83-8787-605c8fd6e1e1" containerName="util" Sep 14 13:04:27.451236 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:27.451225 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a68aa5-3472-4d83-8787-605c8fd6e1e1" containerName="extract" Sep 14 13:04:27.451236 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:27.451230 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a68aa5-3472-4d83-8787-605c8fd6e1e1" containerName="extract" Sep 14 13:04:27.451476 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:27.451282 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a68aa5-3472-4d83-8787-605c8fd6e1e1" containerName="extract" Sep 14 13:04:27.453828 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:27.453811 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pjt6c" Sep 14 13:04:27.456280 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:27.456253 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 14 13:04:27.456280 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:27.456275 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 14 13:04:27.457330 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:27.457309 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-764kg\"" Sep 14 13:04:27.462909 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:27.462890 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pjt6c"] Sep 14 13:04:27.591840 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:27.591807 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cljjg\" (UniqueName: \"kubernetes.io/projected/e58ce889-5584-4c38-9300-6c71f4fbd63e-kube-api-access-cljjg\") pod \"cert-manager-cainjector-559cdd8c87-pjt6c\" (UID: \"e58ce889-5584-4c38-9300-6c71f4fbd63e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pjt6c" Sep 14 13:04:27.591840 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:27.591839 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e58ce889-5584-4c38-9300-6c71f4fbd63e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pjt6c\" (UID: \"e58ce889-5584-4c38-9300-6c71f4fbd63e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pjt6c" Sep 14 13:04:27.692833 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:27.692804 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cljjg\" (UniqueName: \"kubernetes.io/projected/e58ce889-5584-4c38-9300-6c71f4fbd63e-kube-api-access-cljjg\") pod \"cert-manager-cainjector-559cdd8c87-pjt6c\" (UID: \"e58ce889-5584-4c38-9300-6c71f4fbd63e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pjt6c" Sep 14 13:04:27.692833 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:27.692833 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e58ce889-5584-4c38-9300-6c71f4fbd63e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pjt6c\" (UID: \"e58ce889-5584-4c38-9300-6c71f4fbd63e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pjt6c" Sep 14 13:04:27.702247 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:27.702187 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e58ce889-5584-4c38-9300-6c71f4fbd63e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-pjt6c\" (UID: \"e58ce889-5584-4c38-9300-6c71f4fbd63e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pjt6c" Sep 14 13:04:27.702382 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:27.702362 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cljjg\" (UniqueName: \"kubernetes.io/projected/e58ce889-5584-4c38-9300-6c71f4fbd63e-kube-api-access-cljjg\") pod \"cert-manager-cainjector-559cdd8c87-pjt6c\" (UID: \"e58ce889-5584-4c38-9300-6c71f4fbd63e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-pjt6c" Sep 14 13:04:27.773239 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:27.773216 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pjt6c" Sep 14 13:04:28.046564 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:28.046537 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-pjt6c"] Sep 14 13:04:28.049030 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:04:28.048988 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode58ce889_5584_4c38_9300_6c71f4fbd63e.slice/crio-51a8c58b5514ae49bd175db41879b5d6d45b7c53c51e4f8104c0a76ef3bfcd9b WatchSource:0}: Error finding container 51a8c58b5514ae49bd175db41879b5d6d45b7c53c51e4f8104c0a76ef3bfcd9b: Status 404 returned error can't find the container with id 51a8c58b5514ae49bd175db41879b5d6d45b7c53c51e4f8104c0a76ef3bfcd9b Sep 14 13:04:28.475033 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:28.475001 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pjt6c" event={"ID":"e58ce889-5584-4c38-9300-6c71f4fbd63e","Type":"ContainerStarted","Data":"51a8c58b5514ae49bd175db41879b5d6d45b7c53c51e4f8104c0a76ef3bfcd9b"} Sep 14 13:04:30.485439 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:30.485406 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pjt6c" event={"ID":"e58ce889-5584-4c38-9300-6c71f4fbd63e","Type":"ContainerStarted","Data":"a8758705768bfd3dcc4d41290ab0b86ba0119b2aa5b2929cdf63620c525997f5"} Sep 14 13:04:30.504293 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:30.504245 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-pjt6c" podStartSLOduration=1.1428046219999999 podStartE2EDuration="3.504228889s" totalImagesPullingTime="2.361424267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:04:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:04:30.503297305 +0000 UTC m=+371.166192934" watchObservedRunningTime="2026-09-14 13:04:30.504228889 +0000 UTC m=+371.167124520" Sep 14 13:04:31.216360 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:31.216316 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jsg5r"] Sep 14 13:04:31.218975 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:31.218951 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jsg5r" Sep 14 13:04:31.221658 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:31.221635 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-dghqm\"" Sep 14 13:04:31.229168 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:31.229146 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jsg5r"] Sep 14 13:04:31.326744 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:31.326713 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8835d2cf-8a5a-4270-9b5f-f1043511f5fb-bound-sa-token\") pod \"cert-manager-55b4d954d5-jsg5r\" (UID: \"8835d2cf-8a5a-4270-9b5f-f1043511f5fb\") " pod="cert-manager/cert-manager-55b4d954d5-jsg5r" Sep 14 13:04:31.326917 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:31.326769 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wrlx\" (UniqueName: \"kubernetes.io/projected/8835d2cf-8a5a-4270-9b5f-f1043511f5fb-kube-api-access-7wrlx\") pod \"cert-manager-55b4d954d5-jsg5r\" (UID: \"8835d2cf-8a5a-4270-9b5f-f1043511f5fb\") " pod="cert-manager/cert-manager-55b4d954d5-jsg5r" Sep 14 13:04:31.428181 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:31.428144 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wrlx\" (UniqueName: \"kubernetes.io/projected/8835d2cf-8a5a-4270-9b5f-f1043511f5fb-kube-api-access-7wrlx\") pod \"cert-manager-55b4d954d5-jsg5r\" (UID: \"8835d2cf-8a5a-4270-9b5f-f1043511f5fb\") " pod="cert-manager/cert-manager-55b4d954d5-jsg5r" Sep 14 13:04:31.428344 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:31.428217 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8835d2cf-8a5a-4270-9b5f-f1043511f5fb-bound-sa-token\") pod \"cert-manager-55b4d954d5-jsg5r\" (UID: \"8835d2cf-8a5a-4270-9b5f-f1043511f5fb\") " pod="cert-manager/cert-manager-55b4d954d5-jsg5r" Sep 14 13:04:31.437069 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:31.437037 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wrlx\" (UniqueName: \"kubernetes.io/projected/8835d2cf-8a5a-4270-9b5f-f1043511f5fb-kube-api-access-7wrlx\") pod \"cert-manager-55b4d954d5-jsg5r\" (UID: \"8835d2cf-8a5a-4270-9b5f-f1043511f5fb\") " pod="cert-manager/cert-manager-55b4d954d5-jsg5r" Sep 14 13:04:31.437069 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:31.437057 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8835d2cf-8a5a-4270-9b5f-f1043511f5fb-bound-sa-token\") pod \"cert-manager-55b4d954d5-jsg5r\" (UID: \"8835d2cf-8a5a-4270-9b5f-f1043511f5fb\") " pod="cert-manager/cert-manager-55b4d954d5-jsg5r" Sep 14 13:04:31.527976 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:31.527877 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-jsg5r" Sep 14 13:04:31.799278 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:31.799198 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-jsg5r"] Sep 14 13:04:31.802496 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:04:31.802457 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8835d2cf_8a5a_4270_9b5f_f1043511f5fb.slice/crio-93b59cbb1756e1366e2f71404aa5f8737fd34f766944c3ad97bb9e46f8c63baf WatchSource:0}: Error finding container 93b59cbb1756e1366e2f71404aa5f8737fd34f766944c3ad97bb9e46f8c63baf: Status 404 returned error can't find the container with id 93b59cbb1756e1366e2f71404aa5f8737fd34f766944c3ad97bb9e46f8c63baf Sep 14 13:04:32.495991 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:32.495956 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jsg5r" event={"ID":"8835d2cf-8a5a-4270-9b5f-f1043511f5fb","Type":"ContainerStarted","Data":"9d82f116f8246826db8e12ab04c17c044a213f704fbe32579b0e85be345721e2"} Sep 14 13:04:32.496166 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:32.496000 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-jsg5r" event={"ID":"8835d2cf-8a5a-4270-9b5f-f1043511f5fb","Type":"ContainerStarted","Data":"93b59cbb1756e1366e2f71404aa5f8737fd34f766944c3ad97bb9e46f8c63baf"} Sep 14 13:04:32.514864 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:32.514812 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-jsg5r" podStartSLOduration=1.514798378 podStartE2EDuration="1.514798378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:04:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:04:32.512914128 +0000 UTC m=+373.175809759" watchObservedRunningTime="2026-09-14 13:04:32.514798378 +0000 UTC m=+373.177694006" Sep 14 13:04:39.812031 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:39.812002 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fcc7ccbfd-ktknn"] Sep 14 13:04:39.814966 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:39.814911 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:39.819663 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:39.819636 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fcc7ccbfd-ktknn"] Sep 14 13:04:39.904308 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:39.904271 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-console-serving-cert\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:39.904492 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:39.904315 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-console-config\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:39.904610 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:39.904592 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-service-ca\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:39.904688 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:39.904655 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-console-oauth-config\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:39.904745 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:39.904701 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-trusted-ca-bundle\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:39.904745 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:39.904733 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-oauth-serving-cert\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:39.904842 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:39.904793 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjk2n\" (UniqueName: \"kubernetes.io/projected/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-kube-api-access-cjk2n\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:40.005886 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.005848 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-console-oauth-config\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:40.005886 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.005890 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-trusted-ca-bundle\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:40.006130 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.005911 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-oauth-serving-cert\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:40.006130 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.005959 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjk2n\" (UniqueName: \"kubernetes.io/projected/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-kube-api-access-cjk2n\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:40.006130 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.005989 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-console-serving-cert\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:40.006130 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.006024 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-console-config\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:40.006130 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.006086 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-service-ca\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:40.006947 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.006891 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-trusted-ca-bundle\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:40.007069 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.006966 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-console-config\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:40.007069 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.006980 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-service-ca\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:40.007069 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.007000 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-oauth-serving-cert\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:40.008606 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.008587 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-console-oauth-config\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:40.008691 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.008675 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-console-serving-cert\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:40.014606 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.014589 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjk2n\" (UniqueName: \"kubernetes.io/projected/82e2709e-ab9a-4d23-8ade-39babbfbc8aa-kube-api-access-cjk2n\") pod \"console-5fcc7ccbfd-ktknn\" (UID: \"82e2709e-ab9a-4d23-8ade-39babbfbc8aa\") " pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:40.126746 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.126712 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:40.397854 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.397829 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fcc7ccbfd-ktknn"] Sep 14 13:04:40.400113 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:04:40.400082 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82e2709e_ab9a_4d23_8ade_39babbfbc8aa.slice/crio-b4ef66c3785681e0dde30af81d23035f7d6a420ac3ce8feef2b50098b360e6c7 WatchSource:0}: Error finding container b4ef66c3785681e0dde30af81d23035f7d6a420ac3ce8feef2b50098b360e6c7: Status 404 returned error can't find the container with id b4ef66c3785681e0dde30af81d23035f7d6a420ac3ce8feef2b50098b360e6c7 Sep 14 13:04:40.436796 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.436763 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4"] Sep 14 13:04:40.446441 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.446416 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4" Sep 14 13:04:40.449401 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.449301 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 14 13:04:40.449555 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.449528 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4"] Sep 14 13:04:40.449680 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.449656 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cq4js\"" Sep 14 13:04:40.449810 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.449552 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 14 13:04:40.510555 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.510519 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qzhl\" (UniqueName: \"kubernetes.io/projected/63463820-fde2-42dc-ad83-f76da8cd2465-kube-api-access-6qzhl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4\" (UID: \"63463820-fde2-42dc-ad83-f76da8cd2465\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4" Sep 14 13:04:40.510718 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.510596 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63463820-fde2-42dc-ad83-f76da8cd2465-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4\" (UID: \"63463820-fde2-42dc-ad83-f76da8cd2465\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4" Sep 14 13:04:40.510718 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.510615 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63463820-fde2-42dc-ad83-f76da8cd2465-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4\" (UID: \"63463820-fde2-42dc-ad83-f76da8cd2465\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4" Sep 14 13:04:40.541914 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.541881 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fcc7ccbfd-ktknn" event={"ID":"82e2709e-ab9a-4d23-8ade-39babbfbc8aa","Type":"ContainerStarted","Data":"4fb94177ce99712a186315e687cd51f315f5d9b543fa19a1681154d8bf5a7aeb"} Sep 14 13:04:40.542091 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.541954 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fcc7ccbfd-ktknn" event={"ID":"82e2709e-ab9a-4d23-8ade-39babbfbc8aa","Type":"ContainerStarted","Data":"b4ef66c3785681e0dde30af81d23035f7d6a420ac3ce8feef2b50098b360e6c7"} Sep 14 13:04:40.562214 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.562163 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fcc7ccbfd-ktknn" podStartSLOduration=1.562146821 podStartE2EDuration="1.562146821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:04:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:04:40.560459289 +0000 UTC m=+381.223354920" watchObservedRunningTime="2026-09-14 13:04:40.562146821 +0000 UTC m=+381.225042451" Sep 14 13:04:40.611779 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.611739 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qzhl\" (UniqueName: \"kubernetes.io/projected/63463820-fde2-42dc-ad83-f76da8cd2465-kube-api-access-6qzhl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4\" (UID: \"63463820-fde2-42dc-ad83-f76da8cd2465\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4" Sep 14 13:04:40.611977 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.611837 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63463820-fde2-42dc-ad83-f76da8cd2465-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4\" (UID: \"63463820-fde2-42dc-ad83-f76da8cd2465\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4" Sep 14 13:04:40.611977 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.611863 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63463820-fde2-42dc-ad83-f76da8cd2465-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4\" (UID: \"63463820-fde2-42dc-ad83-f76da8cd2465\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4" Sep 14 13:04:40.612335 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.612316 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63463820-fde2-42dc-ad83-f76da8cd2465-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4\" (UID: \"63463820-fde2-42dc-ad83-f76da8cd2465\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4" Sep 14 13:04:40.612390 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.612337 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63463820-fde2-42dc-ad83-f76da8cd2465-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4\" (UID: \"63463820-fde2-42dc-ad83-f76da8cd2465\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4" Sep 14 13:04:40.620593 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.620570 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qzhl\" (UniqueName: \"kubernetes.io/projected/63463820-fde2-42dc-ad83-f76da8cd2465-kube-api-access-6qzhl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4\" (UID: \"63463820-fde2-42dc-ad83-f76da8cd2465\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4" Sep 14 13:04:40.760104 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:40.760019 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4" Sep 14 13:04:41.029500 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:41.029476 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4"] Sep 14 13:04:41.031835 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:04:41.031808 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63463820_fde2_42dc_ad83_f76da8cd2465.slice/crio-0d41c29f75eea068ed9bbbd8f86efbd8a82c103c51ba1eedb196a2d5da164ac6 WatchSource:0}: Error finding container 0d41c29f75eea068ed9bbbd8f86efbd8a82c103c51ba1eedb196a2d5da164ac6: Status 404 returned error can't find the container with id 0d41c29f75eea068ed9bbbd8f86efbd8a82c103c51ba1eedb196a2d5da164ac6 Sep 14 13:04:41.548227 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:41.548190 2564 generic.go:356] "Generic (PLEG): container finished" podID="63463820-fde2-42dc-ad83-f76da8cd2465" containerID="e9d5217900893a6d124e749cc27330f719287bf8b664302543d0669a3e69f02a" exitCode=0 Sep 14 13:04:41.548393 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:41.548280 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4" event={"ID":"63463820-fde2-42dc-ad83-f76da8cd2465","Type":"ContainerDied","Data":"e9d5217900893a6d124e749cc27330f719287bf8b664302543d0669a3e69f02a"} Sep 14 13:04:41.548393 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:41.548328 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4" event={"ID":"63463820-fde2-42dc-ad83-f76da8cd2465","Type":"ContainerStarted","Data":"0d41c29f75eea068ed9bbbd8f86efbd8a82c103c51ba1eedb196a2d5da164ac6"} Sep 14 13:04:42.554768 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:42.554739 2564 generic.go:356] "Generic (PLEG): container finished" podID="63463820-fde2-42dc-ad83-f76da8cd2465" containerID="95b8f65344a589b6af2764923b72c7fd0a7824cde4ea0c02c7cfc62cccf52864" exitCode=0 Sep 14 13:04:42.555135 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:42.554809 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4" event={"ID":"63463820-fde2-42dc-ad83-f76da8cd2465","Type":"ContainerDied","Data":"95b8f65344a589b6af2764923b72c7fd0a7824cde4ea0c02c7cfc62cccf52864"} Sep 14 13:04:43.561121 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:43.561085 2564 generic.go:356] "Generic (PLEG): container finished" podID="63463820-fde2-42dc-ad83-f76da8cd2465" containerID="c095ac9c4d5637e8bdd99582a12c4f1f8345f037187dd6a01d7c7c6cfc34fe21" exitCode=0 Sep 14 13:04:43.561565 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:43.561172 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4" event={"ID":"63463820-fde2-42dc-ad83-f76da8cd2465","Type":"ContainerDied","Data":"c095ac9c4d5637e8bdd99582a12c4f1f8345f037187dd6a01d7c7c6cfc34fe21"} Sep 14 13:04:44.696100 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:44.696077 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4" Sep 14 13:04:44.750062 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:44.750031 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63463820-fde2-42dc-ad83-f76da8cd2465-util\") pod \"63463820-fde2-42dc-ad83-f76da8cd2465\" (UID: \"63463820-fde2-42dc-ad83-f76da8cd2465\") " Sep 14 13:04:44.750225 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:44.750080 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63463820-fde2-42dc-ad83-f76da8cd2465-bundle\") pod \"63463820-fde2-42dc-ad83-f76da8cd2465\" (UID: \"63463820-fde2-42dc-ad83-f76da8cd2465\") " Sep 14 13:04:44.750225 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:44.750111 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qzhl\" (UniqueName: \"kubernetes.io/projected/63463820-fde2-42dc-ad83-f76da8cd2465-kube-api-access-6qzhl\") pod \"63463820-fde2-42dc-ad83-f76da8cd2465\" (UID: \"63463820-fde2-42dc-ad83-f76da8cd2465\") " Sep 14 13:04:44.750764 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:44.750739 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63463820-fde2-42dc-ad83-f76da8cd2465-bundle" (OuterVolumeSpecName: "bundle") pod "63463820-fde2-42dc-ad83-f76da8cd2465" (UID: "63463820-fde2-42dc-ad83-f76da8cd2465"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:04:44.752325 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:44.752300 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63463820-fde2-42dc-ad83-f76da8cd2465-kube-api-access-6qzhl" (OuterVolumeSpecName: "kube-api-access-6qzhl") pod "63463820-fde2-42dc-ad83-f76da8cd2465" (UID: "63463820-fde2-42dc-ad83-f76da8cd2465"). InnerVolumeSpecName "kube-api-access-6qzhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:04:44.755369 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:44.755347 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63463820-fde2-42dc-ad83-f76da8cd2465-util" (OuterVolumeSpecName: "util") pod "63463820-fde2-42dc-ad83-f76da8cd2465" (UID: "63463820-fde2-42dc-ad83-f76da8cd2465"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:04:44.851721 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:44.851636 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/63463820-fde2-42dc-ad83-f76da8cd2465-util\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:04:44.851721 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:44.851663 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/63463820-fde2-42dc-ad83-f76da8cd2465-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:04:44.851721 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:44.851672 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qzhl\" (UniqueName: \"kubernetes.io/projected/63463820-fde2-42dc-ad83-f76da8cd2465-kube-api-access-6qzhl\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:04:45.573411 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:45.573371 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4" Sep 14 13:04:45.573411 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:45.573390 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsq8j4" event={"ID":"63463820-fde2-42dc-ad83-f76da8cd2465","Type":"ContainerDied","Data":"0d41c29f75eea068ed9bbbd8f86efbd8a82c103c51ba1eedb196a2d5da164ac6"} Sep 14 13:04:45.573613 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:45.573446 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d41c29f75eea068ed9bbbd8f86efbd8a82c103c51ba1eedb196a2d5da164ac6" Sep 14 13:04:50.127108 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:50.127068 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:50.127602 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:50.127141 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:50.131819 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:50.131797 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:50.598858 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:50.598771 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fcc7ccbfd-ktknn" Sep 14 13:04:50.646819 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:50.646788 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b96f7cd45-bzhkq"] Sep 14 13:04:51.241034 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.240999 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf"] Sep 14 13:04:51.241413 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.241321 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63463820-fde2-42dc-ad83-f76da8cd2465" containerName="pull" Sep 14 13:04:51.241413 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.241332 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="63463820-fde2-42dc-ad83-f76da8cd2465" containerName="pull" Sep 14 13:04:51.241413 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.241344 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63463820-fde2-42dc-ad83-f76da8cd2465" containerName="util" Sep 14 13:04:51.241413 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.241350 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="63463820-fde2-42dc-ad83-f76da8cd2465" containerName="util" Sep 14 13:04:51.241413 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.241369 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63463820-fde2-42dc-ad83-f76da8cd2465" containerName="extract" Sep 14 13:04:51.241413 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.241374 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="63463820-fde2-42dc-ad83-f76da8cd2465" containerName="extract" Sep 14 13:04:51.241615 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.241437 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="63463820-fde2-42dc-ad83-f76da8cd2465" containerName="extract" Sep 14 13:04:51.244528 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.244511 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf" Sep 14 13:04:51.247197 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.247171 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 14 13:04:51.248461 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.248437 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 14 13:04:51.248560 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.248507 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cq4js\"" Sep 14 13:04:51.253450 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.253429 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf"] Sep 14 13:04:51.309818 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.309783 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzj4v\" (UniqueName: \"kubernetes.io/projected/ebe1b7dd-812c-4859-b706-5f5db6ea42f5-kube-api-access-bzj4v\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf\" (UID: \"ebe1b7dd-812c-4859-b706-5f5db6ea42f5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf" Sep 14 13:04:51.309986 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.309833 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebe1b7dd-812c-4859-b706-5f5db6ea42f5-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf\" (UID: \"ebe1b7dd-812c-4859-b706-5f5db6ea42f5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf" Sep 14 13:04:51.309986 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.309932 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebe1b7dd-812c-4859-b706-5f5db6ea42f5-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf\" (UID: \"ebe1b7dd-812c-4859-b706-5f5db6ea42f5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf" Sep 14 13:04:51.411160 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.411119 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzj4v\" (UniqueName: \"kubernetes.io/projected/ebe1b7dd-812c-4859-b706-5f5db6ea42f5-kube-api-access-bzj4v\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf\" (UID: \"ebe1b7dd-812c-4859-b706-5f5db6ea42f5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf" Sep 14 13:04:51.411358 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.411171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebe1b7dd-812c-4859-b706-5f5db6ea42f5-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf\" (UID: \"ebe1b7dd-812c-4859-b706-5f5db6ea42f5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf" Sep 14 13:04:51.411358 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.411221 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebe1b7dd-812c-4859-b706-5f5db6ea42f5-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf\" (UID: \"ebe1b7dd-812c-4859-b706-5f5db6ea42f5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf" Sep 14 13:04:51.411618 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.411597 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebe1b7dd-812c-4859-b706-5f5db6ea42f5-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf\" (UID: \"ebe1b7dd-812c-4859-b706-5f5db6ea42f5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf" Sep 14 13:04:51.411673 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.411626 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebe1b7dd-812c-4859-b706-5f5db6ea42f5-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf\" (UID: \"ebe1b7dd-812c-4859-b706-5f5db6ea42f5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf" Sep 14 13:04:51.420213 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.420181 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzj4v\" (UniqueName: \"kubernetes.io/projected/ebe1b7dd-812c-4859-b706-5f5db6ea42f5-kube-api-access-bzj4v\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf\" (UID: \"ebe1b7dd-812c-4859-b706-5f5db6ea42f5\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf" Sep 14 13:04:51.554169 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.554086 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf" Sep 14 13:04:51.825673 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:51.825650 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf"] Sep 14 13:04:51.827888 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:04:51.827862 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe1b7dd_812c_4859_b706_5f5db6ea42f5.slice/crio-d35935064428bd76101c1b941c32222920447b86daaaaadc107b59325f3dadaf WatchSource:0}: Error finding container d35935064428bd76101c1b941c32222920447b86daaaaadc107b59325f3dadaf: Status 404 returned error can't find the container with id d35935064428bd76101c1b941c32222920447b86daaaaadc107b59325f3dadaf Sep 14 13:04:52.608216 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.608173 2564 generic.go:356] "Generic (PLEG): container finished" podID="ebe1b7dd-812c-4859-b706-5f5db6ea42f5" containerID="86ac975665c7a088baa252d717b1dc908309df52a1d883a9f5640f904b9ebc3c" exitCode=0 Sep 14 13:04:52.608681 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.608273 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf" event={"ID":"ebe1b7dd-812c-4859-b706-5f5db6ea42f5","Type":"ContainerDied","Data":"86ac975665c7a088baa252d717b1dc908309df52a1d883a9f5640f904b9ebc3c"} Sep 14 13:04:52.608681 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.608350 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf" event={"ID":"ebe1b7dd-812c-4859-b706-5f5db6ea42f5","Type":"ContainerStarted","Data":"d35935064428bd76101c1b941c32222920447b86daaaaadc107b59325f3dadaf"} Sep 14 13:04:52.761977 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.761874 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs"] Sep 14 13:04:52.764889 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.764872 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs" Sep 14 13:04:52.767692 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.767660 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 14 13:04:52.767902 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.767887 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 14 13:04:52.768010 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.767994 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 14 13:04:52.768104 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.768076 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 14 13:04:52.768182 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.768098 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-64cdb\"" Sep 14 13:04:52.768244 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.768182 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 14 13:04:52.777250 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.777229 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs"] Sep 14 13:04:52.826198 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.826166 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/26bec738-d51b-4da8-9eaa-2ac444c83372-manager-config\") pod \"lws-controller-manager-6d5f86f746-xxrhs\" (UID: \"26bec738-d51b-4da8-9eaa-2ac444c83372\") " pod="openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs" Sep 14 13:04:52.826354 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.826205 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26bec738-d51b-4da8-9eaa-2ac444c83372-cert\") pod \"lws-controller-manager-6d5f86f746-xxrhs\" (UID: \"26bec738-d51b-4da8-9eaa-2ac444c83372\") " pod="openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs" Sep 14 13:04:52.826354 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.826237 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/26bec738-d51b-4da8-9eaa-2ac444c83372-metrics-cert\") pod \"lws-controller-manager-6d5f86f746-xxrhs\" (UID: \"26bec738-d51b-4da8-9eaa-2ac444c83372\") " pod="openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs" Sep 14 13:04:52.826354 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.826313 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf6hx\" (UniqueName: \"kubernetes.io/projected/26bec738-d51b-4da8-9eaa-2ac444c83372-kube-api-access-qf6hx\") pod \"lws-controller-manager-6d5f86f746-xxrhs\" (UID: \"26bec738-d51b-4da8-9eaa-2ac444c83372\") " pod="openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs" Sep 14 13:04:52.926817 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.926780 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/26bec738-d51b-4da8-9eaa-2ac444c83372-manager-config\") pod \"lws-controller-manager-6d5f86f746-xxrhs\" (UID: \"26bec738-d51b-4da8-9eaa-2ac444c83372\") " pod="openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs" Sep 14 13:04:52.927015 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.926843 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26bec738-d51b-4da8-9eaa-2ac444c83372-cert\") pod \"lws-controller-manager-6d5f86f746-xxrhs\" (UID: \"26bec738-d51b-4da8-9eaa-2ac444c83372\") " pod="openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs" Sep 14 13:04:52.927015 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.926899 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/26bec738-d51b-4da8-9eaa-2ac444c83372-metrics-cert\") pod \"lws-controller-manager-6d5f86f746-xxrhs\" (UID: \"26bec738-d51b-4da8-9eaa-2ac444c83372\") " pod="openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs" Sep 14 13:04:52.927144 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.927020 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf6hx\" (UniqueName: \"kubernetes.io/projected/26bec738-d51b-4da8-9eaa-2ac444c83372-kube-api-access-qf6hx\") pod \"lws-controller-manager-6d5f86f746-xxrhs\" (UID: \"26bec738-d51b-4da8-9eaa-2ac444c83372\") " pod="openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs" Sep 14 13:04:52.927621 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.927550 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/26bec738-d51b-4da8-9eaa-2ac444c83372-manager-config\") pod \"lws-controller-manager-6d5f86f746-xxrhs\" (UID: \"26bec738-d51b-4da8-9eaa-2ac444c83372\") " pod="openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs" Sep 14 13:04:52.929542 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.929517 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/26bec738-d51b-4da8-9eaa-2ac444c83372-metrics-cert\") pod \"lws-controller-manager-6d5f86f746-xxrhs\" (UID: \"26bec738-d51b-4da8-9eaa-2ac444c83372\") " pod="openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs" Sep 14 13:04:52.929624 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.929598 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26bec738-d51b-4da8-9eaa-2ac444c83372-cert\") pod \"lws-controller-manager-6d5f86f746-xxrhs\" (UID: \"26bec738-d51b-4da8-9eaa-2ac444c83372\") " pod="openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs" Sep 14 13:04:52.935335 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:52.935316 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf6hx\" (UniqueName: \"kubernetes.io/projected/26bec738-d51b-4da8-9eaa-2ac444c83372-kube-api-access-qf6hx\") pod \"lws-controller-manager-6d5f86f746-xxrhs\" (UID: \"26bec738-d51b-4da8-9eaa-2ac444c83372\") " pod="openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs" Sep 14 13:04:53.074810 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.074706 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs" Sep 14 13:04:53.088449 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.088414 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r"] Sep 14 13:04:53.091164 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.091136 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" Sep 14 13:04:53.094742 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.094695 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 14 13:04:53.094860 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.094824 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 14 13:04:53.095094 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.095063 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 14 13:04:53.095094 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.095083 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-bdfph\"" Sep 14 13:04:53.095240 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.095102 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 14 13:04:53.109602 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.109580 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r"] Sep 14 13:04:53.128192 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.128162 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3d67c169-7edc-46cb-ba89-b753cb011b73-apiservice-cert\") pod \"opendatahub-operator-controller-manager-697cf97c89-dwk8r\" (UID: \"3d67c169-7edc-46cb-ba89-b753cb011b73\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" Sep 14 13:04:53.128320 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.128215 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d67c169-7edc-46cb-ba89-b753cb011b73-tmp\") pod \"opendatahub-operator-controller-manager-697cf97c89-dwk8r\" (UID: \"3d67c169-7edc-46cb-ba89-b753cb011b73\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" Sep 14 13:04:53.128320 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.128253 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3d67c169-7edc-46cb-ba89-b753cb011b73-webhook-cert\") pod \"opendatahub-operator-controller-manager-697cf97c89-dwk8r\" (UID: \"3d67c169-7edc-46cb-ba89-b753cb011b73\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" Sep 14 13:04:53.128320 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.128278 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3d67c169-7edc-46cb-ba89-b753cb011b73-manifests\") pod \"opendatahub-operator-controller-manager-697cf97c89-dwk8r\" (UID: \"3d67c169-7edc-46cb-ba89-b753cb011b73\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" Sep 14 13:04:53.128485 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.128350 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf2lf\" (UniqueName: \"kubernetes.io/projected/3d67c169-7edc-46cb-ba89-b753cb011b73-kube-api-access-mf2lf\") pod \"opendatahub-operator-controller-manager-697cf97c89-dwk8r\" (UID: \"3d67c169-7edc-46cb-ba89-b753cb011b73\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" Sep 14 13:04:53.229715 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.229687 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3d67c169-7edc-46cb-ba89-b753cb011b73-apiservice-cert\") pod \"opendatahub-operator-controller-manager-697cf97c89-dwk8r\" (UID: \"3d67c169-7edc-46cb-ba89-b753cb011b73\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" Sep 14 13:04:53.229834 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.229734 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d67c169-7edc-46cb-ba89-b753cb011b73-tmp\") pod \"opendatahub-operator-controller-manager-697cf97c89-dwk8r\" (UID: \"3d67c169-7edc-46cb-ba89-b753cb011b73\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" Sep 14 13:04:53.229834 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.229768 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3d67c169-7edc-46cb-ba89-b753cb011b73-webhook-cert\") pod \"opendatahub-operator-controller-manager-697cf97c89-dwk8r\" (UID: \"3d67c169-7edc-46cb-ba89-b753cb011b73\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" Sep 14 13:04:53.229834 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.229793 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3d67c169-7edc-46cb-ba89-b753cb011b73-manifests\") pod \"opendatahub-operator-controller-manager-697cf97c89-dwk8r\" (UID: \"3d67c169-7edc-46cb-ba89-b753cb011b73\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" Sep 14 13:04:53.230034 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.229868 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf2lf\" (UniqueName: \"kubernetes.io/projected/3d67c169-7edc-46cb-ba89-b753cb011b73-kube-api-access-mf2lf\") pod \"opendatahub-operator-controller-manager-697cf97c89-dwk8r\" (UID: \"3d67c169-7edc-46cb-ba89-b753cb011b73\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" Sep 14 13:04:53.230237 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.230213 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3d67c169-7edc-46cb-ba89-b753cb011b73-manifests\") pod \"opendatahub-operator-controller-manager-697cf97c89-dwk8r\" (UID: \"3d67c169-7edc-46cb-ba89-b753cb011b73\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" Sep 14 13:04:53.232298 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.232273 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3d67c169-7edc-46cb-ba89-b753cb011b73-apiservice-cert\") pod \"opendatahub-operator-controller-manager-697cf97c89-dwk8r\" (UID: \"3d67c169-7edc-46cb-ba89-b753cb011b73\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" Sep 14 13:04:53.232611 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.232591 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d67c169-7edc-46cb-ba89-b753cb011b73-tmp\") pod \"opendatahub-operator-controller-manager-697cf97c89-dwk8r\" (UID: \"3d67c169-7edc-46cb-ba89-b753cb011b73\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" Sep 14 13:04:53.232739 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.232720 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3d67c169-7edc-46cb-ba89-b753cb011b73-webhook-cert\") pod \"opendatahub-operator-controller-manager-697cf97c89-dwk8r\" (UID: \"3d67c169-7edc-46cb-ba89-b753cb011b73\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" Sep 14 13:04:53.244370 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.244340 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf2lf\" (UniqueName: \"kubernetes.io/projected/3d67c169-7edc-46cb-ba89-b753cb011b73-kube-api-access-mf2lf\") pod \"opendatahub-operator-controller-manager-697cf97c89-dwk8r\" (UID: \"3d67c169-7edc-46cb-ba89-b753cb011b73\") " pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" Sep 14 13:04:53.390619 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.390590 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs"] Sep 14 13:04:53.392313 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:04:53.392284 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26bec738_d51b_4da8_9eaa_2ac444c83372.slice/crio-4899b32e745cfb62dc37eeca596c6cafd213fd9ae3aeff2817f82b5b25e05808 WatchSource:0}: Error finding container 4899b32e745cfb62dc37eeca596c6cafd213fd9ae3aeff2817f82b5b25e05808: Status 404 returned error can't find the container with id 4899b32e745cfb62dc37eeca596c6cafd213fd9ae3aeff2817f82b5b25e05808 Sep 14 13:04:53.405092 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.405070 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" Sep 14 13:04:53.614319 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.614278 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs" event={"ID":"26bec738-d51b-4da8-9eaa-2ac444c83372","Type":"ContainerStarted","Data":"4899b32e745cfb62dc37eeca596c6cafd213fd9ae3aeff2817f82b5b25e05808"} Sep 14 13:04:53.616280 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.616254 2564 generic.go:356] "Generic (PLEG): container finished" podID="ebe1b7dd-812c-4859-b706-5f5db6ea42f5" containerID="26d86867f5c13fbd358b410bcc3f9fb0c3e0301540081882ae57831d943f8c9f" exitCode=0 Sep 14 13:04:53.616398 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.616321 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf" event={"ID":"ebe1b7dd-812c-4859-b706-5f5db6ea42f5","Type":"ContainerDied","Data":"26d86867f5c13fbd358b410bcc3f9fb0c3e0301540081882ae57831d943f8c9f"} Sep 14 13:04:53.690172 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:53.690149 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r"] Sep 14 13:04:53.692432 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:04:53.692406 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d67c169_7edc_46cb_ba89_b753cb011b73.slice/crio-33fe6edde4951d475f761c01f6c62d8c1f629dcd54fc2334ca64ea7aa60327aa WatchSource:0}: Error finding container 33fe6edde4951d475f761c01f6c62d8c1f629dcd54fc2334ca64ea7aa60327aa: Status 404 returned error can't find the container with id 33fe6edde4951d475f761c01f6c62d8c1f629dcd54fc2334ca64ea7aa60327aa Sep 14 13:04:54.628561 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:54.628033 2564 generic.go:356] "Generic (PLEG): container finished" podID="ebe1b7dd-812c-4859-b706-5f5db6ea42f5" containerID="8042c82a2d306aeb1fd966f3b3a546a4f31047b0cfaacf6729892dfa9ec6e267" exitCode=0 Sep 14 13:04:54.628561 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:54.628378 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf" event={"ID":"ebe1b7dd-812c-4859-b706-5f5db6ea42f5","Type":"ContainerDied","Data":"8042c82a2d306aeb1fd966f3b3a546a4f31047b0cfaacf6729892dfa9ec6e267"} Sep 14 13:04:54.630848 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:54.630816 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" event={"ID":"3d67c169-7edc-46cb-ba89-b753cb011b73","Type":"ContainerStarted","Data":"33fe6edde4951d475f761c01f6c62d8c1f629dcd54fc2334ca64ea7aa60327aa"} Sep 14 13:04:55.945865 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:55.945838 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf" Sep 14 13:04:56.055335 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:56.055293 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzj4v\" (UniqueName: \"kubernetes.io/projected/ebe1b7dd-812c-4859-b706-5f5db6ea42f5-kube-api-access-bzj4v\") pod \"ebe1b7dd-812c-4859-b706-5f5db6ea42f5\" (UID: \"ebe1b7dd-812c-4859-b706-5f5db6ea42f5\") " Sep 14 13:04:56.055518 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:56.055387 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebe1b7dd-812c-4859-b706-5f5db6ea42f5-bundle\") pod \"ebe1b7dd-812c-4859-b706-5f5db6ea42f5\" (UID: \"ebe1b7dd-812c-4859-b706-5f5db6ea42f5\") " Sep 14 13:04:56.055518 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:56.055506 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebe1b7dd-812c-4859-b706-5f5db6ea42f5-util\") pod \"ebe1b7dd-812c-4859-b706-5f5db6ea42f5\" (UID: \"ebe1b7dd-812c-4859-b706-5f5db6ea42f5\") " Sep 14 13:04:56.056294 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:56.056269 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe1b7dd-812c-4859-b706-5f5db6ea42f5-bundle" (OuterVolumeSpecName: "bundle") pod "ebe1b7dd-812c-4859-b706-5f5db6ea42f5" (UID: "ebe1b7dd-812c-4859-b706-5f5db6ea42f5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:04:56.058183 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:56.058152 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe1b7dd-812c-4859-b706-5f5db6ea42f5-kube-api-access-bzj4v" (OuterVolumeSpecName: "kube-api-access-bzj4v") pod "ebe1b7dd-812c-4859-b706-5f5db6ea42f5" (UID: "ebe1b7dd-812c-4859-b706-5f5db6ea42f5"). InnerVolumeSpecName "kube-api-access-bzj4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:04:56.061660 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:56.061635 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebe1b7dd-812c-4859-b706-5f5db6ea42f5-util" (OuterVolumeSpecName: "util") pod "ebe1b7dd-812c-4859-b706-5f5db6ea42f5" (UID: "ebe1b7dd-812c-4859-b706-5f5db6ea42f5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:04:56.156467 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:56.156431 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ebe1b7dd-812c-4859-b706-5f5db6ea42f5-util\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:04:56.156467 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:56.156462 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzj4v\" (UniqueName: \"kubernetes.io/projected/ebe1b7dd-812c-4859-b706-5f5db6ea42f5-kube-api-access-bzj4v\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:04:56.156467 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:56.156473 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ebe1b7dd-812c-4859-b706-5f5db6ea42f5-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:04:56.644470 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:56.644431 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf" event={"ID":"ebe1b7dd-812c-4859-b706-5f5db6ea42f5","Type":"ContainerDied","Data":"d35935064428bd76101c1b941c32222920447b86daaaaadc107b59325f3dadaf"} Sep 14 13:04:56.644869 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:56.644481 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0zb8qf" Sep 14 13:04:56.644869 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:56.644498 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d35935064428bd76101c1b941c32222920447b86daaaaadc107b59325f3dadaf" Sep 14 13:04:57.651047 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:57.651001 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" event={"ID":"3d67c169-7edc-46cb-ba89-b753cb011b73","Type":"ContainerStarted","Data":"5d07c1cc76b3a9e3f2fba4f619a55179da77d9e6af31e223a00d2b9454041fdb"} Sep 14 13:04:57.652482 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:57.652459 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs" event={"ID":"26bec738-d51b-4da8-9eaa-2ac444c83372","Type":"ContainerStarted","Data":"5f1498bff14f3944f38a61578f93820483954a4531f769d5e0a1c0f71c62ec88"} Sep 14 13:04:57.652600 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:57.652588 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs" Sep 14 13:04:57.696301 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:57.696243 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs" podStartSLOduration=2.244062928 podStartE2EDuration="5.696226788s" totalImagesPullingTime="3.45216386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:04:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:04:57.694685297 +0000 UTC m=+398.357580924" watchObservedRunningTime="2026-09-14 13:04:57.696226788 +0000 UTC m=+398.359122418" Sep 14 13:04:58.659187 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:58.659152 2564 generic.go:356] "Generic (PLEG): container finished" podID="3d67c169-7edc-46cb-ba89-b753cb011b73" containerID="5d07c1cc76b3a9e3f2fba4f619a55179da77d9e6af31e223a00d2b9454041fdb" exitCode=0 Sep 14 13:04:58.659659 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:58.659237 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" event={"ID":"3d67c169-7edc-46cb-ba89-b753cb011b73","Type":"ContainerDied","Data":"5d07c1cc76b3a9e3f2fba4f619a55179da77d9e6af31e223a00d2b9454041fdb"} Sep 14 13:04:59.667146 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:59.667110 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" event={"ID":"3d67c169-7edc-46cb-ba89-b753cb011b73","Type":"ContainerStarted","Data":"2f9acef19a16b77f3c763591d707435895769b5fd8ad3439e92c696430c4ba89"} Sep 14 13:04:59.667543 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:59.667209 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" Sep 14 13:04:59.689791 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:04:59.689740 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" podStartSLOduration=2.026083566 podStartE2EDuration="6.689724512s" totalImagesPullingTime="3.576283685s" totalInitContainerRuntime="1.087357261s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:04:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:04:59.687674553 +0000 UTC m=+400.350570183" watchObservedRunningTime="2026-09-14 13:04:59.689724512 +0000 UTC m=+400.352620175" Sep 14 13:05:08.661494 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:08.661466 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6d5f86f746-xxrhs" Sep 14 13:05:10.675475 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:10.675437 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-697cf97c89-dwk8r" Sep 14 13:05:15.669337 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:15.669287 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b96f7cd45-bzhkq" podUID="fc632355-6859-488f-9cbf-e4a17f602cd9" containerName="console" containerID="cri-o://7b3966e889be08ba0f374e3b072de25409d04683d35a795894596aa8ea704861" gracePeriod=15 Sep 14 13:05:15.908376 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:15.908352 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:05:16.031368 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.031273 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-trusted-ca-bundle\") pod \"fc632355-6859-488f-9cbf-e4a17f602cd9\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " Sep 14 13:05:16.031368 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.031338 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-oauth-serving-cert\") pod \"fc632355-6859-488f-9cbf-e4a17f602cd9\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " Sep 14 13:05:16.031368 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.031363 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-service-ca\") pod \"fc632355-6859-488f-9cbf-e4a17f602cd9\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " Sep 14 13:05:16.031654 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.031388 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc632355-6859-488f-9cbf-e4a17f602cd9-console-oauth-config\") pod \"fc632355-6859-488f-9cbf-e4a17f602cd9\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " Sep 14 13:05:16.031654 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.031504 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc632355-6859-488f-9cbf-e4a17f602cd9-console-serving-cert\") pod \"fc632355-6859-488f-9cbf-e4a17f602cd9\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " Sep 14 13:05:16.031654 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.031533 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-console-config\") pod \"fc632355-6859-488f-9cbf-e4a17f602cd9\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " Sep 14 13:05:16.031654 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.031608 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckc4k\" (UniqueName: \"kubernetes.io/projected/fc632355-6859-488f-9cbf-e4a17f602cd9-kube-api-access-ckc4k\") pod \"fc632355-6859-488f-9cbf-e4a17f602cd9\" (UID: \"fc632355-6859-488f-9cbf-e4a17f602cd9\") " Sep 14 13:05:16.031844 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.031758 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fc632355-6859-488f-9cbf-e4a17f602cd9" (UID: "fc632355-6859-488f-9cbf-e4a17f602cd9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:05:16.031844 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.031833 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-service-ca" (OuterVolumeSpecName: "service-ca") pod "fc632355-6859-488f-9cbf-e4a17f602cd9" (UID: "fc632355-6859-488f-9cbf-e4a17f602cd9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:05:16.032000 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.031886 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-trusted-ca-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:05:16.032000 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.031906 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-console-config" (OuterVolumeSpecName: "console-config") pod "fc632355-6859-488f-9cbf-e4a17f602cd9" (UID: "fc632355-6859-488f-9cbf-e4a17f602cd9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:05:16.032138 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.032113 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fc632355-6859-488f-9cbf-e4a17f602cd9" (UID: "fc632355-6859-488f-9cbf-e4a17f602cd9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:05:16.033843 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.033813 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc632355-6859-488f-9cbf-e4a17f602cd9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fc632355-6859-488f-9cbf-e4a17f602cd9" (UID: "fc632355-6859-488f-9cbf-e4a17f602cd9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:05:16.033843 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.033833 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc632355-6859-488f-9cbf-e4a17f602cd9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fc632355-6859-488f-9cbf-e4a17f602cd9" (UID: "fc632355-6859-488f-9cbf-e4a17f602cd9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:05:16.034036 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.033890 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc632355-6859-488f-9cbf-e4a17f602cd9-kube-api-access-ckc4k" (OuterVolumeSpecName: "kube-api-access-ckc4k") pod "fc632355-6859-488f-9cbf-e4a17f602cd9" (UID: "fc632355-6859-488f-9cbf-e4a17f602cd9"). InnerVolumeSpecName "kube-api-access-ckc4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:05:16.132488 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.132452 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckc4k\" (UniqueName: \"kubernetes.io/projected/fc632355-6859-488f-9cbf-e4a17f602cd9-kube-api-access-ckc4k\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:05:16.132488 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.132482 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-oauth-serving-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:05:16.132488 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.132492 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:05:16.132720 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.132503 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fc632355-6859-488f-9cbf-e4a17f602cd9-console-oauth-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:05:16.132720 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.132511 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fc632355-6859-488f-9cbf-e4a17f602cd9-console-serving-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:05:16.132720 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.132519 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fc632355-6859-488f-9cbf-e4a17f602cd9-console-config\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:05:16.753044 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.753009 2564 generic.go:356] "Generic (PLEG): container finished" podID="fc632355-6859-488f-9cbf-e4a17f602cd9" containerID="7b3966e889be08ba0f374e3b072de25409d04683d35a795894596aa8ea704861" exitCode=0 Sep 14 13:05:16.753421 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.753052 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b96f7cd45-bzhkq" event={"ID":"fc632355-6859-488f-9cbf-e4a17f602cd9","Type":"ContainerDied","Data":"7b3966e889be08ba0f374e3b072de25409d04683d35a795894596aa8ea704861"} Sep 14 13:05:16.753421 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.753058 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b96f7cd45-bzhkq" Sep 14 13:05:16.753421 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.753092 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b96f7cd45-bzhkq" event={"ID":"fc632355-6859-488f-9cbf-e4a17f602cd9","Type":"ContainerDied","Data":"5a0f3986f07e71a6dccfe32962b7053c1e8232a8a83658bb523d2c8c9ecfb3c6"} Sep 14 13:05:16.753421 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.753111 2564 scope.go:117] "RemoveContainer" containerID="7b3966e889be08ba0f374e3b072de25409d04683d35a795894596aa8ea704861" Sep 14 13:05:16.762435 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.762420 2564 scope.go:117] "RemoveContainer" containerID="7b3966e889be08ba0f374e3b072de25409d04683d35a795894596aa8ea704861" Sep 14 13:05:16.762704 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:05:16.762685 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3966e889be08ba0f374e3b072de25409d04683d35a795894596aa8ea704861\": container with ID starting with 7b3966e889be08ba0f374e3b072de25409d04683d35a795894596aa8ea704861 not found: ID does not exist" containerID="7b3966e889be08ba0f374e3b072de25409d04683d35a795894596aa8ea704861" Sep 14 13:05:16.762742 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.762714 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3966e889be08ba0f374e3b072de25409d04683d35a795894596aa8ea704861"} err="failed to get container status \"7b3966e889be08ba0f374e3b072de25409d04683d35a795894596aa8ea704861\": rpc error: code = NotFound desc = could not find container \"7b3966e889be08ba0f374e3b072de25409d04683d35a795894596aa8ea704861\": container with ID starting with 7b3966e889be08ba0f374e3b072de25409d04683d35a795894596aa8ea704861 not found: ID does not exist" Sep 14 13:05:16.775908 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.775875 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b96f7cd45-bzhkq"] Sep 14 13:05:16.779441 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:16.779406 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b96f7cd45-bzhkq"] Sep 14 13:05:17.810954 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:17.810908 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc632355-6859-488f-9cbf-e4a17f602cd9" path="/var/lib/kubelet/pods/fc632355-6859-488f-9cbf-e4a17f602cd9/volumes" Sep 14 13:05:30.425997 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.425963 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz"] Sep 14 13:05:30.426449 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.426327 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebe1b7dd-812c-4859-b706-5f5db6ea42f5" containerName="extract" Sep 14 13:05:30.426449 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.426340 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebe1b7dd-812c-4859-b706-5f5db6ea42f5" containerName="extract" Sep 14 13:05:30.426449 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.426353 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebe1b7dd-812c-4859-b706-5f5db6ea42f5" containerName="pull" Sep 14 13:05:30.426449 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.426358 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebe1b7dd-812c-4859-b706-5f5db6ea42f5" containerName="pull" Sep 14 13:05:30.426449 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.426371 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc632355-6859-488f-9cbf-e4a17f602cd9" containerName="console" Sep 14 13:05:30.426449 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.426376 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc632355-6859-488f-9cbf-e4a17f602cd9" containerName="console" Sep 14 13:05:30.426449 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.426387 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebe1b7dd-812c-4859-b706-5f5db6ea42f5" containerName="util" Sep 14 13:05:30.426449 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.426395 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebe1b7dd-812c-4859-b706-5f5db6ea42f5" containerName="util" Sep 14 13:05:30.426449 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.426451 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebe1b7dd-812c-4859-b706-5f5db6ea42f5" containerName="extract" Sep 14 13:05:30.426729 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.426462 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc632355-6859-488f-9cbf-e4a17f602cd9" containerName="console" Sep 14 13:05:30.515376 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.515338 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz"] Sep 14 13:05:30.515538 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.515471 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.518201 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.518181 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qkchr\"" Sep 14 13:05:30.518333 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.518247 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 14 13:05:30.560211 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.560183 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf1706b-9e71-41e5-82ba-8b1e58997b29-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.560359 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.560219 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf1706b-9e71-41e5-82ba-8b1e58997b29-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.560359 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.560300 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf1706b-9e71-41e5-82ba-8b1e58997b29-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.560359 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.560336 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf1706b-9e71-41e5-82ba-8b1e58997b29-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.560522 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.560380 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf1706b-9e71-41e5-82ba-8b1e58997b29-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.560522 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.560425 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sf4j\" (UniqueName: \"kubernetes.io/projected/bbf1706b-9e71-41e5-82ba-8b1e58997b29-kube-api-access-6sf4j\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.560522 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.560465 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf1706b-9e71-41e5-82ba-8b1e58997b29-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.560522 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.560496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf1706b-9e71-41e5-82ba-8b1e58997b29-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.560522 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.560520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf1706b-9e71-41e5-82ba-8b1e58997b29-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.660910 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.660879 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf1706b-9e71-41e5-82ba-8b1e58997b29-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.660910 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.660916 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf1706b-9e71-41e5-82ba-8b1e58997b29-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.661157 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.660954 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf1706b-9e71-41e5-82ba-8b1e58997b29-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.661157 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.660976 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sf4j\" (UniqueName: \"kubernetes.io/projected/bbf1706b-9e71-41e5-82ba-8b1e58997b29-kube-api-access-6sf4j\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.661157 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.661002 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf1706b-9e71-41e5-82ba-8b1e58997b29-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.661157 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.661022 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf1706b-9e71-41e5-82ba-8b1e58997b29-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.661157 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.661048 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf1706b-9e71-41e5-82ba-8b1e58997b29-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.661157 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.661096 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf1706b-9e71-41e5-82ba-8b1e58997b29-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.661157 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.661135 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf1706b-9e71-41e5-82ba-8b1e58997b29-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.661521 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.661307 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbf1706b-9e71-41e5-82ba-8b1e58997b29-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.661521 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.661367 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf1706b-9e71-41e5-82ba-8b1e58997b29-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.661521 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.661459 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbf1706b-9e71-41e5-82ba-8b1e58997b29-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.661685 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.661549 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbf1706b-9e71-41e5-82ba-8b1e58997b29-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.661916 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.661738 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbf1706b-9e71-41e5-82ba-8b1e58997b29-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.663529 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.663507 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbf1706b-9e71-41e5-82ba-8b1e58997b29-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.663751 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.663731 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbf1706b-9e71-41e5-82ba-8b1e58997b29-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.669851 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.669830 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sf4j\" (UniqueName: \"kubernetes.io/projected/bbf1706b-9e71-41e5-82ba-8b1e58997b29-kube-api-access-6sf4j\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.669980 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.669961 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbf1706b-9e71-41e5-82ba-8b1e58997b29-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz\" (UID: \"bbf1706b-9e71-41e5-82ba-8b1e58997b29\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:30.825446 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:30.825363 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:31.127645 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:31.127620 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz"] Sep 14 13:05:31.129453 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:05:31.129427 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf1706b_9e71_41e5_82ba_8b1e58997b29.slice/crio-a91de0172ce0125eb353efa0140677eda7501c17d0e99a45e55c9d2662db01a0 WatchSource:0}: Error finding container a91de0172ce0125eb353efa0140677eda7501c17d0e99a45e55c9d2662db01a0: Status 404 returned error can't find the container with id a91de0172ce0125eb353efa0140677eda7501c17d0e99a45e55c9d2662db01a0 Sep 14 13:05:31.833323 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:31.833281 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" event={"ID":"bbf1706b-9e71-41e5-82ba-8b1e58997b29","Type":"ContainerStarted","Data":"a91de0172ce0125eb353efa0140677eda7501c17d0e99a45e55c9d2662db01a0"} Sep 14 13:05:33.564872 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:33.564812 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:05:33.565155 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:33.564916 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:05:33.565155 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:33.564993 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:05:33.844948 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:33.844834 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" event={"ID":"bbf1706b-9e71-41e5-82ba-8b1e58997b29","Type":"ContainerStarted","Data":"5dd20881a12737487953595d87112593a432f0ef05538302234a1d2065c0cb1c"} Sep 14 13:05:33.868518 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:33.868462 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" podStartSLOduration=1.435364029 podStartE2EDuration="3.868446242s" totalImagesPullingTime="2.433082213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:05:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:05:33.865747861 +0000 UTC m=+434.528643489" watchObservedRunningTime="2026-09-14 13:05:33.868446242 +0000 UTC m=+434.531341872" Sep 14 13:05:33.942946 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:33.942890 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rwxn9"] Sep 14 13:05:33.945330 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:33.945313 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rwxn9" Sep 14 13:05:33.948051 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:33.948030 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 14 13:05:33.948244 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:33.948227 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-lfvdr\"" Sep 14 13:05:33.958009 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:33.957988 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rwxn9"] Sep 14 13:05:33.989056 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:33.989028 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ee95cc6-a9d1-426d-b595-b83cc025638d-cert\") pod \"odh-model-controller-7998c67dc4-rwxn9\" (UID: \"6ee95cc6-a9d1-426d-b595-b83cc025638d\") " pod="opendatahub/odh-model-controller-7998c67dc4-rwxn9" Sep 14 13:05:33.989192 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:33.989068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpjb9\" (UniqueName: \"kubernetes.io/projected/6ee95cc6-a9d1-426d-b595-b83cc025638d-kube-api-access-cpjb9\") pod \"odh-model-controller-7998c67dc4-rwxn9\" (UID: \"6ee95cc6-a9d1-426d-b595-b83cc025638d\") " pod="opendatahub/odh-model-controller-7998c67dc4-rwxn9" Sep 14 13:05:34.089637 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:34.089604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ee95cc6-a9d1-426d-b595-b83cc025638d-cert\") pod \"odh-model-controller-7998c67dc4-rwxn9\" (UID: \"6ee95cc6-a9d1-426d-b595-b83cc025638d\") " pod="opendatahub/odh-model-controller-7998c67dc4-rwxn9" Sep 14 13:05:34.089811 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:34.089653 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpjb9\" (UniqueName: \"kubernetes.io/projected/6ee95cc6-a9d1-426d-b595-b83cc025638d-kube-api-access-cpjb9\") pod \"odh-model-controller-7998c67dc4-rwxn9\" (UID: \"6ee95cc6-a9d1-426d-b595-b83cc025638d\") " pod="opendatahub/odh-model-controller-7998c67dc4-rwxn9" Sep 14 13:05:34.089811 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:05:34.089762 2564 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 14 13:05:34.089896 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:05:34.089826 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ee95cc6-a9d1-426d-b595-b83cc025638d-cert podName:6ee95cc6-a9d1-426d-b595-b83cc025638d nodeName:}" failed. No retries permitted until 2026-09-14 13:05:34.589805858 +0000 UTC m=+435.252701466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6ee95cc6-a9d1-426d-b595-b83cc025638d-cert") pod "odh-model-controller-7998c67dc4-rwxn9" (UID: "6ee95cc6-a9d1-426d-b595-b83cc025638d") : secret "odh-model-controller-webhook-cert" not found Sep 14 13:05:34.099042 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:34.098971 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpjb9\" (UniqueName: \"kubernetes.io/projected/6ee95cc6-a9d1-426d-b595-b83cc025638d-kube-api-access-cpjb9\") pod \"odh-model-controller-7998c67dc4-rwxn9\" (UID: \"6ee95cc6-a9d1-426d-b595-b83cc025638d\") " pod="opendatahub/odh-model-controller-7998c67dc4-rwxn9" Sep 14 13:05:34.593419 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:34.593387 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ee95cc6-a9d1-426d-b595-b83cc025638d-cert\") pod \"odh-model-controller-7998c67dc4-rwxn9\" (UID: \"6ee95cc6-a9d1-426d-b595-b83cc025638d\") " pod="opendatahub/odh-model-controller-7998c67dc4-rwxn9" Sep 14 13:05:34.593799 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:05:34.593506 2564 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 14 13:05:34.593799 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:05:34.593558 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ee95cc6-a9d1-426d-b595-b83cc025638d-cert podName:6ee95cc6-a9d1-426d-b595-b83cc025638d nodeName:}" failed. No retries permitted until 2026-09-14 13:05:35.593542462 +0000 UTC m=+436.256438069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6ee95cc6-a9d1-426d-b595-b83cc025638d-cert") pod "odh-model-controller-7998c67dc4-rwxn9" (UID: "6ee95cc6-a9d1-426d-b595-b83cc025638d") : secret "odh-model-controller-webhook-cert" not found Sep 14 13:05:34.826104 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:34.826065 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:34.830684 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:34.830664 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:34.849660 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:34.849593 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:34.850628 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:34.850605 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz" Sep 14 13:05:35.603429 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:35.603387 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ee95cc6-a9d1-426d-b595-b83cc025638d-cert\") pod \"odh-model-controller-7998c67dc4-rwxn9\" (UID: \"6ee95cc6-a9d1-426d-b595-b83cc025638d\") " pod="opendatahub/odh-model-controller-7998c67dc4-rwxn9" Sep 14 13:05:35.605901 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:35.605882 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ee95cc6-a9d1-426d-b595-b83cc025638d-cert\") pod \"odh-model-controller-7998c67dc4-rwxn9\" (UID: \"6ee95cc6-a9d1-426d-b595-b83cc025638d\") " pod="opendatahub/odh-model-controller-7998c67dc4-rwxn9" Sep 14 13:05:35.755961 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:35.755910 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-rwxn9" Sep 14 13:05:36.025270 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:36.025241 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-rwxn9"] Sep 14 13:05:36.026994 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:05:36.026970 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee95cc6_a9d1_426d_b595_b83cc025638d.slice/crio-3e42a25d040bfd0772899dbc390cc7294040f2c965aaacf575706ccc9a7c9fdd WatchSource:0}: Error finding container 3e42a25d040bfd0772899dbc390cc7294040f2c965aaacf575706ccc9a7c9fdd: Status 404 returned error can't find the container with id 3e42a25d040bfd0772899dbc390cc7294040f2c965aaacf575706ccc9a7c9fdd Sep 14 13:05:36.864985 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:36.864945 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rwxn9" event={"ID":"6ee95cc6-a9d1-426d-b595-b83cc025638d","Type":"ContainerStarted","Data":"3e42a25d040bfd0772899dbc390cc7294040f2c965aaacf575706ccc9a7c9fdd"} Sep 14 13:05:38.882469 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:38.882426 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-rwxn9" event={"ID":"6ee95cc6-a9d1-426d-b595-b83cc025638d","Type":"ContainerStarted","Data":"c9d2d3f5879c2dbfac257e1f0ef2a5b89d10bb6bbda15d235e2d3aba0f55a479"} Sep 14 13:05:38.882909 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:38.882525 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-rwxn9" Sep 14 13:05:38.902106 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:38.902057 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-rwxn9" podStartSLOduration=3.225286748 podStartE2EDuration="5.902042447s" totalImagesPullingTime="2.676755699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:05:38.899638869 +0000 UTC m=+439.562534499" watchObservedRunningTime="2026-09-14 13:05:38.902042447 +0000 UTC m=+439.564938077" Sep 14 13:05:40.663670 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.663635 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts"] Sep 14 13:05:40.666257 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.666241 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.670984 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.670964 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 14 13:05:40.671271 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.671252 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 14 13:05:40.671734 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.671718 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-27rzz\"" Sep 14 13:05:40.672007 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.671987 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 14 13:05:40.702143 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.702118 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts"] Sep 14 13:05:40.754721 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.754671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f9c24a09-932f-41c9-8391-16b38a0d7ee6-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.754911 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.754751 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c24a09-932f-41c9-8391-16b38a0d7ee6-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.754911 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.754813 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f9c24a09-932f-41c9-8391-16b38a0d7ee6-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.754911 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.754871 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c24a09-932f-41c9-8391-16b38a0d7ee6-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.755114 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.754945 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmdwf\" (UniqueName: \"kubernetes.io/projected/f9c24a09-932f-41c9-8391-16b38a0d7ee6-kube-api-access-fmdwf\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.755114 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.754999 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f9c24a09-932f-41c9-8391-16b38a0d7ee6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.755114 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.755030 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f9c24a09-932f-41c9-8391-16b38a0d7ee6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.855800 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.855768 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c24a09-932f-41c9-8391-16b38a0d7ee6-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.855800 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.855802 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f9c24a09-932f-41c9-8391-16b38a0d7ee6-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.856073 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.855838 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c24a09-932f-41c9-8391-16b38a0d7ee6-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.856073 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.855855 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmdwf\" (UniqueName: \"kubernetes.io/projected/f9c24a09-932f-41c9-8391-16b38a0d7ee6-kube-api-access-fmdwf\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.856073 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.855991 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f9c24a09-932f-41c9-8391-16b38a0d7ee6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.856073 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.856045 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f9c24a09-932f-41c9-8391-16b38a0d7ee6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.856411 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.856090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f9c24a09-932f-41c9-8391-16b38a0d7ee6-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.857007 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.856980 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f9c24a09-932f-41c9-8391-16b38a0d7ee6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.858280 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.858255 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9c24a09-932f-41c9-8391-16b38a0d7ee6-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.858628 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.858607 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f9c24a09-932f-41c9-8391-16b38a0d7ee6-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.858914 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.858891 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f9c24a09-932f-41c9-8391-16b38a0d7ee6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.859030 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.858971 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f9c24a09-932f-41c9-8391-16b38a0d7ee6-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.864485 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.864463 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9c24a09-932f-41c9-8391-16b38a0d7ee6-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.864588 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.864568 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmdwf\" (UniqueName: \"kubernetes.io/projected/f9c24a09-932f-41c9-8391-16b38a0d7ee6-kube-api-access-fmdwf\") pod \"istiod-openshift-gateway-5677ccb49b-9v8ts\" (UID: \"f9c24a09-932f-41c9-8391-16b38a0d7ee6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:40.975309 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:40.975194 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:41.268916 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:41.268890 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts"] Sep 14 13:05:41.270864 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:05:41.270840 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c24a09_932f_41c9_8391_16b38a0d7ee6.slice/crio-63e110aac96c1d3fedaf5a840a07b83ba12bb54b4d3e4e1903d459081287c6b5 WatchSource:0}: Error finding container 63e110aac96c1d3fedaf5a840a07b83ba12bb54b4d3e4e1903d459081287c6b5: Status 404 returned error can't find the container with id 63e110aac96c1d3fedaf5a840a07b83ba12bb54b4d3e4e1903d459081287c6b5 Sep 14 13:05:41.900384 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:41.900353 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" event={"ID":"f9c24a09-932f-41c9-8391-16b38a0d7ee6","Type":"ContainerStarted","Data":"63e110aac96c1d3fedaf5a840a07b83ba12bb54b4d3e4e1903d459081287c6b5"} Sep 14 13:05:43.873233 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:43.873187 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:05:43.873551 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:43.873270 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:05:44.917723 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:44.917667 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" event={"ID":"f9c24a09-932f-41c9-8391-16b38a0d7ee6","Type":"ContainerStarted","Data":"150249f0c8745492eb7886d8332e6eec1251fa6c6f0978190fe43cca21df9bff"} Sep 14 13:05:44.918134 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:44.917906 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:44.919674 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:44.919652 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" Sep 14 13:05:44.940555 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:44.940509 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9v8ts" podStartSLOduration=2.340373042 podStartE2EDuration="4.940490678s" totalImagesPullingTime="2.600117636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:05:44.937492486 +0000 UTC m=+445.600388155" watchObservedRunningTime="2026-09-14 13:05:44.940490678 +0000 UTC m=+445.603386310" Sep 14 13:05:49.889775 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:05:49.889743 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-rwxn9" Sep 14 13:06:38.760817 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.760780 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp"] Sep 14 13:06:38.768209 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.768191 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp" Sep 14 13:06:38.771243 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.771218 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 14 13:06:38.771859 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.771835 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 14 13:06:38.771859 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.771857 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cq4js\"" Sep 14 13:06:38.773207 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.773182 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp"] Sep 14 13:06:38.856206 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.856173 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk"] Sep 14 13:06:38.858920 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.858905 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk" Sep 14 13:06:38.860632 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.860612 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp\" (UID: \"5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp" Sep 14 13:06:38.860740 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.860660 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp\" (UID: \"5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp" Sep 14 13:06:38.860783 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.860770 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wht5n\" (UniqueName: \"kubernetes.io/projected/5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a-kube-api-access-wht5n\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp\" (UID: \"5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp" Sep 14 13:06:38.871124 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.871096 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk"] Sep 14 13:06:38.961378 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.961341 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp\" (UID: \"5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp" Sep 14 13:06:38.961570 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.961422 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cb0b8bf-2bba-4d60-b864-845682217375-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk\" (UID: \"1cb0b8bf-2bba-4d60-b864-845682217375\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk" Sep 14 13:06:38.961570 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.961454 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wht5n\" (UniqueName: \"kubernetes.io/projected/5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a-kube-api-access-wht5n\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp\" (UID: \"5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp" Sep 14 13:06:38.961570 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.961512 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6sch\" (UniqueName: \"kubernetes.io/projected/1cb0b8bf-2bba-4d60-b864-845682217375-kube-api-access-k6sch\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk\" (UID: \"1cb0b8bf-2bba-4d60-b864-845682217375\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk" Sep 14 13:06:38.961570 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.961555 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cb0b8bf-2bba-4d60-b864-845682217375-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk\" (UID: \"1cb0b8bf-2bba-4d60-b864-845682217375\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk" Sep 14 13:06:38.961820 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.961616 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp\" (UID: \"5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp" Sep 14 13:06:38.961820 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.961806 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp\" (UID: \"5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp" Sep 14 13:06:38.962183 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.962162 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp\" (UID: \"5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp" Sep 14 13:06:38.964285 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.964266 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr"] Sep 14 13:06:38.967391 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.967377 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr" Sep 14 13:06:38.970018 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.969999 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wht5n\" (UniqueName: \"kubernetes.io/projected/5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a-kube-api-access-wht5n\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp\" (UID: \"5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp" Sep 14 13:06:38.975622 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:38.975603 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr"] Sep 14 13:06:39.061686 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.061607 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5"] Sep 14 13:06:39.062524 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.062499 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a19197df-04bc-4399-9bd1-9b37a62edc13-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr\" (UID: \"a19197df-04bc-4399-9bd1-9b37a62edc13\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr" Sep 14 13:06:39.062642 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.062620 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cb0b8bf-2bba-4d60-b864-845682217375-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk\" (UID: \"1cb0b8bf-2bba-4d60-b864-845682217375\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk" Sep 14 13:06:39.062686 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.062662 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prkcw\" (UniqueName: \"kubernetes.io/projected/a19197df-04bc-4399-9bd1-9b37a62edc13-kube-api-access-prkcw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr\" (UID: \"a19197df-04bc-4399-9bd1-9b37a62edc13\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr" Sep 14 13:06:39.062723 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.062685 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6sch\" (UniqueName: \"kubernetes.io/projected/1cb0b8bf-2bba-4d60-b864-845682217375-kube-api-access-k6sch\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk\" (UID: \"1cb0b8bf-2bba-4d60-b864-845682217375\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk" Sep 14 13:06:39.062723 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.062706 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a19197df-04bc-4399-9bd1-9b37a62edc13-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr\" (UID: \"a19197df-04bc-4399-9bd1-9b37a62edc13\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr" Sep 14 13:06:39.062827 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.062766 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cb0b8bf-2bba-4d60-b864-845682217375-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk\" (UID: \"1cb0b8bf-2bba-4d60-b864-845682217375\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk" Sep 14 13:06:39.063084 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.063062 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cb0b8bf-2bba-4d60-b864-845682217375-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk\" (UID: \"1cb0b8bf-2bba-4d60-b864-845682217375\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk" Sep 14 13:06:39.063159 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.063118 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cb0b8bf-2bba-4d60-b864-845682217375-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk\" (UID: \"1cb0b8bf-2bba-4d60-b864-845682217375\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk" Sep 14 13:06:39.064622 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.064608 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5" Sep 14 13:06:39.072017 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.071995 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6sch\" (UniqueName: \"kubernetes.io/projected/1cb0b8bf-2bba-4d60-b864-845682217375-kube-api-access-k6sch\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk\" (UID: \"1cb0b8bf-2bba-4d60-b864-845682217375\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk" Sep 14 13:06:39.075056 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.075033 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5"] Sep 14 13:06:39.079174 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.079156 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp" Sep 14 13:06:39.163836 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.163801 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prkcw\" (UniqueName: \"kubernetes.io/projected/a19197df-04bc-4399-9bd1-9b37a62edc13-kube-api-access-prkcw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr\" (UID: \"a19197df-04bc-4399-9bd1-9b37a62edc13\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr" Sep 14 13:06:39.164009 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.163858 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a19197df-04bc-4399-9bd1-9b37a62edc13-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr\" (UID: \"a19197df-04bc-4399-9bd1-9b37a62edc13\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr" Sep 14 13:06:39.164009 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.163899 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31ca7983-67de-4dfd-908b-8a485e4165d9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5\" (UID: \"31ca7983-67de-4dfd-908b-8a485e4165d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5" Sep 14 13:06:39.164009 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.163977 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn6kg\" (UniqueName: \"kubernetes.io/projected/31ca7983-67de-4dfd-908b-8a485e4165d9-kube-api-access-fn6kg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5\" (UID: \"31ca7983-67de-4dfd-908b-8a485e4165d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5" Sep 14 13:06:39.164139 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.164020 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a19197df-04bc-4399-9bd1-9b37a62edc13-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr\" (UID: \"a19197df-04bc-4399-9bd1-9b37a62edc13\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr" Sep 14 13:06:39.164298 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.164280 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a19197df-04bc-4399-9bd1-9b37a62edc13-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr\" (UID: \"a19197df-04bc-4399-9bd1-9b37a62edc13\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr" Sep 14 13:06:39.164350 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.164335 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31ca7983-67de-4dfd-908b-8a485e4165d9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5\" (UID: \"31ca7983-67de-4dfd-908b-8a485e4165d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5" Sep 14 13:06:39.164430 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.164409 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a19197df-04bc-4399-9bd1-9b37a62edc13-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr\" (UID: \"a19197df-04bc-4399-9bd1-9b37a62edc13\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr" Sep 14 13:06:39.173245 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.173205 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prkcw\" (UniqueName: \"kubernetes.io/projected/a19197df-04bc-4399-9bd1-9b37a62edc13-kube-api-access-prkcw\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr\" (UID: \"a19197df-04bc-4399-9bd1-9b37a62edc13\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr" Sep 14 13:06:39.175457 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.175435 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk" Sep 14 13:06:39.265639 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.265607 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31ca7983-67de-4dfd-908b-8a485e4165d9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5\" (UID: \"31ca7983-67de-4dfd-908b-8a485e4165d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5" Sep 14 13:06:39.265799 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.265652 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn6kg\" (UniqueName: \"kubernetes.io/projected/31ca7983-67de-4dfd-908b-8a485e4165d9-kube-api-access-fn6kg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5\" (UID: \"31ca7983-67de-4dfd-908b-8a485e4165d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5" Sep 14 13:06:39.265799 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.265718 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31ca7983-67de-4dfd-908b-8a485e4165d9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5\" (UID: \"31ca7983-67de-4dfd-908b-8a485e4165d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5" Sep 14 13:06:39.266034 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.266011 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31ca7983-67de-4dfd-908b-8a485e4165d9-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5\" (UID: \"31ca7983-67de-4dfd-908b-8a485e4165d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5" Sep 14 13:06:39.266078 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.266040 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31ca7983-67de-4dfd-908b-8a485e4165d9-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5\" (UID: \"31ca7983-67de-4dfd-908b-8a485e4165d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5" Sep 14 13:06:39.274818 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.274796 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn6kg\" (UniqueName: \"kubernetes.io/projected/31ca7983-67de-4dfd-908b-8a485e4165d9-kube-api-access-fn6kg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5\" (UID: \"31ca7983-67de-4dfd-908b-8a485e4165d9\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5" Sep 14 13:06:39.285523 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.285501 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr" Sep 14 13:06:39.358358 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.358315 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp"] Sep 14 13:06:39.360953 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:06:39.360914 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5284fffc_29cd_41c8_b5c8_9eaebf7e6d0a.slice/crio-7abfa4e4cc272e6dfd99eecd6760cc4f41023e28bb75bae596449424c09e5672 WatchSource:0}: Error finding container 7abfa4e4cc272e6dfd99eecd6760cc4f41023e28bb75bae596449424c09e5672: Status 404 returned error can't find the container with id 7abfa4e4cc272e6dfd99eecd6760cc4f41023e28bb75bae596449424c09e5672 Sep 14 13:06:39.382123 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.382101 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5" Sep 14 13:06:39.450556 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.450531 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk"] Sep 14 13:06:39.453058 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:06:39.453033 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cb0b8bf_2bba_4d60_b864_845682217375.slice/crio-1ef5595b74a1358a50223acbf4ed2b964a1985a18c39a6f2d2c49b31053b902d WatchSource:0}: Error finding container 1ef5595b74a1358a50223acbf4ed2b964a1985a18c39a6f2d2c49b31053b902d: Status 404 returned error can't find the container with id 1ef5595b74a1358a50223acbf4ed2b964a1985a18c39a6f2d2c49b31053b902d Sep 14 13:06:39.559681 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.556871 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr"] Sep 14 13:06:39.560282 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:06:39.560246 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19197df_04bc_4399_9bd1_9b37a62edc13.slice/crio-e6c459b78b3291a31c21f25afca32f19d189de168601d50410be27e9dc0a8271 WatchSource:0}: Error finding container e6c459b78b3291a31c21f25afca32f19d189de168601d50410be27e9dc0a8271: Status 404 returned error can't find the container with id e6c459b78b3291a31c21f25afca32f19d189de168601d50410be27e9dc0a8271 Sep 14 13:06:39.655754 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:39.655726 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5"] Sep 14 13:06:39.662915 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:06:39.662891 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ca7983_67de_4dfd_908b_8a485e4165d9.slice/crio-15d613c2c88754d12fd220f716265eae420eec6f8f0bd69059ec03c74befacfa WatchSource:0}: Error finding container 15d613c2c88754d12fd220f716265eae420eec6f8f0bd69059ec03c74befacfa: Status 404 returned error can't find the container with id 15d613c2c88754d12fd220f716265eae420eec6f8f0bd69059ec03c74befacfa Sep 14 13:06:40.196220 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:40.196184 2564 generic.go:356] "Generic (PLEG): container finished" podID="31ca7983-67de-4dfd-908b-8a485e4165d9" containerID="e2ca0696b48e81e065455e6822dd14c57382dcc7768967fcb8b192ba861d3255" exitCode=0 Sep 14 13:06:40.196689 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:40.196274 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5" event={"ID":"31ca7983-67de-4dfd-908b-8a485e4165d9","Type":"ContainerDied","Data":"e2ca0696b48e81e065455e6822dd14c57382dcc7768967fcb8b192ba861d3255"} Sep 14 13:06:40.196689 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:40.196335 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5" event={"ID":"31ca7983-67de-4dfd-908b-8a485e4165d9","Type":"ContainerStarted","Data":"15d613c2c88754d12fd220f716265eae420eec6f8f0bd69059ec03c74befacfa"} Sep 14 13:06:40.197885 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:40.197863 2564 generic.go:356] "Generic (PLEG): container finished" podID="a19197df-04bc-4399-9bd1-9b37a62edc13" containerID="7da2981bb54589a3edf4113b3e942d0b275d43b62a77192a78929e2b211ff725" exitCode=0 Sep 14 13:06:40.197995 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:40.197970 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr" event={"ID":"a19197df-04bc-4399-9bd1-9b37a62edc13","Type":"ContainerDied","Data":"7da2981bb54589a3edf4113b3e942d0b275d43b62a77192a78929e2b211ff725"} Sep 14 13:06:40.198067 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:40.198021 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr" event={"ID":"a19197df-04bc-4399-9bd1-9b37a62edc13","Type":"ContainerStarted","Data":"e6c459b78b3291a31c21f25afca32f19d189de168601d50410be27e9dc0a8271"} Sep 14 13:06:40.199640 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:40.199619 2564 generic.go:356] "Generic (PLEG): container finished" podID="5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a" containerID="802542f9a42508ffced841d16a1b332ce571a70d120fcf3aadfd91ee33e55dc5" exitCode=0 Sep 14 13:06:40.199729 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:40.199691 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp" event={"ID":"5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a","Type":"ContainerDied","Data":"802542f9a42508ffced841d16a1b332ce571a70d120fcf3aadfd91ee33e55dc5"} Sep 14 13:06:40.199788 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:40.199748 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp" event={"ID":"5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a","Type":"ContainerStarted","Data":"7abfa4e4cc272e6dfd99eecd6760cc4f41023e28bb75bae596449424c09e5672"} Sep 14 13:06:40.201350 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:40.201331 2564 generic.go:356] "Generic (PLEG): container finished" podID="1cb0b8bf-2bba-4d60-b864-845682217375" containerID="1b1253beb0391bde8694afa7af623492ded55cddacae3f1850fbc9434368198f" exitCode=0 Sep 14 13:06:40.201453 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:40.201380 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk" event={"ID":"1cb0b8bf-2bba-4d60-b864-845682217375","Type":"ContainerDied","Data":"1b1253beb0391bde8694afa7af623492ded55cddacae3f1850fbc9434368198f"} Sep 14 13:06:40.201453 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:40.201412 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk" event={"ID":"1cb0b8bf-2bba-4d60-b864-845682217375","Type":"ContainerStarted","Data":"1ef5595b74a1358a50223acbf4ed2b964a1985a18c39a6f2d2c49b31053b902d"} Sep 14 13:06:41.209535 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:41.209497 2564 generic.go:356] "Generic (PLEG): container finished" podID="5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a" containerID="58924458316ed867dc4a90bb1853aed82c5a6959210f50acb48553f57091018f" exitCode=0 Sep 14 13:06:41.209999 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:41.209577 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp" event={"ID":"5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a","Type":"ContainerDied","Data":"58924458316ed867dc4a90bb1853aed82c5a6959210f50acb48553f57091018f"} Sep 14 13:06:41.211624 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:41.211601 2564 generic.go:356] "Generic (PLEG): container finished" podID="1cb0b8bf-2bba-4d60-b864-845682217375" containerID="2af7228c5c6617ec82e22a5fe83f862c56741d46cec8c6eced1b137f13fe2c39" exitCode=0 Sep 14 13:06:41.211753 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:41.211723 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk" event={"ID":"1cb0b8bf-2bba-4d60-b864-845682217375","Type":"ContainerDied","Data":"2af7228c5c6617ec82e22a5fe83f862c56741d46cec8c6eced1b137f13fe2c39"} Sep 14 13:06:41.213883 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:41.213858 2564 generic.go:356] "Generic (PLEG): container finished" podID="31ca7983-67de-4dfd-908b-8a485e4165d9" containerID="0f7de62d671e6584a846e5faa56649630bef101a3f81d6da3fabfeb3e64dc5c2" exitCode=0 Sep 14 13:06:41.213979 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:41.213883 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5" event={"ID":"31ca7983-67de-4dfd-908b-8a485e4165d9","Type":"ContainerDied","Data":"0f7de62d671e6584a846e5faa56649630bef101a3f81d6da3fabfeb3e64dc5c2"} Sep 14 13:06:41.216191 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:41.216170 2564 generic.go:356] "Generic (PLEG): container finished" podID="a19197df-04bc-4399-9bd1-9b37a62edc13" containerID="375d2bf6746d96e67f60f0629c89973459e4f9f84ed2d86a3a18f489417d5784" exitCode=0 Sep 14 13:06:41.216277 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:41.216219 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr" event={"ID":"a19197df-04bc-4399-9bd1-9b37a62edc13","Type":"ContainerDied","Data":"375d2bf6746d96e67f60f0629c89973459e4f9f84ed2d86a3a18f489417d5784"} Sep 14 13:06:42.223303 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:42.223263 2564 generic.go:356] "Generic (PLEG): container finished" podID="31ca7983-67de-4dfd-908b-8a485e4165d9" containerID="79c64f2e443c33017a27e961e893fb6f2ab82fadc37b5b12c17e5f00c3c5b81d" exitCode=0 Sep 14 13:06:42.223739 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:42.223356 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5" event={"ID":"31ca7983-67de-4dfd-908b-8a485e4165d9","Type":"ContainerDied","Data":"79c64f2e443c33017a27e961e893fb6f2ab82fadc37b5b12c17e5f00c3c5b81d"} Sep 14 13:06:42.225419 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:42.225394 2564 generic.go:356] "Generic (PLEG): container finished" podID="a19197df-04bc-4399-9bd1-9b37a62edc13" containerID="821e49ac6001d6942093e8a2ab73abf74cee7496dc6dae460f92458460d21ea8" exitCode=0 Sep 14 13:06:42.225532 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:42.225469 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr" event={"ID":"a19197df-04bc-4399-9bd1-9b37a62edc13","Type":"ContainerDied","Data":"821e49ac6001d6942093e8a2ab73abf74cee7496dc6dae460f92458460d21ea8"} Sep 14 13:06:42.227638 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:42.227616 2564 generic.go:356] "Generic (PLEG): container finished" podID="5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a" containerID="a4b3f08c36582827ebc53ddd909c8d7a08ce814b57c8f6658e124ebdbfdc2c7e" exitCode=0 Sep 14 13:06:42.227748 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:42.227689 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp" event={"ID":"5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a","Type":"ContainerDied","Data":"a4b3f08c36582827ebc53ddd909c8d7a08ce814b57c8f6658e124ebdbfdc2c7e"} Sep 14 13:06:42.229680 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:42.229661 2564 generic.go:356] "Generic (PLEG): container finished" podID="1cb0b8bf-2bba-4d60-b864-845682217375" containerID="c35f068929095689a97bfdaaf284f40fd04e762e548b6eed0e15fca5a308d002" exitCode=0 Sep 14 13:06:42.229761 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:42.229690 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk" event={"ID":"1cb0b8bf-2bba-4d60-b864-845682217375","Type":"ContainerDied","Data":"c35f068929095689a97bfdaaf284f40fd04e762e548b6eed0e15fca5a308d002"} Sep 14 13:06:43.402543 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.402520 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr" Sep 14 13:06:43.406097 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.406076 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk" Sep 14 13:06:43.435894 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.435872 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp" Sep 14 13:06:43.440199 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.440173 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5" Sep 14 13:06:43.502332 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.502251 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31ca7983-67de-4dfd-908b-8a485e4165d9-util\") pod \"31ca7983-67de-4dfd-908b-8a485e4165d9\" (UID: \"31ca7983-67de-4dfd-908b-8a485e4165d9\") " Sep 14 13:06:43.502332 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.502289 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a19197df-04bc-4399-9bd1-9b37a62edc13-util\") pod \"a19197df-04bc-4399-9bd1-9b37a62edc13\" (UID: \"a19197df-04bc-4399-9bd1-9b37a62edc13\") " Sep 14 13:06:43.502332 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.502307 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a19197df-04bc-4399-9bd1-9b37a62edc13-bundle\") pod \"a19197df-04bc-4399-9bd1-9b37a62edc13\" (UID: \"a19197df-04bc-4399-9bd1-9b37a62edc13\") " Sep 14 13:06:43.502656 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.502363 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a-bundle\") pod \"5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a\" (UID: \"5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a\") " Sep 14 13:06:43.502656 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.502385 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wht5n\" (UniqueName: \"kubernetes.io/projected/5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a-kube-api-access-wht5n\") pod \"5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a\" (UID: \"5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a\") " Sep 14 13:06:43.502656 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.502398 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cb0b8bf-2bba-4d60-b864-845682217375-util\") pod \"1cb0b8bf-2bba-4d60-b864-845682217375\" (UID: \"1cb0b8bf-2bba-4d60-b864-845682217375\") " Sep 14 13:06:43.502794 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.502729 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prkcw\" (UniqueName: \"kubernetes.io/projected/a19197df-04bc-4399-9bd1-9b37a62edc13-kube-api-access-prkcw\") pod \"a19197df-04bc-4399-9bd1-9b37a62edc13\" (UID: \"a19197df-04bc-4399-9bd1-9b37a62edc13\") " Sep 14 13:06:43.502851 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.502795 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cb0b8bf-2bba-4d60-b864-845682217375-bundle\") pod \"1cb0b8bf-2bba-4d60-b864-845682217375\" (UID: \"1cb0b8bf-2bba-4d60-b864-845682217375\") " Sep 14 13:06:43.502910 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.502885 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a-util\") pod \"5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a\" (UID: \"5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a\") " Sep 14 13:06:43.502983 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.502914 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6sch\" (UniqueName: \"kubernetes.io/projected/1cb0b8bf-2bba-4d60-b864-845682217375-kube-api-access-k6sch\") pod \"1cb0b8bf-2bba-4d60-b864-845682217375\" (UID: \"1cb0b8bf-2bba-4d60-b864-845682217375\") " Sep 14 13:06:43.503035 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.502979 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fn6kg\" (UniqueName: \"kubernetes.io/projected/31ca7983-67de-4dfd-908b-8a485e4165d9-kube-api-access-fn6kg\") pod \"31ca7983-67de-4dfd-908b-8a485e4165d9\" (UID: \"31ca7983-67de-4dfd-908b-8a485e4165d9\") " Sep 14 13:06:43.503035 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.503012 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31ca7983-67de-4dfd-908b-8a485e4165d9-bundle\") pod \"31ca7983-67de-4dfd-908b-8a485e4165d9\" (UID: \"31ca7983-67de-4dfd-908b-8a485e4165d9\") " Sep 14 13:06:43.503825 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.503287 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19197df-04bc-4399-9bd1-9b37a62edc13-bundle" (OuterVolumeSpecName: "bundle") pod "a19197df-04bc-4399-9bd1-9b37a62edc13" (UID: "a19197df-04bc-4399-9bd1-9b37a62edc13"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:06:43.503825 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.503348 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a-bundle" (OuterVolumeSpecName: "bundle") pod "5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a" (UID: "5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:06:43.504586 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.504543 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ca7983-67de-4dfd-908b-8a485e4165d9-bundle" (OuterVolumeSpecName: "bundle") pod "31ca7983-67de-4dfd-908b-8a485e4165d9" (UID: "31ca7983-67de-4dfd-908b-8a485e4165d9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:06:43.505207 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.505170 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb0b8bf-2bba-4d60-b864-845682217375-bundle" (OuterVolumeSpecName: "bundle") pod "1cb0b8bf-2bba-4d60-b864-845682217375" (UID: "1cb0b8bf-2bba-4d60-b864-845682217375"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:06:43.505734 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.505672 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1cb0b8bf-2bba-4d60-b864-845682217375-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:06:43.505734 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.505698 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31ca7983-67de-4dfd-908b-8a485e4165d9-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:06:43.505734 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.505710 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a19197df-04bc-4399-9bd1-9b37a62edc13-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:06:43.505734 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.505725 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a-bundle\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:06:43.508377 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.508346 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19197df-04bc-4399-9bd1-9b37a62edc13-kube-api-access-prkcw" (OuterVolumeSpecName: "kube-api-access-prkcw") pod "a19197df-04bc-4399-9bd1-9b37a62edc13" (UID: "a19197df-04bc-4399-9bd1-9b37a62edc13"). InnerVolumeSpecName "kube-api-access-prkcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:06:43.511454 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.511430 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a-util" (OuterVolumeSpecName: "util") pod "5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a" (UID: "5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:06:43.512121 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.512095 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ca7983-67de-4dfd-908b-8a485e4165d9-util" (OuterVolumeSpecName: "util") pod "31ca7983-67de-4dfd-908b-8a485e4165d9" (UID: "31ca7983-67de-4dfd-908b-8a485e4165d9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:06:43.512494 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.512466 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ca7983-67de-4dfd-908b-8a485e4165d9-kube-api-access-fn6kg" (OuterVolumeSpecName: "kube-api-access-fn6kg") pod "31ca7983-67de-4dfd-908b-8a485e4165d9" (UID: "31ca7983-67de-4dfd-908b-8a485e4165d9"). InnerVolumeSpecName "kube-api-access-fn6kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:06:43.512663 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.512625 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a19197df-04bc-4399-9bd1-9b37a62edc13-util" (OuterVolumeSpecName: "util") pod "a19197df-04bc-4399-9bd1-9b37a62edc13" (UID: "a19197df-04bc-4399-9bd1-9b37a62edc13"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:06:43.515781 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.515752 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a-kube-api-access-wht5n" (OuterVolumeSpecName: "kube-api-access-wht5n") pod "5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a" (UID: "5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a"). InnerVolumeSpecName "kube-api-access-wht5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:06:43.515781 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.515772 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cb0b8bf-2bba-4d60-b864-845682217375-kube-api-access-k6sch" (OuterVolumeSpecName: "kube-api-access-k6sch") pod "1cb0b8bf-2bba-4d60-b864-845682217375" (UID: "1cb0b8bf-2bba-4d60-b864-845682217375"). InnerVolumeSpecName "kube-api-access-k6sch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:06:43.516471 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.516450 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cb0b8bf-2bba-4d60-b864-845682217375-util" (OuterVolumeSpecName: "util") pod "1cb0b8bf-2bba-4d60-b864-845682217375" (UID: "1cb0b8bf-2bba-4d60-b864-845682217375"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:06:43.606886 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.606857 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wht5n\" (UniqueName: \"kubernetes.io/projected/5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a-kube-api-access-wht5n\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:06:43.606886 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.606882 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1cb0b8bf-2bba-4d60-b864-845682217375-util\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:06:43.606886 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.606891 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prkcw\" (UniqueName: \"kubernetes.io/projected/a19197df-04bc-4399-9bd1-9b37a62edc13-kube-api-access-prkcw\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:06:43.607085 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.606900 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a-util\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:06:43.607085 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.606909 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6sch\" (UniqueName: \"kubernetes.io/projected/1cb0b8bf-2bba-4d60-b864-845682217375-kube-api-access-k6sch\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:06:43.607085 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.606918 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn6kg\" (UniqueName: \"kubernetes.io/projected/31ca7983-67de-4dfd-908b-8a485e4165d9-kube-api-access-fn6kg\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:06:43.607085 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.606945 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31ca7983-67de-4dfd-908b-8a485e4165d9-util\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:06:43.607085 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:43.606953 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a19197df-04bc-4399-9bd1-9b37a62edc13-util\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:06:44.242592 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:44.242561 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5" Sep 14 13:06:44.242870 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:44.242556 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9ggx5" event={"ID":"31ca7983-67de-4dfd-908b-8a485e4165d9","Type":"ContainerDied","Data":"15d613c2c88754d12fd220f716265eae420eec6f8f0bd69059ec03c74befacfa"} Sep 14 13:06:44.242870 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:44.242689 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15d613c2c88754d12fd220f716265eae420eec6f8f0bd69059ec03c74befacfa" Sep 14 13:06:44.244746 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:44.244720 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr" Sep 14 13:06:44.244865 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:44.244754 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7v7jxr" event={"ID":"a19197df-04bc-4399-9bd1-9b37a62edc13","Type":"ContainerDied","Data":"e6c459b78b3291a31c21f25afca32f19d189de168601d50410be27e9dc0a8271"} Sep 14 13:06:44.244865 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:44.244794 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6c459b78b3291a31c21f25afca32f19d189de168601d50410be27e9dc0a8271" Sep 14 13:06:44.247232 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:44.247210 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp" event={"ID":"5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a","Type":"ContainerDied","Data":"7abfa4e4cc272e6dfd99eecd6760cc4f41023e28bb75bae596449424c09e5672"} Sep 14 13:06:44.247232 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:44.247223 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ckdmcp" Sep 14 13:06:44.247416 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:44.247252 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7abfa4e4cc272e6dfd99eecd6760cc4f41023e28bb75bae596449424c09e5672" Sep 14 13:06:44.249465 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:44.249442 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk" event={"ID":"1cb0b8bf-2bba-4d60-b864-845682217375","Type":"ContainerDied","Data":"1ef5595b74a1358a50223acbf4ed2b964a1985a18c39a6f2d2c49b31053b902d"} Sep 14 13:06:44.249557 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:44.249482 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ef5595b74a1358a50223acbf4ed2b964a1985a18c39a6f2d2c49b31053b902d" Sep 14 13:06:44.249557 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:06:44.249449 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c68wfgk" Sep 14 13:08:19.799170 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:19.799140 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/1.log" Sep 14 13:08:19.800246 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:19.800227 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/1.log" Sep 14 13:08:57.160716 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.160679 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf"] Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161061 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cb0b8bf-2bba-4d60-b864-845682217375" containerName="extract" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161073 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cb0b8bf-2bba-4d60-b864-845682217375" containerName="extract" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161088 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cb0b8bf-2bba-4d60-b864-845682217375" containerName="pull" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161096 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cb0b8bf-2bba-4d60-b864-845682217375" containerName="pull" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161108 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cb0b8bf-2bba-4d60-b864-845682217375" containerName="util" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161114 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cb0b8bf-2bba-4d60-b864-845682217375" containerName="util" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161122 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ca7983-67de-4dfd-908b-8a485e4165d9" containerName="extract" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161127 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ca7983-67de-4dfd-908b-8a485e4165d9" containerName="extract" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161134 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a19197df-04bc-4399-9bd1-9b37a62edc13" containerName="extract" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161141 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19197df-04bc-4399-9bd1-9b37a62edc13" containerName="extract" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161147 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a" containerName="extract" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161152 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a" containerName="extract" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161163 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ca7983-67de-4dfd-908b-8a485e4165d9" containerName="util" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161171 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ca7983-67de-4dfd-908b-8a485e4165d9" containerName="util" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161177 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ca7983-67de-4dfd-908b-8a485e4165d9" containerName="pull" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161182 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ca7983-67de-4dfd-908b-8a485e4165d9" containerName="pull" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161190 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a" containerName="util" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161195 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a" containerName="util" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161200 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a19197df-04bc-4399-9bd1-9b37a62edc13" containerName="pull" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161204 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19197df-04bc-4399-9bd1-9b37a62edc13" containerName="pull" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161213 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a19197df-04bc-4399-9bd1-9b37a62edc13" containerName="util" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161217 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19197df-04bc-4399-9bd1-9b37a62edc13" containerName="util" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161222 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a" containerName="pull" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161228 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a" containerName="pull" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161295 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a19197df-04bc-4399-9bd1-9b37a62edc13" containerName="extract" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161305 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ca7983-67de-4dfd-908b-8a485e4165d9" containerName="extract" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161314 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5284fffc-29cd-41c8-b5c8-9eaebf7e6d0a" containerName="extract" Sep 14 13:08:57.163077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.161327 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cb0b8bf-2bba-4d60-b864-845682217375" containerName="extract" Sep 14 13:08:57.164130 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.164115 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" Sep 14 13:08:57.168201 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.168172 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 14 13:08:57.168354 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.168218 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 14 13:08:57.168354 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.168235 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 14 13:08:57.169555 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.169535 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 14 13:08:57.169693 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.169553 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kgb2c\"" Sep 14 13:08:57.176911 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.176881 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf"] Sep 14 13:08:57.199016 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.198994 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03c18206-36fd-43ca-93e5-866794d9360d-maas-api-tls\") pod \"maas-api-577bb5997d-9mnnf\" (UID: \"03c18206-36fd-43ca-93e5-866794d9360d\") " pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" Sep 14 13:08:57.199130 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.199027 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c18206-36fd-43ca-93e5-866794d9360d-openshift-service-ca\") pod \"maas-api-577bb5997d-9mnnf\" (UID: \"03c18206-36fd-43ca-93e5-866794d9360d\") " pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" Sep 14 13:08:57.199130 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.199047 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w9mx\" (UniqueName: \"kubernetes.io/projected/03c18206-36fd-43ca-93e5-866794d9360d-kube-api-access-9w9mx\") pod \"maas-api-577bb5997d-9mnnf\" (UID: \"03c18206-36fd-43ca-93e5-866794d9360d\") " pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" Sep 14 13:08:57.199204 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.199139 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c18206-36fd-43ca-93e5-866794d9360d-metrics-certs\") pod \"maas-api-577bb5997d-9mnnf\" (UID: \"03c18206-36fd-43ca-93e5-866794d9360d\") " pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" Sep 14 13:08:57.299782 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.299747 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c18206-36fd-43ca-93e5-866794d9360d-metrics-certs\") pod \"maas-api-577bb5997d-9mnnf\" (UID: \"03c18206-36fd-43ca-93e5-866794d9360d\") " pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" Sep 14 13:08:57.299995 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.299820 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03c18206-36fd-43ca-93e5-866794d9360d-maas-api-tls\") pod \"maas-api-577bb5997d-9mnnf\" (UID: \"03c18206-36fd-43ca-93e5-866794d9360d\") " pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" Sep 14 13:08:57.299995 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.299840 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c18206-36fd-43ca-93e5-866794d9360d-openshift-service-ca\") pod \"maas-api-577bb5997d-9mnnf\" (UID: \"03c18206-36fd-43ca-93e5-866794d9360d\") " pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" Sep 14 13:08:57.299995 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.299861 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w9mx\" (UniqueName: \"kubernetes.io/projected/03c18206-36fd-43ca-93e5-866794d9360d-kube-api-access-9w9mx\") pod \"maas-api-577bb5997d-9mnnf\" (UID: \"03c18206-36fd-43ca-93e5-866794d9360d\") " pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" Sep 14 13:08:57.300575 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.300550 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c18206-36fd-43ca-93e5-866794d9360d-openshift-service-ca\") pod \"maas-api-577bb5997d-9mnnf\" (UID: \"03c18206-36fd-43ca-93e5-866794d9360d\") " pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" Sep 14 13:08:57.302318 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.302295 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03c18206-36fd-43ca-93e5-866794d9360d-maas-api-tls\") pod \"maas-api-577bb5997d-9mnnf\" (UID: \"03c18206-36fd-43ca-93e5-866794d9360d\") " pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" Sep 14 13:08:57.302410 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.302347 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c18206-36fd-43ca-93e5-866794d9360d-metrics-certs\") pod \"maas-api-577bb5997d-9mnnf\" (UID: \"03c18206-36fd-43ca-93e5-866794d9360d\") " pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" Sep 14 13:08:57.310213 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.310190 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w9mx\" (UniqueName: \"kubernetes.io/projected/03c18206-36fd-43ca-93e5-866794d9360d-kube-api-access-9w9mx\") pod \"maas-api-577bb5997d-9mnnf\" (UID: \"03c18206-36fd-43ca-93e5-866794d9360d\") " pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" Sep 14 13:08:57.475185 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.475101 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" Sep 14 13:08:57.757684 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.757660 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf"] Sep 14 13:08:57.759412 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:08:57.759386 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c18206_36fd_43ca_93e5_866794d9360d.slice/crio-ada5b1fb3604a829619e92612a2f8b682437b9b4d2b0a1ebd7dee52d3acacecd WatchSource:0}: Error finding container ada5b1fb3604a829619e92612a2f8b682437b9b4d2b0a1ebd7dee52d3acacecd: Status 404 returned error can't find the container with id ada5b1fb3604a829619e92612a2f8b682437b9b4d2b0a1ebd7dee52d3acacecd Sep 14 13:08:57.760746 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.760730 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 13:08:57.978241 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:08:57.978208 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" event={"ID":"03c18206-36fd-43ca-93e5-866794d9360d","Type":"ContainerStarted","Data":"ada5b1fb3604a829619e92612a2f8b682437b9b4d2b0a1ebd7dee52d3acacecd"} Sep 14 13:09:01.000444 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:01.000399 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" event={"ID":"03c18206-36fd-43ca-93e5-866794d9360d","Type":"ContainerStarted","Data":"4ec72721733e05ddd2ab433764c40de57e8efeb99a4fe5103f830197209e08e9"} Sep 14 13:09:01.000884 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:01.000530 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" Sep 14 13:09:01.032901 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:01.032850 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" podStartSLOduration=1.497643858 podStartE2EDuration="4.032835649s" totalImagesPullingTime="2.535191791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:08:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:09:01.031754358 +0000 UTC m=+641.694650000" watchObservedRunningTime="2026-09-14 13:09:01.032835649 +0000 UTC m=+641.695731272" Sep 14 13:09:07.011247 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:07.011219 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" Sep 14 13:09:29.796443 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:29.796411 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8"] Sep 14 13:09:29.802809 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:29.802786 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:29.806025 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:29.806001 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 14 13:09:29.807520 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:29.807068 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4m98s\"" Sep 14 13:09:29.807520 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:29.807143 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 14 13:09:29.807718 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:29.807561 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 14 13:09:29.814953 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:29.814912 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8"] Sep 14 13:09:29.998436 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:29.998408 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpgdk\" (UniqueName: \"kubernetes.io/projected/f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be-kube-api-access-cpgdk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v5wj8\" (UID: \"f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:29.998636 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:29.998531 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v5wj8\" (UID: \"f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:29.998636 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:29.998563 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v5wj8\" (UID: \"f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:29.998636 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:29.998590 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v5wj8\" (UID: \"f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:29.998636 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:29.998618 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v5wj8\" (UID: \"f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:29.998898 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:29.998677 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v5wj8\" (UID: \"f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:30.099993 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:30.099874 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpgdk\" (UniqueName: \"kubernetes.io/projected/f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be-kube-api-access-cpgdk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v5wj8\" (UID: \"f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:30.100149 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:30.100008 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v5wj8\" (UID: \"f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:30.100149 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:30.100043 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v5wj8\" (UID: \"f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:30.100149 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:30.100060 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v5wj8\" (UID: \"f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:30.100149 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:30.100079 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v5wj8\" (UID: \"f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:30.100149 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:30.100111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v5wj8\" (UID: \"f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:30.100462 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:30.100439 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v5wj8\" (UID: \"f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:30.100530 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:30.100474 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v5wj8\" (UID: \"f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:30.100530 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:30.100498 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v5wj8\" (UID: \"f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:30.102471 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:30.102449 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v5wj8\" (UID: \"f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:30.102644 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:30.102625 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v5wj8\" (UID: \"f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:30.108773 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:30.108749 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpgdk\" (UniqueName: \"kubernetes.io/projected/f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be-kube-api-access-cpgdk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-v5wj8\" (UID: \"f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:30.116654 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:30.116632 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:30.390764 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:30.390736 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8"] Sep 14 13:09:30.391995 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:09:30.391969 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6bb71f4_9da1_40e0_8e6c_c3bcdd7924be.slice/crio-a1d813a91e38aa05102b5aee5810d42d43d53764a57be99d38783a3d7490f649 WatchSource:0}: Error finding container a1d813a91e38aa05102b5aee5810d42d43d53764a57be99d38783a3d7490f649: Status 404 returned error can't find the container with id a1d813a91e38aa05102b5aee5810d42d43d53764a57be99d38783a3d7490f649 Sep 14 13:09:31.157136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:31.157102 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" event={"ID":"f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be","Type":"ContainerStarted","Data":"a1d813a91e38aa05102b5aee5810d42d43d53764a57be99d38783a3d7490f649"} Sep 14 13:09:32.164629 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:32.164591 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" event={"ID":"f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be","Type":"ContainerStarted","Data":"8e75823c446871bf2bbf94d0623bdfcbe529e2a8f1ac9ec0eec8240875c20d8d"} Sep 14 13:09:32.182005 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:32.181947 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" podStartSLOduration=2.207085215 podStartE2EDuration="3.181912239s" totalImagesPullingTime="974.827024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:09:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:09:32.181052997 +0000 UTC m=+672.843948624" watchObservedRunningTime="2026-09-14 13:09:32.181912239 +0000 UTC m=+672.844807906" Sep 14 13:09:40.116981 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.116916 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:40.117428 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.117021 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:40.129642 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.129619 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:40.218670 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.218643 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-v5wj8" Sep 14 13:09:40.404623 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.404541 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95"] Sep 14 13:09:40.408230 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.408208 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:40.410785 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.410764 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 14 13:09:40.419752 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.419723 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95"] Sep 14 13:09:40.498000 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.497970 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9fedfc1-1cbc-4c41-b99f-1f7ae335771d-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95\" (UID: \"f9fedfc1-1cbc-4c41-b99f-1f7ae335771d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:40.498199 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.498016 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbgmk\" (UniqueName: \"kubernetes.io/projected/f9fedfc1-1cbc-4c41-b99f-1f7ae335771d-kube-api-access-bbgmk\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95\" (UID: \"f9fedfc1-1cbc-4c41-b99f-1f7ae335771d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:40.498199 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.498103 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9fedfc1-1cbc-4c41-b99f-1f7ae335771d-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95\" (UID: \"f9fedfc1-1cbc-4c41-b99f-1f7ae335771d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:40.498199 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.498149 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9fedfc1-1cbc-4c41-b99f-1f7ae335771d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95\" (UID: \"f9fedfc1-1cbc-4c41-b99f-1f7ae335771d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:40.498199 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.498179 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9fedfc1-1cbc-4c41-b99f-1f7ae335771d-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95\" (UID: \"f9fedfc1-1cbc-4c41-b99f-1f7ae335771d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:40.498390 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.498212 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9fedfc1-1cbc-4c41-b99f-1f7ae335771d-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95\" (UID: \"f9fedfc1-1cbc-4c41-b99f-1f7ae335771d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:40.599417 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.599385 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9fedfc1-1cbc-4c41-b99f-1f7ae335771d-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95\" (UID: \"f9fedfc1-1cbc-4c41-b99f-1f7ae335771d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:40.599417 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.599422 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbgmk\" (UniqueName: \"kubernetes.io/projected/f9fedfc1-1cbc-4c41-b99f-1f7ae335771d-kube-api-access-bbgmk\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95\" (UID: \"f9fedfc1-1cbc-4c41-b99f-1f7ae335771d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:40.599663 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.599465 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9fedfc1-1cbc-4c41-b99f-1f7ae335771d-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95\" (UID: \"f9fedfc1-1cbc-4c41-b99f-1f7ae335771d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:40.599663 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.599484 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9fedfc1-1cbc-4c41-b99f-1f7ae335771d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95\" (UID: \"f9fedfc1-1cbc-4c41-b99f-1f7ae335771d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:40.599663 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.599502 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9fedfc1-1cbc-4c41-b99f-1f7ae335771d-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95\" (UID: \"f9fedfc1-1cbc-4c41-b99f-1f7ae335771d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:40.599663 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.599633 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9fedfc1-1cbc-4c41-b99f-1f7ae335771d-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95\" (UID: \"f9fedfc1-1cbc-4c41-b99f-1f7ae335771d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:40.599902 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.599819 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9fedfc1-1cbc-4c41-b99f-1f7ae335771d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95\" (UID: \"f9fedfc1-1cbc-4c41-b99f-1f7ae335771d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:40.599902 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.599864 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9fedfc1-1cbc-4c41-b99f-1f7ae335771d-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95\" (UID: \"f9fedfc1-1cbc-4c41-b99f-1f7ae335771d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:40.600036 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.600015 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9fedfc1-1cbc-4c41-b99f-1f7ae335771d-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95\" (UID: \"f9fedfc1-1cbc-4c41-b99f-1f7ae335771d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:40.602048 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.602021 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9fedfc1-1cbc-4c41-b99f-1f7ae335771d-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95\" (UID: \"f9fedfc1-1cbc-4c41-b99f-1f7ae335771d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:40.602177 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.602096 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9fedfc1-1cbc-4c41-b99f-1f7ae335771d-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95\" (UID: \"f9fedfc1-1cbc-4c41-b99f-1f7ae335771d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:40.607942 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.607905 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbgmk\" (UniqueName: \"kubernetes.io/projected/f9fedfc1-1cbc-4c41-b99f-1f7ae335771d-kube-api-access-bbgmk\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95\" (UID: \"f9fedfc1-1cbc-4c41-b99f-1f7ae335771d\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:40.720899 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.720817 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:40.995027 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:40.994999 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95"] Sep 14 13:09:40.996854 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:09:40.996829 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9fedfc1_1cbc_4c41_b99f_1f7ae335771d.slice/crio-218fff0abd6058c4b9c55f6b1e92c9a2a75ed4a99395c7997a264c11e0710939 WatchSource:0}: Error finding container 218fff0abd6058c4b9c55f6b1e92c9a2a75ed4a99395c7997a264c11e0710939: Status 404 returned error can't find the container with id 218fff0abd6058c4b9c55f6b1e92c9a2a75ed4a99395c7997a264c11e0710939 Sep 14 13:09:41.214525 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:41.214490 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" event={"ID":"f9fedfc1-1cbc-4c41-b99f-1f7ae335771d","Type":"ContainerStarted","Data":"218fff0abd6058c4b9c55f6b1e92c9a2a75ed4a99395c7997a264c11e0710939"} Sep 14 13:09:42.231986 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:42.231945 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" event={"ID":"f9fedfc1-1cbc-4c41-b99f-1f7ae335771d","Type":"ContainerStarted","Data":"62406c880812e292e10c3a238bebd2bd0964eebabb8e0756d0cd426c228cd444"} Sep 14 13:09:42.252221 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:42.252171 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" podStartSLOduration=1.9651661059999999 podStartE2EDuration="2.252156145s" totalImagesPullingTime="286.990039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:09:42.251132111 +0000 UTC m=+682.914027743" watchObservedRunningTime="2026-09-14 13:09:42.252156145 +0000 UTC m=+682.915051775" Sep 14 13:09:50.721414 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:50.721383 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:50.721875 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:50.721491 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:50.734120 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:50.734098 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:09:51.291353 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:09:51.291328 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95" Sep 14 13:10:27.824502 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:27.824424 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d"] Sep 14 13:10:27.828188 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:27.828166 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:27.832357 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:27.832340 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 14 13:10:27.841547 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:27.841526 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d"] Sep 14 13:10:27.930496 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:27.930464 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6df2760-f30e-4166-81fb-9351aa871b6e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9tf5d\" (UID: \"c6df2760-f30e-4166-81fb-9351aa871b6e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:27.930496 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:27.930506 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6df2760-f30e-4166-81fb-9351aa871b6e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9tf5d\" (UID: \"c6df2760-f30e-4166-81fb-9351aa871b6e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:27.930722 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:27.930539 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6df2760-f30e-4166-81fb-9351aa871b6e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9tf5d\" (UID: \"c6df2760-f30e-4166-81fb-9351aa871b6e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:27.930722 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:27.930563 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8dz5\" (UniqueName: \"kubernetes.io/projected/c6df2760-f30e-4166-81fb-9351aa871b6e-kube-api-access-b8dz5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9tf5d\" (UID: \"c6df2760-f30e-4166-81fb-9351aa871b6e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:27.930722 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:27.930608 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6df2760-f30e-4166-81fb-9351aa871b6e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9tf5d\" (UID: \"c6df2760-f30e-4166-81fb-9351aa871b6e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:27.930722 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:27.930668 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6df2760-f30e-4166-81fb-9351aa871b6e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9tf5d\" (UID: \"c6df2760-f30e-4166-81fb-9351aa871b6e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:28.031552 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:28.031515 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6df2760-f30e-4166-81fb-9351aa871b6e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9tf5d\" (UID: \"c6df2760-f30e-4166-81fb-9351aa871b6e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:28.031552 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:28.031554 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8dz5\" (UniqueName: \"kubernetes.io/projected/c6df2760-f30e-4166-81fb-9351aa871b6e-kube-api-access-b8dz5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9tf5d\" (UID: \"c6df2760-f30e-4166-81fb-9351aa871b6e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:28.031812 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:28.031592 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6df2760-f30e-4166-81fb-9351aa871b6e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9tf5d\" (UID: \"c6df2760-f30e-4166-81fb-9351aa871b6e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:28.031812 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:28.031628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6df2760-f30e-4166-81fb-9351aa871b6e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9tf5d\" (UID: \"c6df2760-f30e-4166-81fb-9351aa871b6e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:28.031812 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:28.031660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6df2760-f30e-4166-81fb-9351aa871b6e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9tf5d\" (UID: \"c6df2760-f30e-4166-81fb-9351aa871b6e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:28.031812 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:28.031684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6df2760-f30e-4166-81fb-9351aa871b6e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9tf5d\" (UID: \"c6df2760-f30e-4166-81fb-9351aa871b6e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:28.032057 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:28.031959 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6df2760-f30e-4166-81fb-9351aa871b6e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9tf5d\" (UID: \"c6df2760-f30e-4166-81fb-9351aa871b6e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:28.032057 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:28.032017 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6df2760-f30e-4166-81fb-9351aa871b6e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9tf5d\" (UID: \"c6df2760-f30e-4166-81fb-9351aa871b6e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:28.032134 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:28.032060 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6df2760-f30e-4166-81fb-9351aa871b6e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9tf5d\" (UID: \"c6df2760-f30e-4166-81fb-9351aa871b6e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:28.034140 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:28.034116 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6df2760-f30e-4166-81fb-9351aa871b6e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9tf5d\" (UID: \"c6df2760-f30e-4166-81fb-9351aa871b6e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:28.034267 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:28.034252 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6df2760-f30e-4166-81fb-9351aa871b6e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9tf5d\" (UID: \"c6df2760-f30e-4166-81fb-9351aa871b6e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:28.041004 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:28.040981 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8dz5\" (UniqueName: \"kubernetes.io/projected/c6df2760-f30e-4166-81fb-9351aa871b6e-kube-api-access-b8dz5\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-9tf5d\" (UID: \"c6df2760-f30e-4166-81fb-9351aa871b6e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:28.138046 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:28.138019 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:28.417969 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:28.417944 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d"] Sep 14 13:10:28.419842 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:10:28.419816 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6df2760_f30e_4166_81fb_9351aa871b6e.slice/crio-a55b9fd6c0474e5c6db256abb988a7d03df936b8c2460ff2454242b0a7c124ef WatchSource:0}: Error finding container a55b9fd6c0474e5c6db256abb988a7d03df936b8c2460ff2454242b0a7c124ef: Status 404 returned error can't find the container with id a55b9fd6c0474e5c6db256abb988a7d03df936b8c2460ff2454242b0a7c124ef Sep 14 13:10:28.489020 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:28.488988 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" event={"ID":"c6df2760-f30e-4166-81fb-9351aa871b6e","Type":"ContainerStarted","Data":"a55b9fd6c0474e5c6db256abb988a7d03df936b8c2460ff2454242b0a7c124ef"} Sep 14 13:10:29.496039 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:29.496004 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" event={"ID":"c6df2760-f30e-4166-81fb-9351aa871b6e","Type":"ContainerStarted","Data":"7c99e2bed17e52695d4b5bd163f43b4133f59bc1ebe30b6941afac05d4f7c8fb"} Sep 14 13:10:29.519703 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:29.519650 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" podStartSLOduration=2.310968248 podStartE2EDuration="2.519635173s" totalImagesPullingTime="208.666925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:10:29.517796346 +0000 UTC m=+730.180691977" watchObservedRunningTime="2026-09-14 13:10:29.519635173 +0000 UTC m=+730.182530803" Sep 14 13:10:38.138572 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:38.138525 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:38.139080 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:38.138599 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:38.150807 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:38.150784 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:10:38.558619 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:10:38.558538 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-9tf5d" Sep 14 13:12:17.416294 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.416255 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml"] Sep 14 13:12:17.422148 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.422120 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.425023 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.424998 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-bc851d-openshift-default-dockercfg-rpbxt\"" Sep 14 13:12:17.431662 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.431639 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml"] Sep 14 13:12:17.468241 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.468207 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-envoy\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.468431 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.468259 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-token\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.468431 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.468336 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-workload-socket\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.468431 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.468369 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-workload-certs\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.468431 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.468406 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-data\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.468431 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.468425 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k8q2\" (UniqueName: \"kubernetes.io/projected/6ad76071-ccde-4279-a47a-3753c2a4b0e9-kube-api-access-5k8q2\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.468726 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.468529 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-credential-socket\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.468726 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.468560 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istiod-ca-cert\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.468726 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.468635 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-podinfo\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.569318 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.569277 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-podinfo\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.569318 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.569324 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-envoy\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.569557 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.569346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-token\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.569557 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.569380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-workload-socket\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.569557 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.569396 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-workload-certs\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.569557 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.569423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-data\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.569557 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.569438 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k8q2\" (UniqueName: \"kubernetes.io/projected/6ad76071-ccde-4279-a47a-3753c2a4b0e9-kube-api-access-5k8q2\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.569557 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.569496 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-credential-socket\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.569557 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.569530 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istiod-ca-cert\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.570242 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.570211 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-data\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.570363 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.570256 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-workload-socket\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.570363 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.570293 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-credential-socket\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.570363 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.570306 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istiod-ca-cert\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.570548 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.570528 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-workload-certs\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.572368 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.572343 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-envoy\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.572472 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.572447 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-podinfo\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.585197 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.585171 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k8q2\" (UniqueName: \"kubernetes.io/projected/6ad76071-ccde-4279-a47a-3753c2a4b0e9-kube-api-access-5k8q2\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.588784 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.588760 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-token\") pod \"e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:17.735008 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:17.734911 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:18.017345 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:18.017315 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml"] Sep 14 13:12:18.019206 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:12:18.019178 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad76071_ccde_4279_a47a_3753c2a4b0e9.slice/crio-2d7cd9bcb66ab4b5940b9d6ff1968f3deae6ca7ea86f227aa1c421a59227f5cb WatchSource:0}: Error finding container 2d7cd9bcb66ab4b5940b9d6ff1968f3deae6ca7ea86f227aa1c421a59227f5cb: Status 404 returned error can't find the container with id 2d7cd9bcb66ab4b5940b9d6ff1968f3deae6ca7ea86f227aa1c421a59227f5cb Sep 14 13:12:18.021489 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:18.021456 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:12:18.021585 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:18.021539 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:12:18.021649 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:18.021585 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:12:18.093226 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:18.093188 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" event={"ID":"6ad76071-ccde-4279-a47a-3753c2a4b0e9","Type":"ContainerStarted","Data":"1eb004706eddd0311e56e93d76bd106d2b2db21b1f8a65e044f3dbaf42cf70cb"} Sep 14 13:12:18.093338 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:18.093257 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" event={"ID":"6ad76071-ccde-4279-a47a-3753c2a4b0e9","Type":"ContainerStarted","Data":"2d7cd9bcb66ab4b5940b9d6ff1968f3deae6ca7ea86f227aa1c421a59227f5cb"} Sep 14 13:12:18.115737 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:18.115677 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" podStartSLOduration=1.115653362 podStartE2EDuration="1.115653362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:12:18.113248447 +0000 UTC m=+838.776144078" watchObservedRunningTime="2026-09-14 13:12:18.115653362 +0000 UTC m=+838.778548997" Sep 14 13:12:18.735911 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:18.735884 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:19.740347 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:19.740319 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:20.104762 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:20.104683 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:20.105596 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:20.105579 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:21.701583 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:21.701556 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln"] Sep 14 13:12:21.705413 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:21.705393 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" Sep 14 13:12:21.708174 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:21.708153 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-bc851d-serving-cert\"" Sep 14 13:12:21.716425 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:21.716402 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln"] Sep 14 13:12:21.811580 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:21.811550 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2da29be-f347-409b-9f93-f723ceb9deba-metrics-certs\") pod \"maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln\" (UID: \"e2da29be-f347-409b-9f93-f723ceb9deba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" Sep 14 13:12:21.811759 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:21.811613 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2da29be-f347-409b-9f93-f723ceb9deba-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln\" (UID: \"e2da29be-f347-409b-9f93-f723ceb9deba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" Sep 14 13:12:21.811759 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:21.811637 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2da29be-f347-409b-9f93-f723ceb9deba-maas-api-tls\") pod \"maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln\" (UID: \"e2da29be-f347-409b-9f93-f723ceb9deba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" Sep 14 13:12:21.811759 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:21.811659 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76fjq\" (UniqueName: \"kubernetes.io/projected/e2da29be-f347-409b-9f93-f723ceb9deba-kube-api-access-76fjq\") pod \"maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln\" (UID: \"e2da29be-f347-409b-9f93-f723ceb9deba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" Sep 14 13:12:21.912266 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:21.912224 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2da29be-f347-409b-9f93-f723ceb9deba-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln\" (UID: \"e2da29be-f347-409b-9f93-f723ceb9deba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" Sep 14 13:12:21.912266 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:21.912273 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2da29be-f347-409b-9f93-f723ceb9deba-maas-api-tls\") pod \"maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln\" (UID: \"e2da29be-f347-409b-9f93-f723ceb9deba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" Sep 14 13:12:21.912534 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:21.912306 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76fjq\" (UniqueName: \"kubernetes.io/projected/e2da29be-f347-409b-9f93-f723ceb9deba-kube-api-access-76fjq\") pod \"maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln\" (UID: \"e2da29be-f347-409b-9f93-f723ceb9deba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" Sep 14 13:12:21.912534 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:21.912367 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2da29be-f347-409b-9f93-f723ceb9deba-metrics-certs\") pod \"maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln\" (UID: \"e2da29be-f347-409b-9f93-f723ceb9deba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" Sep 14 13:12:21.912534 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:12:21.912392 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-serving-cert: secret "maas-api-e2e-worker-w0-bc851d-serving-cert" not found Sep 14 13:12:21.912534 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:12:21.912471 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2da29be-f347-409b-9f93-f723ceb9deba-maas-api-tls podName:e2da29be-f347-409b-9f93-f723ceb9deba nodeName:}" failed. No retries permitted until 2026-09-14 13:12:22.412450924 +0000 UTC m=+843.075346532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2da29be-f347-409b-9f93-f723ceb9deba-maas-api-tls") pod "maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" (UID: "e2da29be-f347-409b-9f93-f723ceb9deba") : secret "maas-api-e2e-worker-w0-bc851d-serving-cert" not found Sep 14 13:12:21.913007 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:21.912988 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2da29be-f347-409b-9f93-f723ceb9deba-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln\" (UID: \"e2da29be-f347-409b-9f93-f723ceb9deba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" Sep 14 13:12:21.914859 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:21.914836 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2da29be-f347-409b-9f93-f723ceb9deba-metrics-certs\") pod \"maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln\" (UID: \"e2da29be-f347-409b-9f93-f723ceb9deba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" Sep 14 13:12:21.921455 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:21.921434 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76fjq\" (UniqueName: \"kubernetes.io/projected/e2da29be-f347-409b-9f93-f723ceb9deba-kube-api-access-76fjq\") pod \"maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln\" (UID: \"e2da29be-f347-409b-9f93-f723ceb9deba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" Sep 14 13:12:22.417894 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:22.417857 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2da29be-f347-409b-9f93-f723ceb9deba-maas-api-tls\") pod \"maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln\" (UID: \"e2da29be-f347-409b-9f93-f723ceb9deba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" Sep 14 13:12:22.420400 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:22.420371 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2da29be-f347-409b-9f93-f723ceb9deba-maas-api-tls\") pod \"maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln\" (UID: \"e2da29be-f347-409b-9f93-f723ceb9deba\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" Sep 14 13:12:22.618494 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:22.618459 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" Sep 14 13:12:22.732462 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:22.732428 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r"] Sep 14 13:12:22.736469 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:22.736442 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" Sep 14 13:12:22.739419 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:22.739399 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-b3d6c5-serving-cert\"" Sep 14 13:12:22.754794 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:22.754774 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r"] Sep 14 13:12:22.820880 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:22.820846 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8s8j\" (UniqueName: \"kubernetes.io/projected/45f9dea1-c3e6-44b4-85fa-45009d40b328-kube-api-access-f8s8j\") pod \"maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r\" (UID: \"45f9dea1-c3e6-44b4-85fa-45009d40b328\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" Sep 14 13:12:22.820880 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:22.820886 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45f9dea1-c3e6-44b4-85fa-45009d40b328-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r\" (UID: \"45f9dea1-c3e6-44b4-85fa-45009d40b328\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" Sep 14 13:12:22.821123 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:22.820915 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45f9dea1-c3e6-44b4-85fa-45009d40b328-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r\" (UID: \"45f9dea1-c3e6-44b4-85fa-45009d40b328\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" Sep 14 13:12:22.821123 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:22.820965 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45f9dea1-c3e6-44b4-85fa-45009d40b328-metrics-certs\") pod \"maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r\" (UID: \"45f9dea1-c3e6-44b4-85fa-45009d40b328\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" Sep 14 13:12:22.899139 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:22.899116 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln"] Sep 14 13:12:22.900752 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:12:22.900729 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2da29be_f347_409b_9f93_f723ceb9deba.slice/crio-1561fdf69181a1cc898032ec9d22f97feceb5be91458b32e41990908fb7b4c40 WatchSource:0}: Error finding container 1561fdf69181a1cc898032ec9d22f97feceb5be91458b32e41990908fb7b4c40: Status 404 returned error can't find the container with id 1561fdf69181a1cc898032ec9d22f97feceb5be91458b32e41990908fb7b4c40 Sep 14 13:12:22.922653 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:22.922577 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45f9dea1-c3e6-44b4-85fa-45009d40b328-metrics-certs\") pod \"maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r\" (UID: \"45f9dea1-c3e6-44b4-85fa-45009d40b328\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" Sep 14 13:12:22.922758 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:22.922714 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8s8j\" (UniqueName: \"kubernetes.io/projected/45f9dea1-c3e6-44b4-85fa-45009d40b328-kube-api-access-f8s8j\") pod \"maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r\" (UID: \"45f9dea1-c3e6-44b4-85fa-45009d40b328\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" Sep 14 13:12:22.922758 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:22.922751 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45f9dea1-c3e6-44b4-85fa-45009d40b328-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r\" (UID: \"45f9dea1-c3e6-44b4-85fa-45009d40b328\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" Sep 14 13:12:22.922850 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:22.922785 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45f9dea1-c3e6-44b4-85fa-45009d40b328-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r\" (UID: \"45f9dea1-c3e6-44b4-85fa-45009d40b328\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" Sep 14 13:12:22.922897 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:12:22.922887 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-serving-cert: secret "maas-api-e2e-worker-w4-b3d6c5-serving-cert" not found Sep 14 13:12:22.922999 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:12:22.922983 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45f9dea1-c3e6-44b4-85fa-45009d40b328-maas-api-tls podName:45f9dea1-c3e6-44b4-85fa-45009d40b328 nodeName:}" failed. No retries permitted until 2026-09-14 13:12:23.422962214 +0000 UTC m=+844.085857825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45f9dea1-c3e6-44b4-85fa-45009d40b328-maas-api-tls") pod "maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" (UID: "45f9dea1-c3e6-44b4-85fa-45009d40b328") : secret "maas-api-e2e-worker-w4-b3d6c5-serving-cert" not found Sep 14 13:12:22.923472 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:22.923456 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45f9dea1-c3e6-44b4-85fa-45009d40b328-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r\" (UID: \"45f9dea1-c3e6-44b4-85fa-45009d40b328\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" Sep 14 13:12:22.925226 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:22.925207 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45f9dea1-c3e6-44b4-85fa-45009d40b328-metrics-certs\") pod \"maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r\" (UID: \"45f9dea1-c3e6-44b4-85fa-45009d40b328\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" Sep 14 13:12:22.932537 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:22.932512 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8s8j\" (UniqueName: \"kubernetes.io/projected/45f9dea1-c3e6-44b4-85fa-45009d40b328-kube-api-access-f8s8j\") pod \"maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r\" (UID: \"45f9dea1-c3e6-44b4-85fa-45009d40b328\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" Sep 14 13:12:23.122894 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:23.122863 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" event={"ID":"e2da29be-f347-409b-9f93-f723ceb9deba","Type":"ContainerStarted","Data":"1561fdf69181a1cc898032ec9d22f97feceb5be91458b32e41990908fb7b4c40"} Sep 14 13:12:23.427909 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:23.427865 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45f9dea1-c3e6-44b4-85fa-45009d40b328-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r\" (UID: \"45f9dea1-c3e6-44b4-85fa-45009d40b328\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" Sep 14 13:12:23.430552 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:23.430528 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45f9dea1-c3e6-44b4-85fa-45009d40b328-maas-api-tls\") pod \"maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r\" (UID: \"45f9dea1-c3e6-44b4-85fa-45009d40b328\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" Sep 14 13:12:23.633444 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:23.633410 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l"] Sep 14 13:12:23.638013 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:23.637990 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" Sep 14 13:12:23.644868 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:23.644632 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-975227-serving-cert\"" Sep 14 13:12:23.649104 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:23.648883 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" Sep 14 13:12:23.649341 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:23.649282 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l"] Sep 14 13:12:23.733046 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:23.732978 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmq9c\" (UniqueName: \"kubernetes.io/projected/b7c4b47c-464e-4710-a6c0-06f58cf25bde-kube-api-access-fmq9c\") pod \"maas-api-e2e-inf-a-975227-55f756664-zlt6l\" (UID: \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" Sep 14 13:12:23.733516 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:23.733173 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c4b47c-464e-4710-a6c0-06f58cf25bde-metrics-certs\") pod \"maas-api-e2e-inf-a-975227-55f756664-zlt6l\" (UID: \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" Sep 14 13:12:23.733516 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:23.733220 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c4b47c-464e-4710-a6c0-06f58cf25bde-openshift-service-ca\") pod \"maas-api-e2e-inf-a-975227-55f756664-zlt6l\" (UID: \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" Sep 14 13:12:23.733516 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:23.733256 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7c4b47c-464e-4710-a6c0-06f58cf25bde-maas-api-tls\") pod \"maas-api-e2e-inf-a-975227-55f756664-zlt6l\" (UID: \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" Sep 14 13:12:23.834085 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:23.834055 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c4b47c-464e-4710-a6c0-06f58cf25bde-metrics-certs\") pod \"maas-api-e2e-inf-a-975227-55f756664-zlt6l\" (UID: \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" Sep 14 13:12:23.834253 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:23.834097 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c4b47c-464e-4710-a6c0-06f58cf25bde-openshift-service-ca\") pod \"maas-api-e2e-inf-a-975227-55f756664-zlt6l\" (UID: \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" Sep 14 13:12:23.834253 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:23.834119 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7c4b47c-464e-4710-a6c0-06f58cf25bde-maas-api-tls\") pod \"maas-api-e2e-inf-a-975227-55f756664-zlt6l\" (UID: \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" Sep 14 13:12:23.834253 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:23.834180 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmq9c\" (UniqueName: \"kubernetes.io/projected/b7c4b47c-464e-4710-a6c0-06f58cf25bde-kube-api-access-fmq9c\") pod \"maas-api-e2e-inf-a-975227-55f756664-zlt6l\" (UID: \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" Sep 14 13:12:23.834386 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:12:23.834282 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-serving-cert: secret "maas-api-e2e-inf-a-975227-serving-cert" not found Sep 14 13:12:23.834386 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:12:23.834357 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7c4b47c-464e-4710-a6c0-06f58cf25bde-maas-api-tls podName:b7c4b47c-464e-4710-a6c0-06f58cf25bde nodeName:}" failed. No retries permitted until 2026-09-14 13:12:24.334335854 +0000 UTC m=+844.997231464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b7c4b47c-464e-4710-a6c0-06f58cf25bde-maas-api-tls") pod "maas-api-e2e-inf-a-975227-55f756664-zlt6l" (UID: "b7c4b47c-464e-4710-a6c0-06f58cf25bde") : secret "maas-api-e2e-inf-a-975227-serving-cert" not found Sep 14 13:12:23.834879 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:23.834861 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c4b47c-464e-4710-a6c0-06f58cf25bde-openshift-service-ca\") pod \"maas-api-e2e-inf-a-975227-55f756664-zlt6l\" (UID: \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" Sep 14 13:12:23.836632 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:23.836614 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c4b47c-464e-4710-a6c0-06f58cf25bde-metrics-certs\") pod \"maas-api-e2e-inf-a-975227-55f756664-zlt6l\" (UID: \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" Sep 14 13:12:23.850818 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:23.850790 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmq9c\" (UniqueName: \"kubernetes.io/projected/b7c4b47c-464e-4710-a6c0-06f58cf25bde-kube-api-access-fmq9c\") pod \"maas-api-e2e-inf-a-975227-55f756664-zlt6l\" (UID: \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" Sep 14 13:12:23.927144 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:23.927117 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r"] Sep 14 13:12:23.928096 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:12:23.928075 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f9dea1_c3e6_44b4_85fa_45009d40b328.slice/crio-4812074965af9a56e8db91154bdb2b35041b36ccd5b097b7639018123ada6f2a WatchSource:0}: Error finding container 4812074965af9a56e8db91154bdb2b35041b36ccd5b097b7639018123ada6f2a: Status 404 returned error can't find the container with id 4812074965af9a56e8db91154bdb2b35041b36ccd5b097b7639018123ada6f2a Sep 14 13:12:24.131901 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:24.131861 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" event={"ID":"e2da29be-f347-409b-9f93-f723ceb9deba","Type":"ContainerStarted","Data":"e6a714e7a86398317a832b91534d8032ec40da2024e30e72fb9ba21218e2d470"} Sep 14 13:12:24.132111 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:24.131979 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" Sep 14 13:12:24.133254 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:24.133230 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" event={"ID":"45f9dea1-c3e6-44b4-85fa-45009d40b328","Type":"ContainerStarted","Data":"4812074965af9a56e8db91154bdb2b35041b36ccd5b097b7639018123ada6f2a"} Sep 14 13:12:24.153434 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:24.153395 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" podStartSLOduration=2.5315558339999997 podStartE2EDuration="3.153382621s" totalImagesPullingTime="621.826787ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:12:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:12:24.152798356 +0000 UTC m=+844.815694010" watchObservedRunningTime="2026-09-14 13:12:24.153382621 +0000 UTC m=+844.816278251" Sep 14 13:12:24.339512 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:24.339477 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7c4b47c-464e-4710-a6c0-06f58cf25bde-maas-api-tls\") pod \"maas-api-e2e-inf-a-975227-55f756664-zlt6l\" (UID: \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" Sep 14 13:12:24.344196 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:24.344165 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7c4b47c-464e-4710-a6c0-06f58cf25bde-maas-api-tls\") pod \"maas-api-e2e-inf-a-975227-55f756664-zlt6l\" (UID: \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" Sep 14 13:12:24.560672 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:24.560593 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" Sep 14 13:12:24.837498 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:24.837466 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l"] Sep 14 13:12:24.839093 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:12:24.838941 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c4b47c_464e_4710_a6c0_06f58cf25bde.slice/crio-ed9878f958fea920b01d8918904d991db295396dc64bc7282d1b6a82f580b8f1 WatchSource:0}: Error finding container ed9878f958fea920b01d8918904d991db295396dc64bc7282d1b6a82f580b8f1: Status 404 returned error can't find the container with id ed9878f958fea920b01d8918904d991db295396dc64bc7282d1b6a82f580b8f1 Sep 14 13:12:25.140793 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:25.140758 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" event={"ID":"b7c4b47c-464e-4710-a6c0-06f58cf25bde","Type":"ContainerStarted","Data":"ed9878f958fea920b01d8918904d991db295396dc64bc7282d1b6a82f580b8f1"} Sep 14 13:12:25.142486 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:25.142457 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" event={"ID":"45f9dea1-c3e6-44b4-85fa-45009d40b328","Type":"ContainerStarted","Data":"f6e77b622b7a8985371463138e607c509dde3d8fed7d5b1d20539ae9de841df6"} Sep 14 13:12:25.162996 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:25.162950 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" podStartSLOduration=2.829511847 podStartE2EDuration="3.162908731s" totalImagesPullingTime="333.396884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:12:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:12:25.160986729 +0000 UTC m=+845.823882356" watchObservedRunningTime="2026-09-14 13:12:25.162908731 +0000 UTC m=+845.825804361" Sep 14 13:12:26.150149 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:26.150107 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" event={"ID":"b7c4b47c-464e-4710-a6c0-06f58cf25bde","Type":"ContainerStarted","Data":"628b1cc7cf405a9da4a4b42411b1ed19e0315221a25393299b60b6b75d133264"} Sep 14 13:12:26.150611 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:26.150179 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" Sep 14 13:12:26.150611 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:26.150255 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" Sep 14 13:12:26.168977 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:26.168896 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" podStartSLOduration=2.825108958 podStartE2EDuration="3.168879s" totalImagesPullingTime="343.770042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:12:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:12:26.16830222 +0000 UTC m=+846.831197873" watchObservedRunningTime="2026-09-14 13:12:26.168879 +0000 UTC m=+846.831774629" Sep 14 13:12:27.051262 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.051229 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh"] Sep 14 13:12:27.055206 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.055184 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.057903 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.057886 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-360610b9-openshift-default-dockercfg-nwmk5\"" Sep 14 13:12:27.067291 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.067267 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh"] Sep 14 13:12:27.167481 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.167451 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd7vn\" (UniqueName: \"kubernetes.io/projected/7fedd108-9c7b-4c77-983f-5fd972206f7f-kube-api-access-bd7vn\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.167481 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.167485 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-token\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.168019 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.167516 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-envoy\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.168019 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.167600 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-credential-socket\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.168019 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.167637 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-workload-certs\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.168019 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.167700 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-podinfo\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.168019 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.167739 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-workload-socket\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.168019 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.167789 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fedd108-9c7b-4c77-983f-5fd972206f7f-istiod-ca-cert\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.168019 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.167826 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-data\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.269052 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.269017 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd7vn\" (UniqueName: \"kubernetes.io/projected/7fedd108-9c7b-4c77-983f-5fd972206f7f-kube-api-access-bd7vn\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.269259 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.269058 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-token\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.269259 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.269105 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-envoy\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.269259 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.269153 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-credential-socket\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.269259 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.269175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-workload-certs\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.269495 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.269258 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-podinfo\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.269495 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.269336 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-workload-socket\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.269495 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.269403 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fedd108-9c7b-4c77-983f-5fd972206f7f-istiod-ca-cert\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.269495 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.269443 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-data\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.269777 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.269600 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-credential-socket\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.269777 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.269643 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-workload-certs\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.269961 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.269856 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-workload-socket\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.270217 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.270038 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-data\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.270217 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.270175 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fedd108-9c7b-4c77-983f-5fd972206f7f-istiod-ca-cert\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.271537 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.271515 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-envoy\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.271806 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.271790 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-podinfo\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.277914 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.277888 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-token\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.278199 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.278182 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd7vn\" (UniqueName: \"kubernetes.io/projected/7fedd108-9c7b-4c77-983f-5fd972206f7f-kube-api-access-bd7vn\") pod \"e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.366427 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.366392 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:27.653030 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:12:27.652995 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fedd108_9c7b_4c77_983f_5fd972206f7f.slice/crio-689cbd56b0d573a563c769cc6adff2c32fd244e350ba270c75ddb4ef5e1a8587 WatchSource:0}: Error finding container 689cbd56b0d573a563c769cc6adff2c32fd244e350ba270c75ddb4ef5e1a8587: Status 404 returned error can't find the container with id 689cbd56b0d573a563c769cc6adff2c32fd244e350ba270c75ddb4ef5e1a8587 Sep 14 13:12:27.653310 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.653288 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh"] Sep 14 13:12:27.655447 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.655388 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:12:27.655542 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.655476 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:12:27.655542 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:27.655505 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:12:28.164717 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:28.164680 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" event={"ID":"7fedd108-9c7b-4c77-983f-5fd972206f7f","Type":"ContainerStarted","Data":"003e9c79de060573701fd841913fcdd5898b3dc87050442253fd2ba69e09820a"} Sep 14 13:12:28.164902 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:28.164743 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" event={"ID":"7fedd108-9c7b-4c77-983f-5fd972206f7f","Type":"ContainerStarted","Data":"689cbd56b0d573a563c769cc6adff2c32fd244e350ba270c75ddb4ef5e1a8587"} Sep 14 13:12:28.189612 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:28.189562 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" podStartSLOduration=1.1895428510000001 podStartE2EDuration="1.189542851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:12:28.18780994 +0000 UTC m=+848.850705575" watchObservedRunningTime="2026-09-14 13:12:28.189542851 +0000 UTC m=+848.852438480" Sep 14 13:12:28.367239 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:28.367205 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:28.372021 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:28.371994 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:29.171700 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:29.171671 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:29.172537 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:29.172520 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:30.148392 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:30.148363 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" Sep 14 13:12:32.165302 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:32.165252 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" Sep 14 13:12:32.165793 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:32.165360 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" Sep 14 13:12:33.979263 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:33.979228 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf"] Sep 14 13:12:33.990446 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:33.990409 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf"] Sep 14 13:12:33.990617 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:33.990527 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" Sep 14 13:12:33.994618 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:33.994592 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-c14a5e-serving-cert\"" Sep 14 13:12:34.042955 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:34.042887 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c69c2d2a-858b-4c74-a5c1-751db5724efa-metrics-certs\") pod \"maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf\" (UID: \"c69c2d2a-858b-4c74-a5c1-751db5724efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" Sep 14 13:12:34.043142 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:34.043023 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-654st\" (UniqueName: \"kubernetes.io/projected/c69c2d2a-858b-4c74-a5c1-751db5724efa-kube-api-access-654st\") pod \"maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf\" (UID: \"c69c2d2a-858b-4c74-a5c1-751db5724efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" Sep 14 13:12:34.043142 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:34.043077 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69c2d2a-858b-4c74-a5c1-751db5724efa-openshift-service-ca\") pod \"maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf\" (UID: \"c69c2d2a-858b-4c74-a5c1-751db5724efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" Sep 14 13:12:34.043277 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:34.043183 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c69c2d2a-858b-4c74-a5c1-751db5724efa-maas-api-tls\") pod \"maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf\" (UID: \"c69c2d2a-858b-4c74-a5c1-751db5724efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" Sep 14 13:12:34.143777 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:34.143741 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c69c2d2a-858b-4c74-a5c1-751db5724efa-metrics-certs\") pod \"maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf\" (UID: \"c69c2d2a-858b-4c74-a5c1-751db5724efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" Sep 14 13:12:34.143999 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:34.143815 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-654st\" (UniqueName: \"kubernetes.io/projected/c69c2d2a-858b-4c74-a5c1-751db5724efa-kube-api-access-654st\") pod \"maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf\" (UID: \"c69c2d2a-858b-4c74-a5c1-751db5724efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" Sep 14 13:12:34.143999 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:34.143871 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69c2d2a-858b-4c74-a5c1-751db5724efa-openshift-service-ca\") pod \"maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf\" (UID: \"c69c2d2a-858b-4c74-a5c1-751db5724efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" Sep 14 13:12:34.143999 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:34.143977 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c69c2d2a-858b-4c74-a5c1-751db5724efa-maas-api-tls\") pod \"maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf\" (UID: \"c69c2d2a-858b-4c74-a5c1-751db5724efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" Sep 14 13:12:34.144191 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:12:34.144056 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-serving-cert: secret "maas-api-e2e-autores-c14a5e-serving-cert" not found Sep 14 13:12:34.144191 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:12:34.144130 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c69c2d2a-858b-4c74-a5c1-751db5724efa-maas-api-tls podName:c69c2d2a-858b-4c74-a5c1-751db5724efa nodeName:}" failed. No retries permitted until 2026-09-14 13:12:34.644113316 +0000 UTC m=+855.307008923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c69c2d2a-858b-4c74-a5c1-751db5724efa-maas-api-tls") pod "maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" (UID: "c69c2d2a-858b-4c74-a5c1-751db5724efa") : secret "maas-api-e2e-autores-c14a5e-serving-cert" not found Sep 14 13:12:34.144547 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:34.144526 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69c2d2a-858b-4c74-a5c1-751db5724efa-openshift-service-ca\") pod \"maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf\" (UID: \"c69c2d2a-858b-4c74-a5c1-751db5724efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" Sep 14 13:12:34.146355 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:34.146338 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c69c2d2a-858b-4c74-a5c1-751db5724efa-metrics-certs\") pod \"maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf\" (UID: \"c69c2d2a-858b-4c74-a5c1-751db5724efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" Sep 14 13:12:34.153760 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:34.153737 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-654st\" (UniqueName: \"kubernetes.io/projected/c69c2d2a-858b-4c74-a5c1-751db5724efa-kube-api-access-654st\") pod \"maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf\" (UID: \"c69c2d2a-858b-4c74-a5c1-751db5724efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" Sep 14 13:12:34.648823 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:34.648790 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c69c2d2a-858b-4c74-a5c1-751db5724efa-maas-api-tls\") pod \"maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf\" (UID: \"c69c2d2a-858b-4c74-a5c1-751db5724efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" Sep 14 13:12:34.651409 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:34.651383 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c69c2d2a-858b-4c74-a5c1-751db5724efa-maas-api-tls\") pod \"maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf\" (UID: \"c69c2d2a-858b-4c74-a5c1-751db5724efa\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" Sep 14 13:12:34.903918 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:34.903824 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" Sep 14 13:12:35.192032 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:35.191996 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf"] Sep 14 13:12:35.192512 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:12:35.192211 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc69c2d2a_858b_4c74_a5c1_751db5724efa.slice/crio-6b87ac3417ba1ea09ed600e5aac1aa16aad5cce9d8ca18c52ba44a7c31edc1a1 WatchSource:0}: Error finding container 6b87ac3417ba1ea09ed600e5aac1aa16aad5cce9d8ca18c52ba44a7c31edc1a1: Status 404 returned error can't find the container with id 6b87ac3417ba1ea09ed600e5aac1aa16aad5cce9d8ca18c52ba44a7c31edc1a1 Sep 14 13:12:35.218468 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:35.218433 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" event={"ID":"c69c2d2a-858b-4c74-a5c1-751db5724efa","Type":"ContainerStarted","Data":"6b87ac3417ba1ea09ed600e5aac1aa16aad5cce9d8ca18c52ba44a7c31edc1a1"} Sep 14 13:12:36.023754 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.023721 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2"] Sep 14 13:12:36.027500 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.027476 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.031493 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.031463 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cadd9c-openshift-default-dockercfg-lvlqr\"" Sep 14 13:12:36.051353 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.051324 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2"] Sep 14 13:12:36.146117 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.146075 2564 patch_prober.go:28] interesting pod/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 14 13:12:36.146300 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.146149 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" podUID="6ad76071-ccde-4279-a47a-3753c2a4b0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 14 13:12:36.166004 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.165965 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-envoy\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.166221 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.166013 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-token\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.166221 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.166049 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvlwl\" (UniqueName: \"kubernetes.io/projected/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-kube-api-access-hvlwl\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.166221 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.166088 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-credential-socket\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.166221 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.166109 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istiod-ca-cert\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.166221 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.166177 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-podinfo\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.166439 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.166230 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-workload-certs\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.166439 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.166264 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-workload-socket\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.166439 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.166317 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-data\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.226401 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.226353 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" event={"ID":"c69c2d2a-858b-4c74-a5c1-751db5724efa","Type":"ContainerStarted","Data":"4210994e1060391efe24290bfd44fb6cdf13fccfa42d8f4baf2b681afd07f681"} Sep 14 13:12:36.226809 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.226436 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" Sep 14 13:12:36.252385 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.252321 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" podStartSLOduration=2.89328814 podStartE2EDuration="3.252296004s" totalImagesPullingTime="359.007864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:12:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:12:36.25064688 +0000 UTC m=+856.913542508" watchObservedRunningTime="2026-09-14 13:12:36.252296004 +0000 UTC m=+856.915191635" Sep 14 13:12:36.267229 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.267190 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-envoy\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.267475 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.267251 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-token\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.267475 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.267297 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvlwl\" (UniqueName: \"kubernetes.io/projected/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-kube-api-access-hvlwl\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.267475 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.267345 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-credential-socket\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.267475 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.267369 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istiod-ca-cert\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.267475 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.267449 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-podinfo\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.267865 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.267509 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-workload-certs\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.267865 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.267547 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-workload-socket\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.267865 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.267600 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-data\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.268742 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.268211 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-workload-certs\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.268742 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.268396 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istiod-ca-cert\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.268742 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.268547 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-credential-socket\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.268742 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.268703 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-workload-socket\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.269065 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.268899 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-data\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.270542 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.270516 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-envoy\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.271384 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.271361 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-podinfo\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.281314 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.281200 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-token\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.281459 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.281437 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvlwl\" (UniqueName: \"kubernetes.io/projected/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-kube-api-access-hvlwl\") pod \"e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.341292 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.341255 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:36.632434 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.632408 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2"] Sep 14 13:12:36.633292 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:12:36.633269 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd30d61_45b8_4586_8612_2b2caeb8a1ec.slice/crio-d7d5c0579ca58ede4df397e10e6ef26152d39010029e6ac5ab2d542e8a64b4a2 WatchSource:0}: Error finding container d7d5c0579ca58ede4df397e10e6ef26152d39010029e6ac5ab2d542e8a64b4a2: Status 404 returned error can't find the container with id d7d5c0579ca58ede4df397e10e6ef26152d39010029e6ac5ab2d542e8a64b4a2 Sep 14 13:12:36.635800 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.635768 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:12:36.635870 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.635835 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:12:36.635870 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:36.635866 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:12:37.233959 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:37.233904 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" event={"ID":"0fd30d61-45b8-4586-8612-2b2caeb8a1ec","Type":"ContainerStarted","Data":"8ba43d861dda923ac88a1324931e7b1fac21bd170adbb536f178aa00c4125ba8"} Sep 14 13:12:37.234434 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:37.233978 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" event={"ID":"0fd30d61-45b8-4586-8612-2b2caeb8a1ec","Type":"ContainerStarted","Data":"d7d5c0579ca58ede4df397e10e6ef26152d39010029e6ac5ab2d542e8a64b4a2"} Sep 14 13:12:37.257464 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:37.257408 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" podStartSLOduration=2.257391042 podStartE2EDuration="2.257391042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:12:37.255552018 +0000 UTC m=+857.918447648" watchObservedRunningTime="2026-09-14 13:12:37.257391042 +0000 UTC m=+857.920286672" Sep 14 13:12:37.342145 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:37.342101 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:37.347294 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:37.347264 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:38.241218 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:38.241181 2564 generic.go:356] "Generic (PLEG): container finished" podID="6ad76071-ccde-4279-a47a-3753c2a4b0e9" containerID="1eb004706eddd0311e56e93d76bd106d2b2db21b1f8a65e044f3dbaf42cf70cb" exitCode=0 Sep 14 13:12:38.242138 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:38.241250 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" event={"ID":"6ad76071-ccde-4279-a47a-3753c2a4b0e9","Type":"ContainerDied","Data":"1eb004706eddd0311e56e93d76bd106d2b2db21b1f8a65e044f3dbaf42cf70cb"} Sep 14 13:12:38.242138 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:38.241713 2564 scope.go:117] "RemoveContainer" containerID="1eb004706eddd0311e56e93d76bd106d2b2db21b1f8a65e044f3dbaf42cf70cb" Sep 14 13:12:38.242138 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:38.241739 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:38.242616 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:38.242601 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:12:38.242891 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:38.242865 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:12:38.242965 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:38.242951 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:12:38.243007 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:38.242989 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:12:39.249998 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:39.249953 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" event={"ID":"6ad76071-ccde-4279-a47a-3753c2a4b0e9","Type":"ContainerStarted","Data":"c7ffafb0444e93a66177dc12132b903e864070e8932d18f1828512e433d41f25"} Sep 14 13:12:39.735998 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:39.735969 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:39.736166 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:39.736135 2564 patch_prober.go:28] interesting pod/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 14 13:12:39.736214 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:39.736181 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" podUID="6ad76071-ccde-4279-a47a-3753c2a4b0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 14 13:12:40.736011 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:40.735977 2564 patch_prober.go:28] interesting pod/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 14 13:12:40.736387 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:40.736034 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" podUID="6ad76071-ccde-4279-a47a-3753c2a4b0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 14 13:12:41.735596 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:41.735558 2564 patch_prober.go:28] interesting pod/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 14 13:12:41.735777 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:41.735616 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" podUID="6ad76071-ccde-4279-a47a-3753c2a4b0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 14 13:12:42.240698 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:42.240670 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" Sep 14 13:12:42.736160 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:42.736122 2564 patch_prober.go:28] interesting pod/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 14 13:12:42.736344 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:42.736196 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" podUID="6ad76071-ccde-4279-a47a-3753c2a4b0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 14 13:12:43.736164 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:43.736123 2564 patch_prober.go:28] interesting pod/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 14 13:12:43.736562 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:43.736178 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" podUID="6ad76071-ccde-4279-a47a-3753c2a4b0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 14 13:12:44.735277 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:44.735240 2564 patch_prober.go:28] interesting pod/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Sep 14 13:12:44.735477 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:44.735294 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" podUID="6ad76071-ccde-4279-a47a-3753c2a4b0e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Sep 14 13:12:45.739611 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:45.739581 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:45.740065 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:45.740043 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:45.740688 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:45.740668 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:12:49.345664 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:49.345570 2564 generic.go:356] "Generic (PLEG): container finished" podID="7fedd108-9c7b-4c77-983f-5fd972206f7f" containerID="003e9c79de060573701fd841913fcdd5898b3dc87050442253fd2ba69e09820a" exitCode=0 Sep 14 13:12:49.345664 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:49.345647 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" event={"ID":"7fedd108-9c7b-4c77-983f-5fd972206f7f","Type":"ContainerDied","Data":"003e9c79de060573701fd841913fcdd5898b3dc87050442253fd2ba69e09820a"} Sep 14 13:12:49.346253 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:49.346217 2564 scope.go:117] "RemoveContainer" containerID="003e9c79de060573701fd841913fcdd5898b3dc87050442253fd2ba69e09820a" Sep 14 13:12:49.347443 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:49.347416 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:12:49.347505 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:49.347485 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:12:49.347545 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:49.347517 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:12:50.371958 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:50.366055 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" event={"ID":"7fedd108-9c7b-4c77-983f-5fd972206f7f","Type":"ContainerStarted","Data":"52e475a061cbbbe381bda615b0a8e1223fce1e94a492663fb90c909ff3dfcfae"} Sep 14 13:12:50.371958 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:50.367218 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:50.371958 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:50.369043 2564 patch_prober.go:28] interesting pod/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 14 13:12:50.371958 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:50.369097 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" podUID="7fedd108-9c7b-4c77-983f-5fd972206f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 14 13:12:51.367444 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:51.367405 2564 patch_prober.go:28] interesting pod/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 14 13:12:51.374223 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:51.374184 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" podUID="7fedd108-9c7b-4c77-983f-5fd972206f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 14 13:12:52.367535 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:52.367497 2564 patch_prober.go:28] interesting pod/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 14 13:12:52.367854 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:52.367820 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" podUID="7fedd108-9c7b-4c77-983f-5fd972206f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 14 13:12:53.367163 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:53.367118 2564 patch_prober.go:28] interesting pod/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 14 13:12:53.367667 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:53.367224 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" podUID="7fedd108-9c7b-4c77-983f-5fd972206f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 14 13:12:54.367645 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:54.367609 2564 patch_prober.go:28] interesting pod/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 14 13:12:54.368270 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:54.368239 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" podUID="7fedd108-9c7b-4c77-983f-5fd972206f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 14 13:12:55.367393 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:55.367352 2564 patch_prober.go:28] interesting pod/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Sep 14 13:12:55.367682 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:55.367652 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" podUID="7fedd108-9c7b-4c77-983f-5fd972206f7f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Sep 14 13:12:56.375436 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:56.375396 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:56.376758 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:56.376731 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:12:56.377605 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:12:56.377583 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:13:00.881876 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:00.881837 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh"] Sep 14 13:13:00.882387 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:00.882128 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" podUID="7fedd108-9c7b-4c77-983f-5fd972206f7f" containerName="istio-proxy" containerID="cri-o://52e475a061cbbbe381bda615b0a8e1223fce1e94a492663fb90c909ff3dfcfae" gracePeriod=30 Sep 14 13:13:04.462069 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:04.462037 2564 generic.go:356] "Generic (PLEG): container finished" podID="0fd30d61-45b8-4586-8612-2b2caeb8a1ec" containerID="8ba43d861dda923ac88a1324931e7b1fac21bd170adbb536f178aa00c4125ba8" exitCode=0 Sep 14 13:13:04.462504 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:04.462088 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" event={"ID":"0fd30d61-45b8-4586-8612-2b2caeb8a1ec","Type":"ContainerDied","Data":"8ba43d861dda923ac88a1324931e7b1fac21bd170adbb536f178aa00c4125ba8"} Sep 14 13:13:04.462504 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:04.462497 2564 scope.go:117] "RemoveContainer" containerID="8ba43d861dda923ac88a1324931e7b1fac21bd170adbb536f178aa00c4125ba8" Sep 14 13:13:04.463691 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:04.463659 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:13:04.463800 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:04.463747 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:13:04.463800 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:04.463793 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:13:05.474802 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:05.474761 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" event={"ID":"0fd30d61-45b8-4586-8612-2b2caeb8a1ec","Type":"ContainerStarted","Data":"0bb0da257e51dc27f0b0d8ce9b92d8f0ce719fa35a24aed2b4986378c82ea43d"} Sep 14 13:13:05.699681 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:05.699650 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p"] Sep 14 13:13:05.702667 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:05.702652 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" Sep 14 13:13:05.705081 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:05.705060 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ea32f836-serving-cert\"" Sep 14 13:13:05.713694 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:05.713671 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p"] Sep 14 13:13:05.800662 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:05.800571 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-maas-api-tls\") pod \"maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p\" (UID: \"98dfaf91-60ca-4bef-ae98-d5f3266e32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" Sep 14 13:13:05.800662 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:05.800657 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-metrics-certs\") pod \"maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p\" (UID: \"98dfaf91-60ca-4bef-ae98-d5f3266e32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" Sep 14 13:13:05.800844 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:05.800678 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6plsk\" (UniqueName: \"kubernetes.io/projected/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-kube-api-access-6plsk\") pod \"maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p\" (UID: \"98dfaf91-60ca-4bef-ae98-d5f3266e32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" Sep 14 13:13:05.800844 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:05.800708 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-openshift-service-ca\") pod \"maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p\" (UID: \"98dfaf91-60ca-4bef-ae98-d5f3266e32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" Sep 14 13:13:05.902463 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:05.902393 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-metrics-certs\") pod \"maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p\" (UID: \"98dfaf91-60ca-4bef-ae98-d5f3266e32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" Sep 14 13:13:05.902463 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:05.902445 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6plsk\" (UniqueName: \"kubernetes.io/projected/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-kube-api-access-6plsk\") pod \"maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p\" (UID: \"98dfaf91-60ca-4bef-ae98-d5f3266e32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" Sep 14 13:13:05.902901 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:05.902501 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-openshift-service-ca\") pod \"maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p\" (UID: \"98dfaf91-60ca-4bef-ae98-d5f3266e32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" Sep 14 13:13:05.902901 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:05.902634 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-maas-api-tls\") pod \"maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p\" (UID: \"98dfaf91-60ca-4bef-ae98-d5f3266e32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" Sep 14 13:13:05.904770 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:05.904736 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-openshift-service-ca\") pod \"maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p\" (UID: \"98dfaf91-60ca-4bef-ae98-d5f3266e32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" Sep 14 13:13:05.906430 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:05.906407 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-metrics-certs\") pod \"maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p\" (UID: \"98dfaf91-60ca-4bef-ae98-d5f3266e32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" Sep 14 13:13:05.906536 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:05.906515 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-maas-api-tls\") pod \"maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p\" (UID: \"98dfaf91-60ca-4bef-ae98-d5f3266e32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" Sep 14 13:13:05.915149 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:05.915122 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6plsk\" (UniqueName: \"kubernetes.io/projected/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-kube-api-access-6plsk\") pod \"maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p\" (UID: \"98dfaf91-60ca-4bef-ae98-d5f3266e32ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" Sep 14 13:13:06.015674 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.015644 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" Sep 14 13:13:06.285629 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.285604 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:13:06.309361 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.309325 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-envoy\") pod \"7fedd108-9c7b-4c77-983f-5fd972206f7f\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " Sep 14 13:13:06.309546 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.309406 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fedd108-9c7b-4c77-983f-5fd972206f7f-istiod-ca-cert\") pod \"7fedd108-9c7b-4c77-983f-5fd972206f7f\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " Sep 14 13:13:06.309546 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.309459 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-podinfo\") pod \"7fedd108-9c7b-4c77-983f-5fd972206f7f\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " Sep 14 13:13:06.309546 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.309506 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-workload-certs\") pod \"7fedd108-9c7b-4c77-983f-5fd972206f7f\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " Sep 14 13:13:06.309546 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.309540 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-data\") pod \"7fedd108-9c7b-4c77-983f-5fd972206f7f\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " Sep 14 13:13:06.309780 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.309587 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bd7vn\" (UniqueName: \"kubernetes.io/projected/7fedd108-9c7b-4c77-983f-5fd972206f7f-kube-api-access-bd7vn\") pod \"7fedd108-9c7b-4c77-983f-5fd972206f7f\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " Sep 14 13:13:06.309780 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.309625 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-workload-socket\") pod \"7fedd108-9c7b-4c77-983f-5fd972206f7f\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " Sep 14 13:13:06.309780 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.309722 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-token\") pod \"7fedd108-9c7b-4c77-983f-5fd972206f7f\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " Sep 14 13:13:06.309780 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.309763 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-credential-socket\") pod \"7fedd108-9c7b-4c77-983f-5fd972206f7f\" (UID: \"7fedd108-9c7b-4c77-983f-5fd972206f7f\") " Sep 14 13:13:06.310434 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.310268 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7fedd108-9c7b-4c77-983f-5fd972206f7f" (UID: "7fedd108-9c7b-4c77-983f-5fd972206f7f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:13:06.311030 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.310668 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7fedd108-9c7b-4c77-983f-5fd972206f7f" (UID: "7fedd108-9c7b-4c77-983f-5fd972206f7f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:13:06.311030 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.310738 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-data" (OuterVolumeSpecName: "istio-data") pod "7fedd108-9c7b-4c77-983f-5fd972206f7f" (UID: "7fedd108-9c7b-4c77-983f-5fd972206f7f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:13:06.311030 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.310899 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7fedd108-9c7b-4c77-983f-5fd972206f7f" (UID: "7fedd108-9c7b-4c77-983f-5fd972206f7f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:13:06.311407 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.311160 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fedd108-9c7b-4c77-983f-5fd972206f7f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7fedd108-9c7b-4c77-983f-5fd972206f7f" (UID: "7fedd108-9c7b-4c77-983f-5fd972206f7f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:13:06.311470 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.311404 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:13:06.311470 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.311422 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:13:06.311470 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.311433 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fedd108-9c7b-4c77-983f-5fd972206f7f-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:13:06.311470 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.311441 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:13:06.311470 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.311450 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:13:06.316856 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.316817 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fedd108-9c7b-4c77-983f-5fd972206f7f-kube-api-access-bd7vn" (OuterVolumeSpecName: "kube-api-access-bd7vn") pod "7fedd108-9c7b-4c77-983f-5fd972206f7f" (UID: "7fedd108-9c7b-4c77-983f-5fd972206f7f"). InnerVolumeSpecName "kube-api-access-bd7vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:13:06.318881 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.318837 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-token" (OuterVolumeSpecName: "istio-token") pod "7fedd108-9c7b-4c77-983f-5fd972206f7f" (UID: "7fedd108-9c7b-4c77-983f-5fd972206f7f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:13:06.319162 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.319128 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7fedd108-9c7b-4c77-983f-5fd972206f7f" (UID: "7fedd108-9c7b-4c77-983f-5fd972206f7f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:13:06.320372 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.320341 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7fedd108-9c7b-4c77-983f-5fd972206f7f" (UID: "7fedd108-9c7b-4c77-983f-5fd972206f7f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 13:13:06.322600 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.322575 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p"] Sep 14 13:13:06.323288 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:13:06.323265 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98dfaf91_60ca_4bef_ae98_d5f3266e32ea.slice/crio-f710604e70794ebabc72b25cfe1ce5fede5b709f338028c2fd84ca66c6ebb841 WatchSource:0}: Error finding container f710604e70794ebabc72b25cfe1ce5fede5b709f338028c2fd84ca66c6ebb841: Status 404 returned error can't find the container with id f710604e70794ebabc72b25cfe1ce5fede5b709f338028c2fd84ca66c6ebb841 Sep 14 13:13:06.342389 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.342326 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:13:06.343213 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.343184 2564 patch_prober.go:28] interesting pod/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 14 13:13:06.343300 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.343238 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" podUID="0fd30d61-45b8-4586-8612-2b2caeb8a1ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 14 13:13:06.416024 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.413183 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:13:06.416024 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.413258 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:13:06.416024 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.413275 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fedd108-9c7b-4c77-983f-5fd972206f7f-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:13:06.416024 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.413291 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd7vn\" (UniqueName: \"kubernetes.io/projected/7fedd108-9c7b-4c77-983f-5fd972206f7f-kube-api-access-bd7vn\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:13:06.487139 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.487104 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" event={"ID":"98dfaf91-60ca-4bef-ae98-d5f3266e32ea","Type":"ContainerStarted","Data":"f710604e70794ebabc72b25cfe1ce5fede5b709f338028c2fd84ca66c6ebb841"} Sep 14 13:13:06.489176 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.489148 2564 generic.go:356] "Generic (PLEG): container finished" podID="7fedd108-9c7b-4c77-983f-5fd972206f7f" containerID="52e475a061cbbbe381bda615b0a8e1223fce1e94a492663fb90c909ff3dfcfae" exitCode=0 Sep 14 13:13:06.489628 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.489382 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" event={"ID":"7fedd108-9c7b-4c77-983f-5fd972206f7f","Type":"ContainerDied","Data":"52e475a061cbbbe381bda615b0a8e1223fce1e94a492663fb90c909ff3dfcfae"} Sep 14 13:13:06.489628 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.489456 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" event={"ID":"7fedd108-9c7b-4c77-983f-5fd972206f7f","Type":"ContainerDied","Data":"689cbd56b0d573a563c769cc6adff2c32fd244e350ba270c75ddb4ef5e1a8587"} Sep 14 13:13:06.489628 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.489488 2564 scope.go:117] "RemoveContainer" containerID="52e475a061cbbbe381bda615b0a8e1223fce1e94a492663fb90c909ff3dfcfae" Sep 14 13:13:06.489993 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.489975 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh" Sep 14 13:13:06.501823 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.501805 2564 scope.go:117] "RemoveContainer" containerID="003e9c79de060573701fd841913fcdd5898b3dc87050442253fd2ba69e09820a" Sep 14 13:13:06.511380 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.511362 2564 scope.go:117] "RemoveContainer" containerID="52e475a061cbbbe381bda615b0a8e1223fce1e94a492663fb90c909ff3dfcfae" Sep 14 13:13:06.511648 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:13:06.511629 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52e475a061cbbbe381bda615b0a8e1223fce1e94a492663fb90c909ff3dfcfae\": container with ID starting with 52e475a061cbbbe381bda615b0a8e1223fce1e94a492663fb90c909ff3dfcfae not found: ID does not exist" containerID="52e475a061cbbbe381bda615b0a8e1223fce1e94a492663fb90c909ff3dfcfae" Sep 14 13:13:06.511694 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.511657 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52e475a061cbbbe381bda615b0a8e1223fce1e94a492663fb90c909ff3dfcfae"} err="failed to get container status \"52e475a061cbbbe381bda615b0a8e1223fce1e94a492663fb90c909ff3dfcfae\": rpc error: code = NotFound desc = could not find container \"52e475a061cbbbe381bda615b0a8e1223fce1e94a492663fb90c909ff3dfcfae\": container with ID starting with 52e475a061cbbbe381bda615b0a8e1223fce1e94a492663fb90c909ff3dfcfae not found: ID does not exist" Sep 14 13:13:06.511694 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.511674 2564 scope.go:117] "RemoveContainer" containerID="003e9c79de060573701fd841913fcdd5898b3dc87050442253fd2ba69e09820a" Sep 14 13:13:06.511990 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:13:06.511967 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"003e9c79de060573701fd841913fcdd5898b3dc87050442253fd2ba69e09820a\": container with ID starting with 003e9c79de060573701fd841913fcdd5898b3dc87050442253fd2ba69e09820a not found: ID does not exist" containerID="003e9c79de060573701fd841913fcdd5898b3dc87050442253fd2ba69e09820a" Sep 14 13:13:06.512053 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.511998 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"003e9c79de060573701fd841913fcdd5898b3dc87050442253fd2ba69e09820a"} err="failed to get container status \"003e9c79de060573701fd841913fcdd5898b3dc87050442253fd2ba69e09820a\": rpc error: code = NotFound desc = could not find container \"003e9c79de060573701fd841913fcdd5898b3dc87050442253fd2ba69e09820a\": container with ID starting with 003e9c79de060573701fd841913fcdd5898b3dc87050442253fd2ba69e09820a not found: ID does not exist" Sep 14 13:13:06.518087 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.518059 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh"] Sep 14 13:13:06.521572 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:06.521551 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-360610b9-openshift-default-68cf9f49bb-8fnmh"] Sep 14 13:13:07.342592 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:07.342506 2564 patch_prober.go:28] interesting pod/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 14 13:13:07.342592 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:07.342563 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" podUID="0fd30d61-45b8-4586-8612-2b2caeb8a1ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 14 13:13:07.496374 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:07.496343 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" event={"ID":"98dfaf91-60ca-4bef-ae98-d5f3266e32ea","Type":"ContainerStarted","Data":"3284698b200a5172d211d877bbfb23233934d5fc983140c1fe72c93417d5f285"} Sep 14 13:13:07.496793 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:07.496423 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" Sep 14 13:13:07.513592 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:07.513544 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" podStartSLOduration=2.1878711969999998 podStartE2EDuration="2.513528941s" totalImagesPullingTime="325.657744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:13:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:13:07.512549226 +0000 UTC m=+888.175444857" watchObservedRunningTime="2026-09-14 13:13:07.513528941 +0000 UTC m=+888.176424571" Sep 14 13:13:07.811816 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:07.811784 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fedd108-9c7b-4c77-983f-5fd972206f7f" path="/var/lib/kubelet/pods/7fedd108-9c7b-4c77-983f-5fd972206f7f/volumes" Sep 14 13:13:08.242632 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:08.242593 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:13:08.342115 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:08.342083 2564 patch_prober.go:28] interesting pod/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 14 13:13:08.342282 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:08.342132 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" podUID="0fd30d61-45b8-4586-8612-2b2caeb8a1ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 14 13:13:09.341831 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:09.341794 2564 patch_prober.go:28] interesting pod/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 14 13:13:09.342304 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:09.341851 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" podUID="0fd30d61-45b8-4586-8612-2b2caeb8a1ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 14 13:13:10.342080 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:10.342038 2564 patch_prober.go:28] interesting pod/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Sep 14 13:13:10.342559 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:10.342115 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" podUID="0fd30d61-45b8-4586-8612-2b2caeb8a1ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Sep 14 13:13:11.346881 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:11.346798 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:13:11.348007 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:11.347987 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:13:13.509308 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:13.509275 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" Sep 14 13:13:19.841471 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:19.841446 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/1.log" Sep 14 13:13:19.842290 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:19.842272 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/1.log" Sep 14 13:13:36.301191 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.301159 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p"] Sep 14 13:13:36.301725 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.301401 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" podUID="98dfaf91-60ca-4bef-ae98-d5f3266e32ea" containerName="maas-api" containerID="cri-o://3284698b200a5172d211d877bbfb23233934d5fc983140c1fe72c93417d5f285" gracePeriod=30 Sep 14 13:13:36.558072 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.558009 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" Sep 14 13:13:36.612466 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.612430 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6plsk\" (UniqueName: \"kubernetes.io/projected/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-kube-api-access-6plsk\") pod \"98dfaf91-60ca-4bef-ae98-d5f3266e32ea\" (UID: \"98dfaf91-60ca-4bef-ae98-d5f3266e32ea\") " Sep 14 13:13:36.612651 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.612475 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-maas-api-tls\") pod \"98dfaf91-60ca-4bef-ae98-d5f3266e32ea\" (UID: \"98dfaf91-60ca-4bef-ae98-d5f3266e32ea\") " Sep 14 13:13:36.612651 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.612497 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-metrics-certs\") pod \"98dfaf91-60ca-4bef-ae98-d5f3266e32ea\" (UID: \"98dfaf91-60ca-4bef-ae98-d5f3266e32ea\") " Sep 14 13:13:36.614915 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.614880 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-kube-api-access-6plsk" (OuterVolumeSpecName: "kube-api-access-6plsk") pod "98dfaf91-60ca-4bef-ae98-d5f3266e32ea" (UID: "98dfaf91-60ca-4bef-ae98-d5f3266e32ea"). InnerVolumeSpecName "kube-api-access-6plsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:13:36.614915 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.614954 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "98dfaf91-60ca-4bef-ae98-d5f3266e32ea" (UID: "98dfaf91-60ca-4bef-ae98-d5f3266e32ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:13:36.615132 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.615028 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "98dfaf91-60ca-4bef-ae98-d5f3266e32ea" (UID: "98dfaf91-60ca-4bef-ae98-d5f3266e32ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:13:36.670652 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.670618 2564 generic.go:356] "Generic (PLEG): container finished" podID="98dfaf91-60ca-4bef-ae98-d5f3266e32ea" containerID="3284698b200a5172d211d877bbfb23233934d5fc983140c1fe72c93417d5f285" exitCode=0 Sep 14 13:13:36.670807 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.670691 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" Sep 14 13:13:36.670807 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.670709 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" event={"ID":"98dfaf91-60ca-4bef-ae98-d5f3266e32ea","Type":"ContainerDied","Data":"3284698b200a5172d211d877bbfb23233934d5fc983140c1fe72c93417d5f285"} Sep 14 13:13:36.670807 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.670773 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p" event={"ID":"98dfaf91-60ca-4bef-ae98-d5f3266e32ea","Type":"ContainerDied","Data":"f710604e70794ebabc72b25cfe1ce5fede5b709f338028c2fd84ca66c6ebb841"} Sep 14 13:13:36.671006 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.670806 2564 scope.go:117] "RemoveContainer" containerID="3284698b200a5172d211d877bbfb23233934d5fc983140c1fe72c93417d5f285" Sep 14 13:13:36.680582 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.680561 2564 scope.go:117] "RemoveContainer" containerID="3284698b200a5172d211d877bbfb23233934d5fc983140c1fe72c93417d5f285" Sep 14 13:13:36.680877 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:13:36.680859 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3284698b200a5172d211d877bbfb23233934d5fc983140c1fe72c93417d5f285\": container with ID starting with 3284698b200a5172d211d877bbfb23233934d5fc983140c1fe72c93417d5f285 not found: ID does not exist" containerID="3284698b200a5172d211d877bbfb23233934d5fc983140c1fe72c93417d5f285" Sep 14 13:13:36.680971 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.680886 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3284698b200a5172d211d877bbfb23233934d5fc983140c1fe72c93417d5f285"} err="failed to get container status \"3284698b200a5172d211d877bbfb23233934d5fc983140c1fe72c93417d5f285\": rpc error: code = NotFound desc = could not find container \"3284698b200a5172d211d877bbfb23233934d5fc983140c1fe72c93417d5f285\": container with ID starting with 3284698b200a5172d211d877bbfb23233934d5fc983140c1fe72c93417d5f285 not found: ID does not exist" Sep 14 13:13:36.713857 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.713835 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-openshift-service-ca\") pod \"98dfaf91-60ca-4bef-ae98-d5f3266e32ea\" (UID: \"98dfaf91-60ca-4bef-ae98-d5f3266e32ea\") " Sep 14 13:13:36.714186 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.714164 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "98dfaf91-60ca-4bef-ae98-d5f3266e32ea" (UID: "98dfaf91-60ca-4bef-ae98-d5f3266e32ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:13:36.714258 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.714240 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-maas-api-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:13:36.714327 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.714260 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-metrics-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:13:36.714327 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.714273 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6plsk\" (UniqueName: \"kubernetes.io/projected/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-kube-api-access-6plsk\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:13:36.814978 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.814868 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98dfaf91-60ca-4bef-ae98-d5f3266e32ea-openshift-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:13:36.994701 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.994671 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p"] Sep 14 13:13:36.999312 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:36.999285 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ea32f836-65c4bfd88-42f5p"] Sep 14 13:13:37.378604 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.378572 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6"] Sep 14 13:13:37.379046 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.379032 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fedd108-9c7b-4c77-983f-5fd972206f7f" containerName="istio-proxy" Sep 14 13:13:37.379106 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.379050 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fedd108-9c7b-4c77-983f-5fd972206f7f" containerName="istio-proxy" Sep 14 13:13:37.379106 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.379077 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98dfaf91-60ca-4bef-ae98-d5f3266e32ea" containerName="maas-api" Sep 14 13:13:37.379106 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.379082 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="98dfaf91-60ca-4bef-ae98-d5f3266e32ea" containerName="maas-api" Sep 14 13:13:37.379200 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.379146 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fedd108-9c7b-4c77-983f-5fd972206f7f" containerName="istio-proxy" Sep 14 13:13:37.379200 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.379154 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="98dfaf91-60ca-4bef-ae98-d5f3266e32ea" containerName="maas-api" Sep 14 13:13:37.382767 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.382749 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.385535 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.385516 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-dd15e28a-openshift-default-dockercfg-dwxgk\"" Sep 14 13:13:37.395517 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.395497 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6"] Sep 14 13:13:37.420312 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.420273 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istiod-ca-cert\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.420487 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.420349 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-workload-socket\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.420487 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.420372 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-data\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.420487 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.420396 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-podinfo\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.420487 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.420444 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-credential-socket\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.420487 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.420479 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-workload-certs\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.420736 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.420528 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-envoy\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.420736 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.420583 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-token\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.420736 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.420631 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmpj4\" (UniqueName: \"kubernetes.io/projected/3804e1ad-b945-4677-b3ab-0d1a4bc90397-kube-api-access-jmpj4\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.521691 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.521642 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istiod-ca-cert\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.521885 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.521742 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-workload-socket\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.521885 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.521769 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-data\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.521885 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.521793 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-podinfo\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.521885 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.521825 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-credential-socket\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.521885 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.521862 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-workload-certs\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.522175 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.521906 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-envoy\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.522175 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.521986 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-token\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.522332 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.522304 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-data\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.522475 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.522386 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmpj4\" (UniqueName: \"kubernetes.io/projected/3804e1ad-b945-4677-b3ab-0d1a4bc90397-kube-api-access-jmpj4\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.522475 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.522413 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-credential-socket\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.522602 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.522583 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istiod-ca-cert\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.522721 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.522682 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-workload-certs\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.522836 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.522763 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-workload-socket\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.524578 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.524554 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-envoy\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.524736 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.524723 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-podinfo\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.530334 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.530317 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-token\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.530510 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.530492 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmpj4\" (UniqueName: \"kubernetes.io/projected/3804e1ad-b945-4677-b3ab-0d1a4bc90397-kube-api-access-jmpj4\") pod \"e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.694551 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.694468 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:37.812224 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:37.812186 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98dfaf91-60ca-4bef-ae98-d5f3266e32ea" path="/var/lib/kubelet/pods/98dfaf91-60ca-4bef-ae98-d5f3266e32ea/volumes" Sep 14 13:13:38.001837 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:38.001760 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6"] Sep 14 13:13:38.013014 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:13:38.012986 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3804e1ad_b945_4677_b3ab_0d1a4bc90397.slice/crio-a63d5ee4cee873b53c2217789c11d2249ffcb0e7d48b9e429e77c2441f8942f8 WatchSource:0}: Error finding container a63d5ee4cee873b53c2217789c11d2249ffcb0e7d48b9e429e77c2441f8942f8: Status 404 returned error can't find the container with id a63d5ee4cee873b53c2217789c11d2249ffcb0e7d48b9e429e77c2441f8942f8 Sep 14 13:13:38.015390 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:38.015360 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:13:38.015473 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:38.015423 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:13:38.015473 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:38.015449 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:13:38.685168 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:38.685133 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" event={"ID":"3804e1ad-b945-4677-b3ab-0d1a4bc90397","Type":"ContainerStarted","Data":"2f39383cd5ea4acc5d233c64b32813b555d29f6c36787282a09f8366d0981678"} Sep 14 13:13:38.685571 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:38.685191 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" event={"ID":"3804e1ad-b945-4677-b3ab-0d1a4bc90397","Type":"ContainerStarted","Data":"a63d5ee4cee873b53c2217789c11d2249ffcb0e7d48b9e429e77c2441f8942f8"} Sep 14 13:13:38.695445 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:38.695419 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:38.707891 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:38.707838 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" podStartSLOduration=1.707822623 podStartE2EDuration="1.707822623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:13:38.704800463 +0000 UTC m=+919.367696085" watchObservedRunningTime="2026-09-14 13:13:38.707822623 +0000 UTC m=+919.370718252" Sep 14 13:13:39.700676 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:39.700646 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:40.697325 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:40.697290 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:13:40.698090 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:13:40.698071 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:14:01.828855 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:01.828823 2564 generic.go:356] "Generic (PLEG): container finished" podID="3804e1ad-b945-4677-b3ab-0d1a4bc90397" containerID="2f39383cd5ea4acc5d233c64b32813b555d29f6c36787282a09f8366d0981678" exitCode=0 Sep 14 13:14:01.828855 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:01.828859 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" event={"ID":"3804e1ad-b945-4677-b3ab-0d1a4bc90397","Type":"ContainerDied","Data":"2f39383cd5ea4acc5d233c64b32813b555d29f6c36787282a09f8366d0981678"} Sep 14 13:14:01.829310 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:01.829220 2564 scope.go:117] "RemoveContainer" containerID="2f39383cd5ea4acc5d233c64b32813b555d29f6c36787282a09f8366d0981678" Sep 14 13:14:01.830358 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:01.830335 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:14:01.830433 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:01.830405 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:14:01.830470 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:01.830436 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:14:02.838558 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:02.838523 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" event={"ID":"3804e1ad-b945-4677-b3ab-0d1a4bc90397","Type":"ContainerStarted","Data":"ca74f05cd1e35e07fd33bfd0f814d32bb6012a421f3a7279b14124bd19d5486b"} Sep 14 13:14:03.694662 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:03.694616 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:14:03.694903 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:03.694850 2564 patch_prober.go:28] interesting pod/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 14 13:14:03.695030 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:03.694906 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" podUID="3804e1ad-b945-4677-b3ab-0d1a4bc90397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 14 13:14:04.695191 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:04.695161 2564 patch_prober.go:28] interesting pod/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 14 13:14:04.695578 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:04.695213 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" podUID="3804e1ad-b945-4677-b3ab-0d1a4bc90397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 14 13:14:05.695699 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:05.695660 2564 patch_prober.go:28] interesting pod/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 14 13:14:05.696122 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:05.695730 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" podUID="3804e1ad-b945-4677-b3ab-0d1a4bc90397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 14 13:14:06.695803 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:06.695767 2564 patch_prober.go:28] interesting pod/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 14 13:14:06.696250 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:06.695824 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" podUID="3804e1ad-b945-4677-b3ab-0d1a4bc90397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 14 13:14:07.695916 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:07.695874 2564 patch_prober.go:28] interesting pod/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 14 13:14:07.696407 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:07.695970 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" podUID="3804e1ad-b945-4677-b3ab-0d1a4bc90397" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 14 13:14:08.699104 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:08.699073 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:14:08.699547 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:08.699433 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:14:08.700062 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:08.700045 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:14:13.791691 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:13.791655 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6"] Sep 14 13:14:13.792204 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:13.791959 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" podUID="3804e1ad-b945-4677-b3ab-0d1a4bc90397" containerName="istio-proxy" containerID="cri-o://ca74f05cd1e35e07fd33bfd0f814d32bb6012a421f3a7279b14124bd19d5486b" gracePeriod=30 Sep 14 13:14:18.223169 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.223045 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6"] Sep 14 13:14:18.223792 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.223625 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fedd108-9c7b-4c77-983f-5fd972206f7f" containerName="istio-proxy" Sep 14 13:14:18.223792 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.223644 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fedd108-9c7b-4c77-983f-5fd972206f7f" containerName="istio-proxy" Sep 14 13:14:18.223949 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.223816 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fedd108-9c7b-4c77-983f-5fd972206f7f" containerName="istio-proxy" Sep 14 13:14:18.227038 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.227011 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" Sep 14 13:14:18.231144 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.231122 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3448586b-serving-cert\"" Sep 14 13:14:18.241172 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.241146 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6"] Sep 14 13:14:18.253069 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.253044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnmm8\" (UniqueName: \"kubernetes.io/projected/2b3de65a-6575-443d-a806-6cd50ce7482a-kube-api-access-xnmm8\") pod \"maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6\" (UID: \"2b3de65a-6575-443d-a806-6cd50ce7482a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" Sep 14 13:14:18.253213 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.253105 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b3de65a-6575-443d-a806-6cd50ce7482a-maas-api-tls\") pod \"maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6\" (UID: \"2b3de65a-6575-443d-a806-6cd50ce7482a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" Sep 14 13:14:18.253213 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.253199 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b3de65a-6575-443d-a806-6cd50ce7482a-metrics-certs\") pod \"maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6\" (UID: \"2b3de65a-6575-443d-a806-6cd50ce7482a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" Sep 14 13:14:18.253361 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.253246 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b3de65a-6575-443d-a806-6cd50ce7482a-openshift-service-ca\") pod \"maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6\" (UID: \"2b3de65a-6575-443d-a806-6cd50ce7482a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" Sep 14 13:14:18.354137 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.354107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b3de65a-6575-443d-a806-6cd50ce7482a-maas-api-tls\") pod \"maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6\" (UID: \"2b3de65a-6575-443d-a806-6cd50ce7482a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" Sep 14 13:14:18.354313 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.354176 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b3de65a-6575-443d-a806-6cd50ce7482a-metrics-certs\") pod \"maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6\" (UID: \"2b3de65a-6575-443d-a806-6cd50ce7482a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" Sep 14 13:14:18.354313 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.354238 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b3de65a-6575-443d-a806-6cd50ce7482a-openshift-service-ca\") pod \"maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6\" (UID: \"2b3de65a-6575-443d-a806-6cd50ce7482a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" Sep 14 13:14:18.354313 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:14:18.354268 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-serving-cert: secret "maas-api-e2e-derive-3448586b-serving-cert" not found Sep 14 13:14:18.354423 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:14:18.354335 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b3de65a-6575-443d-a806-6cd50ce7482a-maas-api-tls podName:2b3de65a-6575-443d-a806-6cd50ce7482a nodeName:}" failed. No retries permitted until 2026-09-14 13:14:18.854315093 +0000 UTC m=+959.517210735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2b3de65a-6575-443d-a806-6cd50ce7482a-maas-api-tls") pod "maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" (UID: "2b3de65a-6575-443d-a806-6cd50ce7482a") : secret "maas-api-e2e-derive-3448586b-serving-cert" not found Sep 14 13:14:18.354423 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.354355 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnmm8\" (UniqueName: \"kubernetes.io/projected/2b3de65a-6575-443d-a806-6cd50ce7482a-kube-api-access-xnmm8\") pod \"maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6\" (UID: \"2b3de65a-6575-443d-a806-6cd50ce7482a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" Sep 14 13:14:18.354908 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.354882 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b3de65a-6575-443d-a806-6cd50ce7482a-openshift-service-ca\") pod \"maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6\" (UID: \"2b3de65a-6575-443d-a806-6cd50ce7482a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" Sep 14 13:14:18.356737 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.356719 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b3de65a-6575-443d-a806-6cd50ce7482a-metrics-certs\") pod \"maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6\" (UID: \"2b3de65a-6575-443d-a806-6cd50ce7482a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" Sep 14 13:14:18.383328 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.383304 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnmm8\" (UniqueName: \"kubernetes.io/projected/2b3de65a-6575-443d-a806-6cd50ce7482a-kube-api-access-xnmm8\") pod \"maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6\" (UID: \"2b3de65a-6575-443d-a806-6cd50ce7482a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" Sep 14 13:14:18.859692 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.859660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b3de65a-6575-443d-a806-6cd50ce7482a-maas-api-tls\") pod \"maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6\" (UID: \"2b3de65a-6575-443d-a806-6cd50ce7482a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" Sep 14 13:14:18.862114 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.862094 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b3de65a-6575-443d-a806-6cd50ce7482a-maas-api-tls\") pod \"maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6\" (UID: \"2b3de65a-6575-443d-a806-6cd50ce7482a\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" Sep 14 13:14:18.968840 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.968805 2564 generic.go:356] "Generic (PLEG): container finished" podID="3804e1ad-b945-4677-b3ab-0d1a4bc90397" containerID="ca74f05cd1e35e07fd33bfd0f814d32bb6012a421f3a7279b14124bd19d5486b" exitCode=0 Sep 14 13:14:18.969013 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.968902 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" event={"ID":"3804e1ad-b945-4677-b3ab-0d1a4bc90397","Type":"ContainerDied","Data":"ca74f05cd1e35e07fd33bfd0f814d32bb6012a421f3a7279b14124bd19d5486b"} Sep 14 13:14:18.969013 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:18.968986 2564 scope.go:117] "RemoveContainer" containerID="2f39383cd5ea4acc5d233c64b32813b555d29f6c36787282a09f8366d0981678" Sep 14 13:14:19.053578 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.053525 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:14:19.143353 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.143320 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" Sep 14 13:14:19.162670 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.162648 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istiod-ca-cert\") pod \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " Sep 14 13:14:19.162771 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.162691 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-envoy\") pod \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " Sep 14 13:14:19.162771 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.162754 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-token\") pod \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " Sep 14 13:14:19.162859 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.162794 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-credential-socket\") pod \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " Sep 14 13:14:19.162859 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.162823 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-podinfo\") pod \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " Sep 14 13:14:19.163179 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.162977 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-workload-socket\") pod \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " Sep 14 13:14:19.163179 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.163033 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3804e1ad-b945-4677-b3ab-0d1a4bc90397" (UID: "3804e1ad-b945-4677-b3ab-0d1a4bc90397"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:14:19.163179 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.163098 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-data\") pod \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " Sep 14 13:14:19.163179 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.163103 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3804e1ad-b945-4677-b3ab-0d1a4bc90397" (UID: "3804e1ad-b945-4677-b3ab-0d1a4bc90397"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:14:19.163179 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.163127 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmpj4\" (UniqueName: \"kubernetes.io/projected/3804e1ad-b945-4677-b3ab-0d1a4bc90397-kube-api-access-jmpj4\") pod \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " Sep 14 13:14:19.163179 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.163154 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-workload-certs\") pod \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\" (UID: \"3804e1ad-b945-4677-b3ab-0d1a4bc90397\") " Sep 14 13:14:19.163566 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.163211 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3804e1ad-b945-4677-b3ab-0d1a4bc90397" (UID: "3804e1ad-b945-4677-b3ab-0d1a4bc90397"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:14:19.163566 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.163438 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3804e1ad-b945-4677-b3ab-0d1a4bc90397" (UID: "3804e1ad-b945-4677-b3ab-0d1a4bc90397"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:14:19.163566 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.163540 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-data" (OuterVolumeSpecName: "istio-data") pod "3804e1ad-b945-4677-b3ab-0d1a4bc90397" (UID: "3804e1ad-b945-4677-b3ab-0d1a4bc90397"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:14:19.163713 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.163609 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:14:19.163713 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.163627 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:14:19.163713 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.163640 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:14:19.163713 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.163653 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:14:19.165450 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.165423 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3804e1ad-b945-4677-b3ab-0d1a4bc90397-kube-api-access-jmpj4" (OuterVolumeSpecName: "kube-api-access-jmpj4") pod "3804e1ad-b945-4677-b3ab-0d1a4bc90397" (UID: "3804e1ad-b945-4677-b3ab-0d1a4bc90397"). InnerVolumeSpecName "kube-api-access-jmpj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:14:19.165573 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.165453 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-token" (OuterVolumeSpecName: "istio-token") pod "3804e1ad-b945-4677-b3ab-0d1a4bc90397" (UID: "3804e1ad-b945-4677-b3ab-0d1a4bc90397"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:14:19.165573 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.165547 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3804e1ad-b945-4677-b3ab-0d1a4bc90397" (UID: "3804e1ad-b945-4677-b3ab-0d1a4bc90397"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:14:19.165573 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.165564 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3804e1ad-b945-4677-b3ab-0d1a4bc90397" (UID: "3804e1ad-b945-4677-b3ab-0d1a4bc90397"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 13:14:19.264852 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.264821 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:14:19.264852 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.264845 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:14:19.264852 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.264853 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:14:19.265309 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.264861 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmpj4\" (UniqueName: \"kubernetes.io/projected/3804e1ad-b945-4677-b3ab-0d1a4bc90397-kube-api-access-jmpj4\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:14:19.265309 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.264872 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3804e1ad-b945-4677-b3ab-0d1a4bc90397-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:14:19.417363 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.417335 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6"] Sep 14 13:14:19.418403 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:14:19.418375 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b3de65a_6575_443d_a806_6cd50ce7482a.slice/crio-9c118bb2aa566c738a151f699f41dcaaf9e919845585d76b7088afc4e7799643 WatchSource:0}: Error finding container 9c118bb2aa566c738a151f699f41dcaaf9e919845585d76b7088afc4e7799643: Status 404 returned error can't find the container with id 9c118bb2aa566c738a151f699f41dcaaf9e919845585d76b7088afc4e7799643 Sep 14 13:14:19.419748 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.419726 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 13:14:19.976131 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.976102 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" Sep 14 13:14:19.976314 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.976141 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6" event={"ID":"3804e1ad-b945-4677-b3ab-0d1a4bc90397","Type":"ContainerDied","Data":"a63d5ee4cee873b53c2217789c11d2249ffcb0e7d48b9e429e77c2441f8942f8"} Sep 14 13:14:19.976314 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.976202 2564 scope.go:117] "RemoveContainer" containerID="ca74f05cd1e35e07fd33bfd0f814d32bb6012a421f3a7279b14124bd19d5486b" Sep 14 13:14:19.978360 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.978336 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" event={"ID":"2b3de65a-6575-443d-a806-6cd50ce7482a","Type":"ContainerStarted","Data":"48d63329bf30cb94e94c27d0a09eabb327ddd0daa193eff4089abcb536c31d30"} Sep 14 13:14:19.978557 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.978542 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" Sep 14 13:14:19.978688 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:19.978674 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" event={"ID":"2b3de65a-6575-443d-a806-6cd50ce7482a","Type":"ContainerStarted","Data":"9c118bb2aa566c738a151f699f41dcaaf9e919845585d76b7088afc4e7799643"} Sep 14 13:14:20.001762 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:20.001713 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" podStartSLOduration=1.558633921 podStartE2EDuration="2.001698894s" totalImagesPullingTime="443.064973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:14:19.99804318 +0000 UTC m=+960.660938837" watchObservedRunningTime="2026-09-14 13:14:20.001698894 +0000 UTC m=+960.664594523" Sep 14 13:14:20.013640 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:20.013594 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6"] Sep 14 13:14:20.017221 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:20.017198 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-dd15e28a-openshift-default-59f89bb656-bqcl6"] Sep 14 13:14:21.812805 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:21.812767 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3804e1ad-b945-4677-b3ab-0d1a4bc90397" path="/var/lib/kubelet/pods/3804e1ad-b945-4677-b3ab-0d1a4bc90397/volumes" Sep 14 13:14:25.991439 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:25.991411 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" Sep 14 13:14:32.491681 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:32.491645 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf"] Sep 14 13:14:32.492134 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:32.491918 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" podUID="c69c2d2a-858b-4c74-a5c1-751db5724efa" containerName="maas-api" containerID="cri-o://4210994e1060391efe24290bfd44fb6cdf13fccfa42d8f4baf2b681afd07f681" gracePeriod=30 Sep 14 13:14:32.757570 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:32.757503 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" Sep 14 13:14:32.922526 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:32.922494 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-654st\" (UniqueName: \"kubernetes.io/projected/c69c2d2a-858b-4c74-a5c1-751db5724efa-kube-api-access-654st\") pod \"c69c2d2a-858b-4c74-a5c1-751db5724efa\" (UID: \"c69c2d2a-858b-4c74-a5c1-751db5724efa\") " Sep 14 13:14:32.922724 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:32.922552 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69c2d2a-858b-4c74-a5c1-751db5724efa-openshift-service-ca\") pod \"c69c2d2a-858b-4c74-a5c1-751db5724efa\" (UID: \"c69c2d2a-858b-4c74-a5c1-751db5724efa\") " Sep 14 13:14:32.922724 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:32.922596 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c69c2d2a-858b-4c74-a5c1-751db5724efa-metrics-certs\") pod \"c69c2d2a-858b-4c74-a5c1-751db5724efa\" (UID: \"c69c2d2a-858b-4c74-a5c1-751db5724efa\") " Sep 14 13:14:32.922724 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:32.922671 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c69c2d2a-858b-4c74-a5c1-751db5724efa-maas-api-tls\") pod \"c69c2d2a-858b-4c74-a5c1-751db5724efa\" (UID: \"c69c2d2a-858b-4c74-a5c1-751db5724efa\") " Sep 14 13:14:32.922915 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:32.922879 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c69c2d2a-858b-4c74-a5c1-751db5724efa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c69c2d2a-858b-4c74-a5c1-751db5724efa" (UID: "c69c2d2a-858b-4c74-a5c1-751db5724efa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:14:32.923080 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:32.923062 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c69c2d2a-858b-4c74-a5c1-751db5724efa-openshift-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:14:32.924773 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:32.924746 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c69c2d2a-858b-4c74-a5c1-751db5724efa-kube-api-access-654st" (OuterVolumeSpecName: "kube-api-access-654st") pod "c69c2d2a-858b-4c74-a5c1-751db5724efa" (UID: "c69c2d2a-858b-4c74-a5c1-751db5724efa"). InnerVolumeSpecName "kube-api-access-654st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:14:32.924999 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:32.924902 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c69c2d2a-858b-4c74-a5c1-751db5724efa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c69c2d2a-858b-4c74-a5c1-751db5724efa" (UID: "c69c2d2a-858b-4c74-a5c1-751db5724efa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:14:32.924999 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:32.924964 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c69c2d2a-858b-4c74-a5c1-751db5724efa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c69c2d2a-858b-4c74-a5c1-751db5724efa" (UID: "c69c2d2a-858b-4c74-a5c1-751db5724efa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:14:33.024316 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:33.024232 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c69c2d2a-858b-4c74-a5c1-751db5724efa-maas-api-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:14:33.024316 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:33.024258 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-654st\" (UniqueName: \"kubernetes.io/projected/c69c2d2a-858b-4c74-a5c1-751db5724efa-kube-api-access-654st\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:14:33.024316 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:33.024268 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c69c2d2a-858b-4c74-a5c1-751db5724efa-metrics-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:14:33.057920 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:33.057883 2564 generic.go:356] "Generic (PLEG): container finished" podID="c69c2d2a-858b-4c74-a5c1-751db5724efa" containerID="4210994e1060391efe24290bfd44fb6cdf13fccfa42d8f4baf2b681afd07f681" exitCode=0 Sep 14 13:14:33.058125 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:33.057952 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" Sep 14 13:14:33.058125 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:33.057970 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" event={"ID":"c69c2d2a-858b-4c74-a5c1-751db5724efa","Type":"ContainerDied","Data":"4210994e1060391efe24290bfd44fb6cdf13fccfa42d8f4baf2b681afd07f681"} Sep 14 13:14:33.058125 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:33.058025 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf" event={"ID":"c69c2d2a-858b-4c74-a5c1-751db5724efa","Type":"ContainerDied","Data":"6b87ac3417ba1ea09ed600e5aac1aa16aad5cce9d8ca18c52ba44a7c31edc1a1"} Sep 14 13:14:33.058125 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:33.058046 2564 scope.go:117] "RemoveContainer" containerID="4210994e1060391efe24290bfd44fb6cdf13fccfa42d8f4baf2b681afd07f681" Sep 14 13:14:33.067949 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:33.067919 2564 scope.go:117] "RemoveContainer" containerID="4210994e1060391efe24290bfd44fb6cdf13fccfa42d8f4baf2b681afd07f681" Sep 14 13:14:33.068212 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:14:33.068190 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4210994e1060391efe24290bfd44fb6cdf13fccfa42d8f4baf2b681afd07f681\": container with ID starting with 4210994e1060391efe24290bfd44fb6cdf13fccfa42d8f4baf2b681afd07f681 not found: ID does not exist" containerID="4210994e1060391efe24290bfd44fb6cdf13fccfa42d8f4baf2b681afd07f681" Sep 14 13:14:33.068278 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:33.068221 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4210994e1060391efe24290bfd44fb6cdf13fccfa42d8f4baf2b681afd07f681"} err="failed to get container status \"4210994e1060391efe24290bfd44fb6cdf13fccfa42d8f4baf2b681afd07f681\": rpc error: code = NotFound desc = could not find container \"4210994e1060391efe24290bfd44fb6cdf13fccfa42d8f4baf2b681afd07f681\": container with ID starting with 4210994e1060391efe24290bfd44fb6cdf13fccfa42d8f4baf2b681afd07f681 not found: ID does not exist" Sep 14 13:14:33.079818 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:33.079790 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf"] Sep 14 13:14:33.083330 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:33.083310 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-c14a5e-5769f8d65-kgbhf"] Sep 14 13:14:33.812654 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:33.812617 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c69c2d2a-858b-4c74-a5c1-751db5724efa" path="/var/lib/kubelet/pods/c69c2d2a-858b-4c74-a5c1-751db5724efa/volumes" Sep 14 13:14:47.241860 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:47.241788 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6"] Sep 14 13:14:47.242382 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:47.242021 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" podUID="2b3de65a-6575-443d-a806-6cd50ce7482a" containerName="maas-api" containerID="cri-o://48d63329bf30cb94e94c27d0a09eabb327ddd0daa193eff4089abcb536c31d30" gracePeriod=30 Sep 14 13:14:47.502860 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:47.502802 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" Sep 14 13:14:47.573748 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:47.573717 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b3de65a-6575-443d-a806-6cd50ce7482a-maas-api-tls\") pod \"2b3de65a-6575-443d-a806-6cd50ce7482a\" (UID: \"2b3de65a-6575-443d-a806-6cd50ce7482a\") " Sep 14 13:14:47.573908 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:47.573764 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnmm8\" (UniqueName: \"kubernetes.io/projected/2b3de65a-6575-443d-a806-6cd50ce7482a-kube-api-access-xnmm8\") pod \"2b3de65a-6575-443d-a806-6cd50ce7482a\" (UID: \"2b3de65a-6575-443d-a806-6cd50ce7482a\") " Sep 14 13:14:47.573908 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:47.573861 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b3de65a-6575-443d-a806-6cd50ce7482a-metrics-certs\") pod \"2b3de65a-6575-443d-a806-6cd50ce7482a\" (UID: \"2b3de65a-6575-443d-a806-6cd50ce7482a\") " Sep 14 13:14:47.574017 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:47.573908 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b3de65a-6575-443d-a806-6cd50ce7482a-openshift-service-ca\") pod \"2b3de65a-6575-443d-a806-6cd50ce7482a\" (UID: \"2b3de65a-6575-443d-a806-6cd50ce7482a\") " Sep 14 13:14:47.574268 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:47.574237 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b3de65a-6575-443d-a806-6cd50ce7482a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2b3de65a-6575-443d-a806-6cd50ce7482a" (UID: "2b3de65a-6575-443d-a806-6cd50ce7482a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:14:47.574397 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:47.574334 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b3de65a-6575-443d-a806-6cd50ce7482a-openshift-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:14:47.576119 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:47.576086 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b3de65a-6575-443d-a806-6cd50ce7482a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2b3de65a-6575-443d-a806-6cd50ce7482a" (UID: "2b3de65a-6575-443d-a806-6cd50ce7482a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:14:47.576221 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:47.576154 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b3de65a-6575-443d-a806-6cd50ce7482a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2b3de65a-6575-443d-a806-6cd50ce7482a" (UID: "2b3de65a-6575-443d-a806-6cd50ce7482a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:14:47.576221 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:47.576182 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b3de65a-6575-443d-a806-6cd50ce7482a-kube-api-access-xnmm8" (OuterVolumeSpecName: "kube-api-access-xnmm8") pod "2b3de65a-6575-443d-a806-6cd50ce7482a" (UID: "2b3de65a-6575-443d-a806-6cd50ce7482a"). InnerVolumeSpecName "kube-api-access-xnmm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:14:47.675856 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:47.675827 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b3de65a-6575-443d-a806-6cd50ce7482a-maas-api-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:14:47.675856 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:47.675855 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnmm8\" (UniqueName: \"kubernetes.io/projected/2b3de65a-6575-443d-a806-6cd50ce7482a-kube-api-access-xnmm8\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:14:47.675856 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:47.675865 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b3de65a-6575-443d-a806-6cd50ce7482a-metrics-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:14:48.157077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:48.157044 2564 generic.go:356] "Generic (PLEG): container finished" podID="2b3de65a-6575-443d-a806-6cd50ce7482a" containerID="48d63329bf30cb94e94c27d0a09eabb327ddd0daa193eff4089abcb536c31d30" exitCode=0 Sep 14 13:14:48.157251 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:48.157089 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" event={"ID":"2b3de65a-6575-443d-a806-6cd50ce7482a","Type":"ContainerDied","Data":"48d63329bf30cb94e94c27d0a09eabb327ddd0daa193eff4089abcb536c31d30"} Sep 14 13:14:48.157251 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:48.157131 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" event={"ID":"2b3de65a-6575-443d-a806-6cd50ce7482a","Type":"ContainerDied","Data":"9c118bb2aa566c738a151f699f41dcaaf9e919845585d76b7088afc4e7799643"} Sep 14 13:14:48.157251 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:48.157152 2564 scope.go:117] "RemoveContainer" containerID="48d63329bf30cb94e94c27d0a09eabb327ddd0daa193eff4089abcb536c31d30" Sep 14 13:14:48.157251 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:48.157090 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6" Sep 14 13:14:48.166814 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:48.166799 2564 scope.go:117] "RemoveContainer" containerID="48d63329bf30cb94e94c27d0a09eabb327ddd0daa193eff4089abcb536c31d30" Sep 14 13:14:48.167108 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:14:48.167084 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d63329bf30cb94e94c27d0a09eabb327ddd0daa193eff4089abcb536c31d30\": container with ID starting with 48d63329bf30cb94e94c27d0a09eabb327ddd0daa193eff4089abcb536c31d30 not found: ID does not exist" containerID="48d63329bf30cb94e94c27d0a09eabb327ddd0daa193eff4089abcb536c31d30" Sep 14 13:14:48.167184 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:48.167116 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d63329bf30cb94e94c27d0a09eabb327ddd0daa193eff4089abcb536c31d30"} err="failed to get container status \"48d63329bf30cb94e94c27d0a09eabb327ddd0daa193eff4089abcb536c31d30\": rpc error: code = NotFound desc = could not find container \"48d63329bf30cb94e94c27d0a09eabb327ddd0daa193eff4089abcb536c31d30\": container with ID starting with 48d63329bf30cb94e94c27d0a09eabb327ddd0daa193eff4089abcb536c31d30 not found: ID does not exist" Sep 14 13:14:48.176301 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:48.176271 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6"] Sep 14 13:14:48.180251 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:48.180226 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3448586b-68cf54c4d4-pwbl6"] Sep 14 13:14:49.812344 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:14:49.812312 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b3de65a-6575-443d-a806-6cd50ce7482a" path="/var/lib/kubelet/pods/2b3de65a-6575-443d-a806-6cd50ce7482a/volumes" Sep 14 13:15:16.811001 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.810969 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq"] Sep 14 13:15:16.811425 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.811411 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b3de65a-6575-443d-a806-6cd50ce7482a" containerName="maas-api" Sep 14 13:15:16.811477 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.811427 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b3de65a-6575-443d-a806-6cd50ce7482a" containerName="maas-api" Sep 14 13:15:16.811477 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.811443 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3804e1ad-b945-4677-b3ab-0d1a4bc90397" containerName="istio-proxy" Sep 14 13:15:16.811477 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.811448 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3804e1ad-b945-4677-b3ab-0d1a4bc90397" containerName="istio-proxy" Sep 14 13:15:16.811477 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.811460 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3804e1ad-b945-4677-b3ab-0d1a4bc90397" containerName="istio-proxy" Sep 14 13:15:16.811477 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.811466 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3804e1ad-b945-4677-b3ab-0d1a4bc90397" containerName="istio-proxy" Sep 14 13:15:16.811477 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.811474 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c69c2d2a-858b-4c74-a5c1-751db5724efa" containerName="maas-api" Sep 14 13:15:16.811477 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.811478 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c69c2d2a-858b-4c74-a5c1-751db5724efa" containerName="maas-api" Sep 14 13:15:16.811695 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.811539 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3804e1ad-b945-4677-b3ab-0d1a4bc90397" containerName="istio-proxy" Sep 14 13:15:16.811695 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.811548 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b3de65a-6575-443d-a806-6cd50ce7482a" containerName="maas-api" Sep 14 13:15:16.811695 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.811554 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c69c2d2a-858b-4c74-a5c1-751db5724efa" containerName="maas-api" Sep 14 13:15:16.811695 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.811561 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3804e1ad-b945-4677-b3ab-0d1a4bc90397" containerName="istio-proxy" Sep 14 13:15:16.813889 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.813873 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:16.816627 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.816611 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-ace1c6-openshift-default-dockercfg-7r5wh\"" Sep 14 13:15:16.831383 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.831363 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq"] Sep 14 13:15:16.968521 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.968485 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-workload-socket\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:16.968521 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.968529 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-workload-certs\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:16.968776 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.968573 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v26nb\" (UniqueName: \"kubernetes.io/projected/ba2a8136-484e-44db-9b90-e02984478f87-kube-api-access-v26nb\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:16.968776 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.968598 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba2a8136-484e-44db-9b90-e02984478f87-istiod-ca-cert\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:16.968776 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.968627 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-istio-data\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:16.968776 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.968674 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba2a8136-484e-44db-9b90-e02984478f87-istio-token\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:16.968776 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.968773 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-istio-envoy\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:16.969097 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.968867 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-credential-socket\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:16.969097 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:16.969000 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba2a8136-484e-44db-9b90-e02984478f87-istio-podinfo\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:17.070395 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.070314 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v26nb\" (UniqueName: \"kubernetes.io/projected/ba2a8136-484e-44db-9b90-e02984478f87-kube-api-access-v26nb\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:17.070395 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.070353 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba2a8136-484e-44db-9b90-e02984478f87-istiod-ca-cert\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:17.070395 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.070373 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-istio-data\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:17.070395 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.070387 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba2a8136-484e-44db-9b90-e02984478f87-istio-token\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:17.070740 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.070436 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-istio-envoy\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:17.070740 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.070489 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-credential-socket\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:17.070740 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.070541 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba2a8136-484e-44db-9b90-e02984478f87-istio-podinfo\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:17.070740 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.070614 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-workload-socket\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:17.070740 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.070659 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-workload-certs\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:17.071027 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.070821 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-istio-data\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:17.071027 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.070959 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-credential-socket\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:17.071105 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.071022 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-workload-socket\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:17.071105 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.071092 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-workload-certs\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:17.071307 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.071285 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba2a8136-484e-44db-9b90-e02984478f87-istiod-ca-cert\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:17.073086 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.073064 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-istio-envoy\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:17.073411 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.073387 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba2a8136-484e-44db-9b90-e02984478f87-istio-podinfo\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:17.080685 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.080656 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v26nb\" (UniqueName: \"kubernetes.io/projected/ba2a8136-484e-44db-9b90-e02984478f87-kube-api-access-v26nb\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:17.081483 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.081462 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba2a8136-484e-44db-9b90-e02984478f87-istio-token\") pod \"e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:17.123840 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.123815 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:17.410275 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.410247 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq"] Sep 14 13:15:17.411391 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:15:17.411357 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2a8136_484e_44db_9b90_e02984478f87.slice/crio-7123555bd70bbabeea18575e5ddde48aab6806a935801a5b7755af77392ff051 WatchSource:0}: Error finding container 7123555bd70bbabeea18575e5ddde48aab6806a935801a5b7755af77392ff051: Status 404 returned error can't find the container with id 7123555bd70bbabeea18575e5ddde48aab6806a935801a5b7755af77392ff051 Sep 14 13:15:17.413660 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.413625 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:15:17.413765 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.413703 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:15:17.413765 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:17.413746 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:15:18.349905 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:18.349867 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" event={"ID":"ba2a8136-484e-44db-9b90-e02984478f87","Type":"ContainerStarted","Data":"90ec4371213c183c13f301110cd291173d910cbc581a28f08678a90bc07128f3"} Sep 14 13:15:18.350459 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:18.349936 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" event={"ID":"ba2a8136-484e-44db-9b90-e02984478f87","Type":"ContainerStarted","Data":"7123555bd70bbabeea18575e5ddde48aab6806a935801a5b7755af77392ff051"} Sep 14 13:15:18.375959 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:18.375897 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" podStartSLOduration=2.37587925 podStartE2EDuration="2.37587925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:15:18.374304922 +0000 UTC m=+1019.037200567" watchObservedRunningTime="2026-09-14 13:15:18.37587925 +0000 UTC m=+1019.038774880" Sep 14 13:15:19.124800 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:19.124766 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:19.129685 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:19.129662 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:19.356037 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:19.356003 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:19.356884 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:19.356865 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:23.659447 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:23.659417 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf"] Sep 14 13:15:23.664543 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:23.664520 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" Sep 14 13:15:23.667090 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:23.667061 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-ace1c6-serving-cert\"" Sep 14 13:15:23.677233 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:23.677208 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf"] Sep 14 13:15:23.734539 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:23.734512 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brhdk\" (UniqueName: \"kubernetes.io/projected/f191b5f6-77e5-4829-b52b-9541e9d7a77a-kube-api-access-brhdk\") pod \"maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf\" (UID: \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" Sep 14 13:15:23.734735 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:23.734568 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f191b5f6-77e5-4829-b52b-9541e9d7a77a-metrics-certs\") pod \"maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf\" (UID: \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" Sep 14 13:15:23.734735 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:23.734631 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f191b5f6-77e5-4829-b52b-9541e9d7a77a-maas-api-tls\") pod \"maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf\" (UID: \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" Sep 14 13:15:23.734735 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:23.734685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f191b5f6-77e5-4829-b52b-9541e9d7a77a-openshift-service-ca\") pod \"maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf\" (UID: \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" Sep 14 13:15:23.835533 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:23.835499 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f191b5f6-77e5-4829-b52b-9541e9d7a77a-metrics-certs\") pod \"maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf\" (UID: \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" Sep 14 13:15:23.835533 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:23.835537 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f191b5f6-77e5-4829-b52b-9541e9d7a77a-maas-api-tls\") pod \"maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf\" (UID: \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" Sep 14 13:15:23.835774 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:23.835568 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f191b5f6-77e5-4829-b52b-9541e9d7a77a-openshift-service-ca\") pod \"maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf\" (UID: \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" Sep 14 13:15:23.835774 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:23.835680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brhdk\" (UniqueName: \"kubernetes.io/projected/f191b5f6-77e5-4829-b52b-9541e9d7a77a-kube-api-access-brhdk\") pod \"maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf\" (UID: \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" Sep 14 13:15:23.835774 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:15:23.835709 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-serving-cert: secret "maas-api-e2e-explicit-ace1c6-serving-cert" not found Sep 14 13:15:23.835993 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:15:23.835793 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f191b5f6-77e5-4829-b52b-9541e9d7a77a-maas-api-tls podName:f191b5f6-77e5-4829-b52b-9541e9d7a77a nodeName:}" failed. No retries permitted until 2026-09-14 13:15:24.335772505 +0000 UTC m=+1024.998668136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f191b5f6-77e5-4829-b52b-9541e9d7a77a-maas-api-tls") pod "maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" (UID: "f191b5f6-77e5-4829-b52b-9541e9d7a77a") : secret "maas-api-e2e-explicit-ace1c6-serving-cert" not found Sep 14 13:15:23.836317 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:23.836297 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f191b5f6-77e5-4829-b52b-9541e9d7a77a-openshift-service-ca\") pod \"maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf\" (UID: \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" Sep 14 13:15:23.838074 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:23.838043 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f191b5f6-77e5-4829-b52b-9541e9d7a77a-metrics-certs\") pod \"maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf\" (UID: \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" Sep 14 13:15:23.847624 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:23.847594 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brhdk\" (UniqueName: \"kubernetes.io/projected/f191b5f6-77e5-4829-b52b-9541e9d7a77a-kube-api-access-brhdk\") pod \"maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf\" (UID: \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" Sep 14 13:15:24.341140 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:24.341107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f191b5f6-77e5-4829-b52b-9541e9d7a77a-maas-api-tls\") pod \"maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf\" (UID: \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" Sep 14 13:15:24.343644 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:24.343616 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f191b5f6-77e5-4829-b52b-9541e9d7a77a-maas-api-tls\") pod \"maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf\" (UID: \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" Sep 14 13:15:24.576803 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:24.576769 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" Sep 14 13:15:24.851736 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:24.851715 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf"] Sep 14 13:15:24.852770 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:15:24.852736 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf191b5f6_77e5_4829_b52b_9541e9d7a77a.slice/crio-f899548b05a49361471cb3d1c65f2628ef5e8d2e030a02e996b1b8824dd367d1 WatchSource:0}: Error finding container f899548b05a49361471cb3d1c65f2628ef5e8d2e030a02e996b1b8824dd367d1: Status 404 returned error can't find the container with id f899548b05a49361471cb3d1c65f2628ef5e8d2e030a02e996b1b8824dd367d1 Sep 14 13:15:25.392740 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:25.392711 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" event={"ID":"f191b5f6-77e5-4829-b52b-9541e9d7a77a","Type":"ContainerStarted","Data":"f899548b05a49361471cb3d1c65f2628ef5e8d2e030a02e996b1b8824dd367d1"} Sep 14 13:15:26.400235 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:26.400190 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" event={"ID":"f191b5f6-77e5-4829-b52b-9541e9d7a77a","Type":"ContainerStarted","Data":"018d7b9a4898b3324c385c3b26a31a394ac8a93680b7e31441b7d92daa21e09d"} Sep 14 13:15:26.400628 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:26.400257 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" Sep 14 13:15:26.427312 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:26.427265 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" podStartSLOduration=2.942187442 podStartE2EDuration="3.427247993s" totalImagesPullingTime="485.060551ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:15:26.424450327 +0000 UTC m=+1027.087345957" watchObservedRunningTime="2026-09-14 13:15:26.427247993 +0000 UTC m=+1027.090143622" Sep 14 13:15:32.412653 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:32.412624 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" Sep 14 13:15:43.528730 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:43.528696 2564 generic.go:356] "Generic (PLEG): container finished" podID="ba2a8136-484e-44db-9b90-e02984478f87" containerID="90ec4371213c183c13f301110cd291173d910cbc581a28f08678a90bc07128f3" exitCode=0 Sep 14 13:15:43.529151 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:43.528769 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" event={"ID":"ba2a8136-484e-44db-9b90-e02984478f87","Type":"ContainerDied","Data":"90ec4371213c183c13f301110cd291173d910cbc581a28f08678a90bc07128f3"} Sep 14 13:15:43.529232 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:43.529217 2564 scope.go:117] "RemoveContainer" containerID="90ec4371213c183c13f301110cd291173d910cbc581a28f08678a90bc07128f3" Sep 14 13:15:43.530544 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:43.530519 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:15:43.530607 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:43.530585 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:15:43.530646 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:43.530617 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:15:44.537643 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:44.537609 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" event={"ID":"ba2a8136-484e-44db-9b90-e02984478f87","Type":"ContainerStarted","Data":"02f7101b4ae498fcd6f83c2cb657faf68286aaf3ddeb623822622c9e1247b9bf"} Sep 14 13:15:45.125035 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:45.124995 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:45.126012 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:45.125989 2564 patch_prober.go:28] interesting pod/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 14 13:15:45.126113 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:45.126033 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" podUID="ba2a8136-484e-44db-9b90-e02984478f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 14 13:15:46.124453 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:46.124417 2564 patch_prober.go:28] interesting pod/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 14 13:15:46.124847 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:46.124477 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" podUID="ba2a8136-484e-44db-9b90-e02984478f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 14 13:15:47.124459 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:47.124423 2564 patch_prober.go:28] interesting pod/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 14 13:15:47.124876 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:47.124488 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" podUID="ba2a8136-484e-44db-9b90-e02984478f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 14 13:15:48.124274 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:48.124234 2564 patch_prober.go:28] interesting pod/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 14 13:15:48.124506 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:48.124290 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" podUID="ba2a8136-484e-44db-9b90-e02984478f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 14 13:15:49.125161 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:49.125126 2564 patch_prober.go:28] interesting pod/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Sep 14 13:15:49.125553 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:49.125196 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" podUID="ba2a8136-484e-44db-9b90-e02984478f87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Sep 14 13:15:49.356732 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:49.356700 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:50.128917 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:50.128891 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:15:50.130220 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:15:50.130198 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:16:36.933502 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:36.933414 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb"] Sep 14 13:16:36.937450 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:36.937428 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" Sep 14 13:16:36.940335 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:36.940305 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-758ecaaa-serving-cert\"" Sep 14 13:16:36.948276 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:36.948254 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb"] Sep 14 13:16:36.996013 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:36.995980 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da85ae1-9da7-4aae-b5b4-697df35f96e4-openshift-service-ca\") pod \"maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb\" (UID: \"0da85ae1-9da7-4aae-b5b4-697df35f96e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" Sep 14 13:16:36.996240 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:36.996020 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7prtg\" (UniqueName: \"kubernetes.io/projected/0da85ae1-9da7-4aae-b5b4-697df35f96e4-kube-api-access-7prtg\") pod \"maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb\" (UID: \"0da85ae1-9da7-4aae-b5b4-697df35f96e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" Sep 14 13:16:36.996240 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:36.996195 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da85ae1-9da7-4aae-b5b4-697df35f96e4-metrics-certs\") pod \"maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb\" (UID: \"0da85ae1-9da7-4aae-b5b4-697df35f96e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" Sep 14 13:16:36.996331 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:36.996253 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da85ae1-9da7-4aae-b5b4-697df35f96e4-maas-api-tls\") pod \"maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb\" (UID: \"0da85ae1-9da7-4aae-b5b4-697df35f96e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" Sep 14 13:16:37.097217 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:37.097174 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da85ae1-9da7-4aae-b5b4-697df35f96e4-metrics-certs\") pod \"maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb\" (UID: \"0da85ae1-9da7-4aae-b5b4-697df35f96e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" Sep 14 13:16:37.097406 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:37.097248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da85ae1-9da7-4aae-b5b4-697df35f96e4-maas-api-tls\") pod \"maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb\" (UID: \"0da85ae1-9da7-4aae-b5b4-697df35f96e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" Sep 14 13:16:37.097406 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:37.097303 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da85ae1-9da7-4aae-b5b4-697df35f96e4-openshift-service-ca\") pod \"maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb\" (UID: \"0da85ae1-9da7-4aae-b5b4-697df35f96e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" Sep 14 13:16:37.097406 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:37.097349 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7prtg\" (UniqueName: \"kubernetes.io/projected/0da85ae1-9da7-4aae-b5b4-697df35f96e4-kube-api-access-7prtg\") pod \"maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb\" (UID: \"0da85ae1-9da7-4aae-b5b4-697df35f96e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" Sep 14 13:16:37.098153 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:37.098067 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da85ae1-9da7-4aae-b5b4-697df35f96e4-openshift-service-ca\") pod \"maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb\" (UID: \"0da85ae1-9da7-4aae-b5b4-697df35f96e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" Sep 14 13:16:37.099836 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:37.099816 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da85ae1-9da7-4aae-b5b4-697df35f96e4-metrics-certs\") pod \"maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb\" (UID: \"0da85ae1-9da7-4aae-b5b4-697df35f96e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" Sep 14 13:16:37.099947 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:37.099872 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da85ae1-9da7-4aae-b5b4-697df35f96e4-maas-api-tls\") pod \"maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb\" (UID: \"0da85ae1-9da7-4aae-b5b4-697df35f96e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" Sep 14 13:16:37.105559 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:37.105531 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7prtg\" (UniqueName: \"kubernetes.io/projected/0da85ae1-9da7-4aae-b5b4-697df35f96e4-kube-api-access-7prtg\") pod \"maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb\" (UID: \"0da85ae1-9da7-4aae-b5b4-697df35f96e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" Sep 14 13:16:37.250366 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:37.250271 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" Sep 14 13:16:37.529002 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:37.528973 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb"] Sep 14 13:16:37.530103 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:16:37.530075 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da85ae1_9da7_4aae_b5b4_697df35f96e4.slice/crio-51752b5c48d3048d08238a6e1eb983a40e647b000d01fd3f2394dac9942304f9 WatchSource:0}: Error finding container 51752b5c48d3048d08238a6e1eb983a40e647b000d01fd3f2394dac9942304f9: Status 404 returned error can't find the container with id 51752b5c48d3048d08238a6e1eb983a40e647b000d01fd3f2394dac9942304f9 Sep 14 13:16:37.861906 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:37.861816 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" event={"ID":"0da85ae1-9da7-4aae-b5b4-697df35f96e4","Type":"ContainerStarted","Data":"51752b5c48d3048d08238a6e1eb983a40e647b000d01fd3f2394dac9942304f9"} Sep 14 13:16:38.869632 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:38.869587 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" event={"ID":"0da85ae1-9da7-4aae-b5b4-697df35f96e4","Type":"ContainerStarted","Data":"8ec6a81a03536342ba6418d56c6a34692e553d30e42417c4b7e636a40a7f0330"} Sep 14 13:16:38.870094 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:38.869650 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" Sep 14 13:16:38.889698 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:38.889646 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" podStartSLOduration=2.487942129 podStartE2EDuration="2.889629598s" totalImagesPullingTime="401.687469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:16:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:16:38.887389129 +0000 UTC m=+1099.550284759" watchObservedRunningTime="2026-09-14 13:16:38.889629598 +0000 UTC m=+1099.552525228" Sep 14 13:16:44.881170 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:16:44.881138 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" Sep 14 13:17:21.141461 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:21.141427 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb"] Sep 14 13:17:21.142111 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:21.141671 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" podUID="0da85ae1-9da7-4aae-b5b4-697df35f96e4" containerName="maas-api" containerID="cri-o://8ec6a81a03536342ba6418d56c6a34692e553d30e42417c4b7e636a40a7f0330" gracePeriod=30 Sep 14 13:17:21.396613 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:21.396552 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" Sep 14 13:17:21.463761 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:21.463735 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da85ae1-9da7-4aae-b5b4-697df35f96e4-metrics-certs\") pod \"0da85ae1-9da7-4aae-b5b4-697df35f96e4\" (UID: \"0da85ae1-9da7-4aae-b5b4-697df35f96e4\") " Sep 14 13:17:21.463969 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:21.463798 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7prtg\" (UniqueName: \"kubernetes.io/projected/0da85ae1-9da7-4aae-b5b4-697df35f96e4-kube-api-access-7prtg\") pod \"0da85ae1-9da7-4aae-b5b4-697df35f96e4\" (UID: \"0da85ae1-9da7-4aae-b5b4-697df35f96e4\") " Sep 14 13:17:21.463969 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:21.463838 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da85ae1-9da7-4aae-b5b4-697df35f96e4-maas-api-tls\") pod \"0da85ae1-9da7-4aae-b5b4-697df35f96e4\" (UID: \"0da85ae1-9da7-4aae-b5b4-697df35f96e4\") " Sep 14 13:17:21.463969 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:21.463869 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da85ae1-9da7-4aae-b5b4-697df35f96e4-openshift-service-ca\") pod \"0da85ae1-9da7-4aae-b5b4-697df35f96e4\" (UID: \"0da85ae1-9da7-4aae-b5b4-697df35f96e4\") " Sep 14 13:17:21.464311 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:21.464285 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da85ae1-9da7-4aae-b5b4-697df35f96e4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0da85ae1-9da7-4aae-b5b4-697df35f96e4" (UID: "0da85ae1-9da7-4aae-b5b4-697df35f96e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:17:21.466017 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:21.465987 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da85ae1-9da7-4aae-b5b4-697df35f96e4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0da85ae1-9da7-4aae-b5b4-697df35f96e4" (UID: "0da85ae1-9da7-4aae-b5b4-697df35f96e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:17:21.466017 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:21.466004 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da85ae1-9da7-4aae-b5b4-697df35f96e4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0da85ae1-9da7-4aae-b5b4-697df35f96e4" (UID: "0da85ae1-9da7-4aae-b5b4-697df35f96e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:17:21.466155 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:21.466134 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da85ae1-9da7-4aae-b5b4-697df35f96e4-kube-api-access-7prtg" (OuterVolumeSpecName: "kube-api-access-7prtg") pod "0da85ae1-9da7-4aae-b5b4-697df35f96e4" (UID: "0da85ae1-9da7-4aae-b5b4-697df35f96e4"). InnerVolumeSpecName "kube-api-access-7prtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:17:21.565452 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:21.565317 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0da85ae1-9da7-4aae-b5b4-697df35f96e4-metrics-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:17:21.565452 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:21.565357 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7prtg\" (UniqueName: \"kubernetes.io/projected/0da85ae1-9da7-4aae-b5b4-697df35f96e4-kube-api-access-7prtg\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:17:21.565452 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:21.565372 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0da85ae1-9da7-4aae-b5b4-697df35f96e4-maas-api-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:17:21.565452 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:21.565387 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0da85ae1-9da7-4aae-b5b4-697df35f96e4-openshift-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:17:22.139206 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:22.139173 2564 generic.go:356] "Generic (PLEG): container finished" podID="0da85ae1-9da7-4aae-b5b4-697df35f96e4" containerID="8ec6a81a03536342ba6418d56c6a34692e553d30e42417c4b7e636a40a7f0330" exitCode=0 Sep 14 13:17:22.139206 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:22.139209 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" event={"ID":"0da85ae1-9da7-4aae-b5b4-697df35f96e4","Type":"ContainerDied","Data":"8ec6a81a03536342ba6418d56c6a34692e553d30e42417c4b7e636a40a7f0330"} Sep 14 13:17:22.139430 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:22.139217 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" Sep 14 13:17:22.139430 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:22.139249 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb" event={"ID":"0da85ae1-9da7-4aae-b5b4-697df35f96e4","Type":"ContainerDied","Data":"51752b5c48d3048d08238a6e1eb983a40e647b000d01fd3f2394dac9942304f9"} Sep 14 13:17:22.139430 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:22.139270 2564 scope.go:117] "RemoveContainer" containerID="8ec6a81a03536342ba6418d56c6a34692e553d30e42417c4b7e636a40a7f0330" Sep 14 13:17:22.148884 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:22.148690 2564 scope.go:117] "RemoveContainer" containerID="8ec6a81a03536342ba6418d56c6a34692e553d30e42417c4b7e636a40a7f0330" Sep 14 13:17:22.149151 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:17:22.148960 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec6a81a03536342ba6418d56c6a34692e553d30e42417c4b7e636a40a7f0330\": container with ID starting with 8ec6a81a03536342ba6418d56c6a34692e553d30e42417c4b7e636a40a7f0330 not found: ID does not exist" containerID="8ec6a81a03536342ba6418d56c6a34692e553d30e42417c4b7e636a40a7f0330" Sep 14 13:17:22.149151 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:22.148993 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec6a81a03536342ba6418d56c6a34692e553d30e42417c4b7e636a40a7f0330"} err="failed to get container status \"8ec6a81a03536342ba6418d56c6a34692e553d30e42417c4b7e636a40a7f0330\": rpc error: code = NotFound desc = could not find container \"8ec6a81a03536342ba6418d56c6a34692e553d30e42417c4b7e636a40a7f0330\": container with ID starting with 8ec6a81a03536342ba6418d56c6a34692e553d30e42417c4b7e636a40a7f0330 not found: ID does not exist" Sep 14 13:17:22.159054 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:22.159031 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb"] Sep 14 13:17:22.164512 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:22.164493 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-758ecaaa-6954dd9ff9-nl9fb"] Sep 14 13:17:23.812508 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:23.812463 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0da85ae1-9da7-4aae-b5b4-697df35f96e4" path="/var/lib/kubelet/pods/0da85ae1-9da7-4aae-b5b4-697df35f96e4/volumes" Sep 14 13:17:42.998455 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:42.998368 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw"] Sep 14 13:17:42.998948 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:42.998906 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da85ae1-9da7-4aae-b5b4-697df35f96e4" containerName="maas-api" Sep 14 13:17:42.998948 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:42.998945 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da85ae1-9da7-4aae-b5b4-697df35f96e4" containerName="maas-api" Sep 14 13:17:42.999091 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:42.999077 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da85ae1-9da7-4aae-b5b4-697df35f96e4" containerName="maas-api" Sep 14 13:17:43.002866 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.002841 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.005734 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.005713 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-31063a-openshift-default-dockercfg-6jfnm\"" Sep 14 13:17:43.016274 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.016242 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw"] Sep 14 13:17:43.082030 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.082002 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-credential-socket\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.082030 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.082034 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-workload-socket\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.082252 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.082057 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-istio-envoy\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.082252 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.082074 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhrlr\" (UniqueName: \"kubernetes.io/projected/be11482a-7155-4cec-b70f-0456b0566843-kube-api-access-zhrlr\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.082252 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.082162 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-istio-data\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.082252 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.082185 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be11482a-7155-4cec-b70f-0456b0566843-istiod-ca-cert\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.082252 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.082226 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-workload-certs\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.082451 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.082272 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be11482a-7155-4cec-b70f-0456b0566843-istio-token\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.082451 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.082318 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be11482a-7155-4cec-b70f-0456b0566843-istio-podinfo\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.183275 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.183237 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-workload-certs\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.183453 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.183286 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be11482a-7155-4cec-b70f-0456b0566843-istio-token\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.183453 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.183325 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be11482a-7155-4cec-b70f-0456b0566843-istio-podinfo\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.183581 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.183488 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-credential-socket\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.183581 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.183534 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-workload-socket\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.183581 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.183573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-istio-envoy\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.183761 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.183600 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhrlr\" (UniqueName: \"kubernetes.io/projected/be11482a-7155-4cec-b70f-0456b0566843-kube-api-access-zhrlr\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.183761 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.183646 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-workload-certs\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.183761 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.183675 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-istio-data\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.183761 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.183705 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be11482a-7155-4cec-b70f-0456b0566843-istiod-ca-cert\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.184016 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.183782 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-credential-socket\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.184016 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.183879 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-workload-socket\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.184129 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.184083 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-istio-data\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.184423 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.184400 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be11482a-7155-4cec-b70f-0456b0566843-istiod-ca-cert\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.185902 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.185882 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be11482a-7155-4cec-b70f-0456b0566843-istio-podinfo\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.186046 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.186025 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-istio-envoy\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.191699 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.191676 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhrlr\" (UniqueName: \"kubernetes.io/projected/be11482a-7155-4cec-b70f-0456b0566843-kube-api-access-zhrlr\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.191791 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.191758 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be11482a-7155-4cec-b70f-0456b0566843-istio-token\") pod \"e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.314202 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.314109 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:43.600906 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.600875 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw"] Sep 14 13:17:43.601846 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:17:43.601820 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe11482a_7155_4cec_b70f_0456b0566843.slice/crio-89d9558bc2ee1b7001758af9c199d6fcb760ca22a947df8cae9ec07fb17aff11 WatchSource:0}: Error finding container 89d9558bc2ee1b7001758af9c199d6fcb760ca22a947df8cae9ec07fb17aff11: Status 404 returned error can't find the container with id 89d9558bc2ee1b7001758af9c199d6fcb760ca22a947df8cae9ec07fb17aff11 Sep 14 13:17:43.604219 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.604187 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:17:43.604326 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.604264 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:17:43.604326 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:43.604311 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:17:44.282256 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:44.282218 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" event={"ID":"be11482a-7155-4cec-b70f-0456b0566843","Type":"ContainerStarted","Data":"c52727df75243ddcf885d166e8335d9c8f5408d6f88cfb77303788f7b2594074"} Sep 14 13:17:44.282727 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:44.282277 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" event={"ID":"be11482a-7155-4cec-b70f-0456b0566843","Type":"ContainerStarted","Data":"89d9558bc2ee1b7001758af9c199d6fcb760ca22a947df8cae9ec07fb17aff11"} Sep 14 13:17:44.302800 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:44.302756 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" podStartSLOduration=2.30274117 podStartE2EDuration="2.30274117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:17:44.301465789 +0000 UTC m=+1164.964361422" watchObservedRunningTime="2026-09-14 13:17:44.30274117 +0000 UTC m=+1164.965636801" Sep 14 13:17:44.314523 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:44.314490 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:44.316269 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:44.316246 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 14 13:17:44.316377 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:44.316297 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" podUID="be11482a-7155-4cec-b70f-0456b0566843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 14 13:17:45.318782 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:45.318755 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:46.294829 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:46.294796 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:46.295707 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:46.295686 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:17:48.588260 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:48.588222 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf"] Sep 14 13:17:48.588810 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:48.588446 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" podUID="f191b5f6-77e5-4829-b52b-9541e9d7a77a" containerName="maas-api" containerID="cri-o://018d7b9a4898b3324c385c3b26a31a394ac8a93680b7e31441b7d92daa21e09d" gracePeriod=30 Sep 14 13:17:48.841368 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:48.841308 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" Sep 14 13:17:48.947598 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:48.947564 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f191b5f6-77e5-4829-b52b-9541e9d7a77a-maas-api-tls\") pod \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\" (UID: \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\") " Sep 14 13:17:48.947766 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:48.947688 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brhdk\" (UniqueName: \"kubernetes.io/projected/f191b5f6-77e5-4829-b52b-9541e9d7a77a-kube-api-access-brhdk\") pod \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\" (UID: \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\") " Sep 14 13:17:48.947829 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:48.947786 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f191b5f6-77e5-4829-b52b-9541e9d7a77a-openshift-service-ca\") pod \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\" (UID: \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\") " Sep 14 13:17:48.947829 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:48.947818 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f191b5f6-77e5-4829-b52b-9541e9d7a77a-metrics-certs\") pod \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\" (UID: \"f191b5f6-77e5-4829-b52b-9541e9d7a77a\") " Sep 14 13:17:48.948232 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:48.948197 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f191b5f6-77e5-4829-b52b-9541e9d7a77a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f191b5f6-77e5-4829-b52b-9541e9d7a77a" (UID: "f191b5f6-77e5-4829-b52b-9541e9d7a77a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:17:48.950015 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:48.949982 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f191b5f6-77e5-4829-b52b-9541e9d7a77a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f191b5f6-77e5-4829-b52b-9541e9d7a77a" (UID: "f191b5f6-77e5-4829-b52b-9541e9d7a77a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:17:48.950015 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:48.949998 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f191b5f6-77e5-4829-b52b-9541e9d7a77a-kube-api-access-brhdk" (OuterVolumeSpecName: "kube-api-access-brhdk") pod "f191b5f6-77e5-4829-b52b-9541e9d7a77a" (UID: "f191b5f6-77e5-4829-b52b-9541e9d7a77a"). InnerVolumeSpecName "kube-api-access-brhdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:17:48.950470 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:48.950444 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f191b5f6-77e5-4829-b52b-9541e9d7a77a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f191b5f6-77e5-4829-b52b-9541e9d7a77a" (UID: "f191b5f6-77e5-4829-b52b-9541e9d7a77a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:17:49.049202 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.049167 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f191b5f6-77e5-4829-b52b-9541e9d7a77a-maas-api-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:17:49.049202 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.049196 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brhdk\" (UniqueName: \"kubernetes.io/projected/f191b5f6-77e5-4829-b52b-9541e9d7a77a-kube-api-access-brhdk\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:17:49.049202 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.049205 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f191b5f6-77e5-4829-b52b-9541e9d7a77a-openshift-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:17:49.049455 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.049213 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f191b5f6-77e5-4829-b52b-9541e9d7a77a-metrics-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:17:49.317226 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.317194 2564 generic.go:356] "Generic (PLEG): container finished" podID="f191b5f6-77e5-4829-b52b-9541e9d7a77a" containerID="018d7b9a4898b3324c385c3b26a31a394ac8a93680b7e31441b7d92daa21e09d" exitCode=0 Sep 14 13:17:49.317419 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.317236 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" Sep 14 13:17:49.317419 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.317284 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" event={"ID":"f191b5f6-77e5-4829-b52b-9541e9d7a77a","Type":"ContainerDied","Data":"018d7b9a4898b3324c385c3b26a31a394ac8a93680b7e31441b7d92daa21e09d"} Sep 14 13:17:49.317419 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.317333 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf" event={"ID":"f191b5f6-77e5-4829-b52b-9541e9d7a77a","Type":"ContainerDied","Data":"f899548b05a49361471cb3d1c65f2628ef5e8d2e030a02e996b1b8824dd367d1"} Sep 14 13:17:49.317419 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.317355 2564 scope.go:117] "RemoveContainer" containerID="018d7b9a4898b3324c385c3b26a31a394ac8a93680b7e31441b7d92daa21e09d" Sep 14 13:17:49.327467 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.327446 2564 scope.go:117] "RemoveContainer" containerID="018d7b9a4898b3324c385c3b26a31a394ac8a93680b7e31441b7d92daa21e09d" Sep 14 13:17:49.327756 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:17:49.327738 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"018d7b9a4898b3324c385c3b26a31a394ac8a93680b7e31441b7d92daa21e09d\": container with ID starting with 018d7b9a4898b3324c385c3b26a31a394ac8a93680b7e31441b7d92daa21e09d not found: ID does not exist" containerID="018d7b9a4898b3324c385c3b26a31a394ac8a93680b7e31441b7d92daa21e09d" Sep 14 13:17:49.327815 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.327767 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"018d7b9a4898b3324c385c3b26a31a394ac8a93680b7e31441b7d92daa21e09d"} err="failed to get container status \"018d7b9a4898b3324c385c3b26a31a394ac8a93680b7e31441b7d92daa21e09d\": rpc error: code = NotFound desc = could not find container \"018d7b9a4898b3324c385c3b26a31a394ac8a93680b7e31441b7d92daa21e09d\": container with ID starting with 018d7b9a4898b3324c385c3b26a31a394ac8a93680b7e31441b7d92daa21e09d not found: ID does not exist" Sep 14 13:17:49.345675 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.345650 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf"] Sep 14 13:17:49.351293 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.351268 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ace1c6-66b5d85584-6wjpf"] Sep 14 13:17:49.537587 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.537554 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq"] Sep 14 13:17:49.538017 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.537993 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f191b5f6-77e5-4829-b52b-9541e9d7a77a" containerName="maas-api" Sep 14 13:17:49.538017 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.538018 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f191b5f6-77e5-4829-b52b-9541e9d7a77a" containerName="maas-api" Sep 14 13:17:49.538131 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.538101 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f191b5f6-77e5-4829-b52b-9541e9d7a77a" containerName="maas-api" Sep 14 13:17:49.542859 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.542839 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" Sep 14 13:17:49.545606 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.545587 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-31063a-serving-cert\"" Sep 14 13:17:49.553238 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.553215 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq"] Sep 14 13:17:49.656239 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.656196 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a94e70e-b37d-4e4b-8b46-a746ea37849f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq\" (UID: \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" Sep 14 13:17:49.656976 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.656949 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a94e70e-b37d-4e4b-8b46-a746ea37849f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq\" (UID: \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" Sep 14 13:17:49.657258 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.657229 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a94e70e-b37d-4e4b-8b46-a746ea37849f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq\" (UID: \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" Sep 14 13:17:49.657442 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.657425 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q99t\" (UniqueName: \"kubernetes.io/projected/4a94e70e-b37d-4e4b-8b46-a746ea37849f-kube-api-access-6q99t\") pod \"maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq\" (UID: \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" Sep 14 13:17:49.758661 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.758628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q99t\" (UniqueName: \"kubernetes.io/projected/4a94e70e-b37d-4e4b-8b46-a746ea37849f-kube-api-access-6q99t\") pod \"maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq\" (UID: \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" Sep 14 13:17:49.758853 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.758667 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a94e70e-b37d-4e4b-8b46-a746ea37849f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq\" (UID: \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" Sep 14 13:17:49.758853 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.758716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a94e70e-b37d-4e4b-8b46-a746ea37849f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq\" (UID: \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" Sep 14 13:17:49.758853 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.758765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a94e70e-b37d-4e4b-8b46-a746ea37849f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq\" (UID: \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" Sep 14 13:17:49.758853 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:17:49.758845 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-serving-cert: secret "maas-api-e2e-shared-a-w2-31063a-serving-cert" not found Sep 14 13:17:49.759099 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:17:49.758903 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a94e70e-b37d-4e4b-8b46-a746ea37849f-maas-api-tls podName:4a94e70e-b37d-4e4b-8b46-a746ea37849f nodeName:}" failed. No retries permitted until 2026-09-14 13:17:50.258888063 +0000 UTC m=+1170.921783675 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a94e70e-b37d-4e4b-8b46-a746ea37849f-maas-api-tls") pod "maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" (UID: "4a94e70e-b37d-4e4b-8b46-a746ea37849f") : secret "maas-api-e2e-shared-a-w2-31063a-serving-cert" not found Sep 14 13:17:49.759382 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.759365 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a94e70e-b37d-4e4b-8b46-a746ea37849f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq\" (UID: \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" Sep 14 13:17:49.761479 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.761451 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a94e70e-b37d-4e4b-8b46-a746ea37849f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq\" (UID: \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" Sep 14 13:17:49.767660 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.767640 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q99t\" (UniqueName: \"kubernetes.io/projected/4a94e70e-b37d-4e4b-8b46-a746ea37849f-kube-api-access-6q99t\") pod \"maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq\" (UID: \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" Sep 14 13:17:49.812431 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:49.812399 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f191b5f6-77e5-4829-b52b-9541e9d7a77a" path="/var/lib/kubelet/pods/f191b5f6-77e5-4829-b52b-9541e9d7a77a/volumes" Sep 14 13:17:50.263615 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:50.263579 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a94e70e-b37d-4e4b-8b46-a746ea37849f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq\" (UID: \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" Sep 14 13:17:50.266168 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:50.266142 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a94e70e-b37d-4e4b-8b46-a746ea37849f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq\" (UID: \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" Sep 14 13:17:50.455708 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:50.455679 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" Sep 14 13:17:50.731846 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:50.731818 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq"] Sep 14 13:17:50.733443 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:17:50.733416 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a94e70e_b37d_4e4b_8b46_a746ea37849f.slice/crio-bf33d2ddf93a5270d92f9c3ce95f5d12229ddf2cae545097a910bc80eeb19f5e WatchSource:0}: Error finding container bf33d2ddf93a5270d92f9c3ce95f5d12229ddf2cae545097a910bc80eeb19f5e: Status 404 returned error can't find the container with id bf33d2ddf93a5270d92f9c3ce95f5d12229ddf2cae545097a910bc80eeb19f5e Sep 14 13:17:51.335094 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:51.335068 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" event={"ID":"4a94e70e-b37d-4e4b-8b46-a746ea37849f","Type":"ContainerStarted","Data":"e65e16907910cf1164e6cd30b036777dfaa53f50a76b55018eca698c561fad86"} Sep 14 13:17:51.335220 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:51.335114 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" event={"ID":"4a94e70e-b37d-4e4b-8b46-a746ea37849f","Type":"ContainerStarted","Data":"bf33d2ddf93a5270d92f9c3ce95f5d12229ddf2cae545097a910bc80eeb19f5e"} Sep 14 13:17:51.335282 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:51.335242 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" Sep 14 13:17:51.353617 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:51.353566 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" podStartSLOduration=1.8225312200000001 podStartE2EDuration="2.353548978s" totalImagesPullingTime="531.017758ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:17:51.351665329 +0000 UTC m=+1172.014560962" watchObservedRunningTime="2026-09-14 13:17:51.353548978 +0000 UTC m=+1172.016444609" Sep 14 13:17:57.348816 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:17:57.348786 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" Sep 14 13:18:17.072385 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.072341 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7"] Sep 14 13:18:17.079247 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.079207 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.083492 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.083466 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-b8be88-openshift-default-dockercfg-cklpw\"" Sep 14 13:18:17.091906 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.091880 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7"] Sep 14 13:18:17.151621 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.151584 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-workload-certs\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.151798 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.151629 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-credential-socket\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.151798 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.151693 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484e65bc-00ba-458b-b212-edda1a7a97e3-istiod-ca-cert\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.151798 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.151729 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-workload-socket\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.151798 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.151774 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-podinfo\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.151985 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.151866 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49wpj\" (UniqueName: \"kubernetes.io/projected/484e65bc-00ba-458b-b212-edda1a7a97e3-kube-api-access-49wpj\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.151985 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.151888 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-envoy\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.151985 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.151945 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-token\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.152109 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.151986 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-data\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.252557 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.252518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49wpj\" (UniqueName: \"kubernetes.io/projected/484e65bc-00ba-458b-b212-edda1a7a97e3-kube-api-access-49wpj\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.252557 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.252558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-envoy\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.252833 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.252731 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-token\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.252833 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.252768 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-data\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.252833 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.252832 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-workload-certs\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.253046 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.252865 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-credential-socket\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.253046 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.252895 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484e65bc-00ba-458b-b212-edda1a7a97e3-istiod-ca-cert\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.253046 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.252951 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-workload-socket\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.253046 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.253020 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-podinfo\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.253280 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.253259 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-workload-certs\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.253349 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.253261 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-data\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.253398 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.253339 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-credential-socket\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.253530 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.253508 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-workload-socket\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.254011 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.253983 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484e65bc-00ba-458b-b212-edda1a7a97e3-istiod-ca-cert\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.264208 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.264177 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-envoy\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.264827 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.264805 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-podinfo\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.269545 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.267675 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-token\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.271218 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.271199 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49wpj\" (UniqueName: \"kubernetes.io/projected/484e65bc-00ba-458b-b212-edda1a7a97e3-kube-api-access-49wpj\") pod \"e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.393130 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.393093 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:17.689946 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.689898 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7"] Sep 14 13:18:17.691419 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:18:17.691386 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod484e65bc_00ba_458b_b212_edda1a7a97e3.slice/crio-cab36624f7e378b7af9e9fff51b513802e1e57739de55113048fe2842034c418 WatchSource:0}: Error finding container cab36624f7e378b7af9e9fff51b513802e1e57739de55113048fe2842034c418: Status 404 returned error can't find the container with id cab36624f7e378b7af9e9fff51b513802e1e57739de55113048fe2842034c418 Sep 14 13:18:17.693682 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.693649 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:18:17.693780 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.693726 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:18:17.693780 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:17.693770 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:18:18.514502 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:18.514409 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" event={"ID":"484e65bc-00ba-458b-b212-edda1a7a97e3","Type":"ContainerStarted","Data":"8494d7354344aba6233b6f881199a78dc5e0b1372f2ea76bd3fca8b148eadf88"} Sep 14 13:18:18.514502 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:18.514462 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" event={"ID":"484e65bc-00ba-458b-b212-edda1a7a97e3","Type":"ContainerStarted","Data":"cab36624f7e378b7af9e9fff51b513802e1e57739de55113048fe2842034c418"} Sep 14 13:18:18.540053 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:18.539999 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" podStartSLOduration=1.539982927 podStartE2EDuration="1.539982927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:18:18.539673917 +0000 UTC m=+1199.202569548" watchObservedRunningTime="2026-09-14 13:18:18.539982927 +0000 UTC m=+1199.202878557" Sep 14 13:18:19.393568 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:19.393533 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:19.399223 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:19.399197 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:19.521012 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:19.520980 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:19.522110 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:19.522089 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:19.884151 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:19.884120 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/1.log" Sep 14 13:18:19.890144 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:19.890121 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/1.log" Sep 14 13:18:21.534274 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:21.534195 2564 generic.go:356] "Generic (PLEG): container finished" podID="be11482a-7155-4cec-b70f-0456b0566843" containerID="c52727df75243ddcf885d166e8335d9c8f5408d6f88cfb77303788f7b2594074" exitCode=0 Sep 14 13:18:21.534673 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:21.534273 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" event={"ID":"be11482a-7155-4cec-b70f-0456b0566843","Type":"ContainerDied","Data":"c52727df75243ddcf885d166e8335d9c8f5408d6f88cfb77303788f7b2594074"} Sep 14 13:18:21.534781 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:21.534763 2564 scope.go:117] "RemoveContainer" containerID="c52727df75243ddcf885d166e8335d9c8f5408d6f88cfb77303788f7b2594074" Sep 14 13:18:21.535976 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:21.535919 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:18:21.536049 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:21.536016 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:18:21.536049 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:21.536045 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:18:22.542544 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:22.542510 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" event={"ID":"be11482a-7155-4cec-b70f-0456b0566843","Type":"ContainerStarted","Data":"702e3ae6c8293e47032e0e39c746a2b97f7ba2f907fb2060e700b5bee9d51c1f"} Sep 14 13:18:23.315215 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:23.315178 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:18:23.315435 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:23.315387 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 14 13:18:23.315516 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:23.315447 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" podUID="be11482a-7155-4cec-b70f-0456b0566843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 14 13:18:24.315179 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:24.315141 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 14 13:18:24.315587 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:24.315212 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" podUID="be11482a-7155-4cec-b70f-0456b0566843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 14 13:18:25.315221 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:25.315184 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 14 13:18:25.315795 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:25.315761 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" podUID="be11482a-7155-4cec-b70f-0456b0566843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 14 13:18:25.418008 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:25.417973 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l"] Sep 14 13:18:25.418235 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:25.418189 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" podUID="b7c4b47c-464e-4710-a6c0-06f58cf25bde" containerName="maas-api" containerID="cri-o://628b1cc7cf405a9da4a4b42411b1ed19e0315221a25393299b60b6b75d133264" gracePeriod=30 Sep 14 13:18:25.593661 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:25.593551 2564 generic.go:356] "Generic (PLEG): container finished" podID="b7c4b47c-464e-4710-a6c0-06f58cf25bde" containerID="628b1cc7cf405a9da4a4b42411b1ed19e0315221a25393299b60b6b75d133264" exitCode=0 Sep 14 13:18:25.593661 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:25.593617 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" event={"ID":"b7c4b47c-464e-4710-a6c0-06f58cf25bde","Type":"ContainerDied","Data":"628b1cc7cf405a9da4a4b42411b1ed19e0315221a25393299b60b6b75d133264"} Sep 14 13:18:25.749721 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:25.749693 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" Sep 14 13:18:25.881454 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:25.881419 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c4b47c-464e-4710-a6c0-06f58cf25bde-metrics-certs\") pod \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\" (UID: \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\") " Sep 14 13:18:25.881821 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:25.881800 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmq9c\" (UniqueName: \"kubernetes.io/projected/b7c4b47c-464e-4710-a6c0-06f58cf25bde-kube-api-access-fmq9c\") pod \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\" (UID: \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\") " Sep 14 13:18:25.882009 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:25.881990 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7c4b47c-464e-4710-a6c0-06f58cf25bde-maas-api-tls\") pod \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\" (UID: \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\") " Sep 14 13:18:25.882182 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:25.882166 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c4b47c-464e-4710-a6c0-06f58cf25bde-openshift-service-ca\") pod \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\" (UID: \"b7c4b47c-464e-4710-a6c0-06f58cf25bde\") " Sep 14 13:18:25.883057 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:25.883033 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7c4b47c-464e-4710-a6c0-06f58cf25bde-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b7c4b47c-464e-4710-a6c0-06f58cf25bde" (UID: "b7c4b47c-464e-4710-a6c0-06f58cf25bde"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:18:25.886999 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:25.886965 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c4b47c-464e-4710-a6c0-06f58cf25bde-kube-api-access-fmq9c" (OuterVolumeSpecName: "kube-api-access-fmq9c") pod "b7c4b47c-464e-4710-a6c0-06f58cf25bde" (UID: "b7c4b47c-464e-4710-a6c0-06f58cf25bde"). InnerVolumeSpecName "kube-api-access-fmq9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:18:25.887377 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:25.887351 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c4b47c-464e-4710-a6c0-06f58cf25bde-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b7c4b47c-464e-4710-a6c0-06f58cf25bde" (UID: "b7c4b47c-464e-4710-a6c0-06f58cf25bde"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:18:25.889305 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:25.889284 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c4b47c-464e-4710-a6c0-06f58cf25bde-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b7c4b47c-464e-4710-a6c0-06f58cf25bde" (UID: "b7c4b47c-464e-4710-a6c0-06f58cf25bde"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:18:25.984100 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:25.984063 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmq9c\" (UniqueName: \"kubernetes.io/projected/b7c4b47c-464e-4710-a6c0-06f58cf25bde-kube-api-access-fmq9c\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:18:25.984100 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:25.984094 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7c4b47c-464e-4710-a6c0-06f58cf25bde-maas-api-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:18:25.984100 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:25.984110 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7c4b47c-464e-4710-a6c0-06f58cf25bde-openshift-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:18:25.984346 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:25.984121 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7c4b47c-464e-4710-a6c0-06f58cf25bde-metrics-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:18:26.315308 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:26.315216 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 14 13:18:26.315308 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:26.315277 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" podUID="be11482a-7155-4cec-b70f-0456b0566843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 14 13:18:26.605349 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:26.605250 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" Sep 14 13:18:26.605349 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:26.605269 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l" event={"ID":"b7c4b47c-464e-4710-a6c0-06f58cf25bde","Type":"ContainerDied","Data":"ed9878f958fea920b01d8918904d991db295396dc64bc7282d1b6a82f580b8f1"} Sep 14 13:18:26.605349 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:26.605340 2564 scope.go:117] "RemoveContainer" containerID="628b1cc7cf405a9da4a4b42411b1ed19e0315221a25393299b60b6b75d133264" Sep 14 13:18:26.642353 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:26.642325 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l"] Sep 14 13:18:26.650029 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:26.650001 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-975227-55f756664-zlt6l"] Sep 14 13:18:27.314476 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:27.314443 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 14 13:18:27.314652 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:27.314492 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" podUID="be11482a-7155-4cec-b70f-0456b0566843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 14 13:18:27.812761 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:27.812715 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7c4b47c-464e-4710-a6c0-06f58cf25bde" path="/var/lib/kubelet/pods/b7c4b47c-464e-4710-a6c0-06f58cf25bde/volumes" Sep 14 13:18:28.315384 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:28.315339 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Sep 14 13:18:28.315570 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:28.315414 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" podUID="be11482a-7155-4cec-b70f-0456b0566843" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Sep 14 13:18:29.318711 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:29.318682 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:18:29.319150 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:29.319132 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:18:29.319777 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:29.319759 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:18:32.852511 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:32.852474 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq"] Sep 14 13:18:32.853012 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:32.852746 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" podUID="ba2a8136-484e-44db-9b90-e02984478f87" containerName="istio-proxy" containerID="cri-o://02f7101b4ae498fcd6f83c2cb657faf68286aaf3ddeb623822622c9e1247b9bf" gracePeriod=30 Sep 14 13:18:33.690370 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.690338 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs"] Sep 14 13:18:33.690860 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.690841 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7c4b47c-464e-4710-a6c0-06f58cf25bde" containerName="maas-api" Sep 14 13:18:33.690982 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.690863 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7c4b47c-464e-4710-a6c0-06f58cf25bde" containerName="maas-api" Sep 14 13:18:33.691041 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.690981 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7c4b47c-464e-4710-a6c0-06f58cf25bde" containerName="maas-api" Sep 14 13:18:33.694701 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.694678 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.698324 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.698300 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-4c0b93-openshift-default-dockercfg-jndcn\"" Sep 14 13:18:33.713938 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.713899 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs"] Sep 14 13:18:33.764611 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.764584 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-token\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.764780 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.764636 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-workload-certs\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.764780 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.764699 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39e4e21-c72a-4608-990a-b2fe34e05856-istiod-ca-cert\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.764780 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.764762 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-workload-socket\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.764906 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.764790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-data\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.764906 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.764815 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-podinfo\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.764906 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.764879 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq7t8\" (UniqueName: \"kubernetes.io/projected/e39e4e21-c72a-4608-990a-b2fe34e05856-kube-api-access-dq7t8\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.765038 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.764912 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-credential-socket\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.765038 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.764962 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-envoy\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.866115 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.866073 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-workload-certs\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.866636 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.866128 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39e4e21-c72a-4608-990a-b2fe34e05856-istiod-ca-cert\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.866636 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.866169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-workload-socket\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.866636 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.866209 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-data\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.866636 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.866253 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-podinfo\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.866636 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.866315 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq7t8\" (UniqueName: \"kubernetes.io/projected/e39e4e21-c72a-4608-990a-b2fe34e05856-kube-api-access-dq7t8\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.866636 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.866348 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-credential-socket\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.866636 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.866380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-envoy\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.866636 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.866459 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-token\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.866636 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.866610 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-workload-socket\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.866636 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.866631 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-workload-certs\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.867148 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.866699 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-data\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.867148 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.866858 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-credential-socket\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.867271 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.867248 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39e4e21-c72a-4608-990a-b2fe34e05856-istiod-ca-cert\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.868860 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.868836 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-envoy\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.869081 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.869063 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-podinfo\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.879187 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.879160 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-token\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:33.879271 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:33.879210 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq7t8\" (UniqueName: \"kubernetes.io/projected/e39e4e21-c72a-4608-990a-b2fe34e05856-kube-api-access-dq7t8\") pod \"e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:34.006205 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:34.006113 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:34.300671 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:34.299245 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs"] Sep 14 13:18:34.303122 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:18:34.303073 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39e4e21_c72a_4608_990a_b2fe34e05856.slice/crio-c435e78a7421013b4bdbe69ca39b2b702dba5eaab1a724829c50ed95fe3fb164 WatchSource:0}: Error finding container c435e78a7421013b4bdbe69ca39b2b702dba5eaab1a724829c50ed95fe3fb164: Status 404 returned error can't find the container with id c435e78a7421013b4bdbe69ca39b2b702dba5eaab1a724829c50ed95fe3fb164 Sep 14 13:18:34.305575 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:34.305535 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:18:34.305682 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:34.305627 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:18:34.305682 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:34.305668 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:18:34.657365 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:34.657328 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" event={"ID":"e39e4e21-c72a-4608-990a-b2fe34e05856","Type":"ContainerStarted","Data":"7289e0dbcc47aed67042b3c7371320cb04f092e262f8cfccd31cf319ea0b3db3"} Sep 14 13:18:34.657540 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:34.657391 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" event={"ID":"e39e4e21-c72a-4608-990a-b2fe34e05856","Type":"ContainerStarted","Data":"c435e78a7421013b4bdbe69ca39b2b702dba5eaab1a724829c50ed95fe3fb164"} Sep 14 13:18:34.688140 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:34.688093 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" podStartSLOduration=1.688079558 podStartE2EDuration="1.688079558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:18:34.687650187 +0000 UTC m=+1215.350545817" watchObservedRunningTime="2026-09-14 13:18:34.688079558 +0000 UTC m=+1215.350975188" Sep 14 13:18:35.007291 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:35.007208 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:35.009238 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:35.009203 2564 patch_prober.go:28] interesting pod/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Sep 14 13:18:35.009378 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:35.009260 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" podUID="e39e4e21-c72a-4608-990a-b2fe34e05856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Sep 14 13:18:35.131729 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:35.131692 2564 patch_prober.go:28] interesting pod/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 13:18:35.131900 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:35.131758 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" podUID="ba2a8136-484e-44db-9b90-e02984478f87" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 13:18:36.010634 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:36.010606 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:36.670349 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:36.670316 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:36.671272 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:36.671251 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:18:38.115372 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.115312 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:18:38.220245 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.220214 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba2a8136-484e-44db-9b90-e02984478f87-istio-token\") pod \"ba2a8136-484e-44db-9b90-e02984478f87\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " Sep 14 13:18:38.220451 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.220274 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba2a8136-484e-44db-9b90-e02984478f87-istio-podinfo\") pod \"ba2a8136-484e-44db-9b90-e02984478f87\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " Sep 14 13:18:38.220451 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.220329 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba2a8136-484e-44db-9b90-e02984478f87-istiod-ca-cert\") pod \"ba2a8136-484e-44db-9b90-e02984478f87\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " Sep 14 13:18:38.220451 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.220375 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-workload-certs\") pod \"ba2a8136-484e-44db-9b90-e02984478f87\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " Sep 14 13:18:38.220451 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.220406 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-workload-socket\") pod \"ba2a8136-484e-44db-9b90-e02984478f87\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " Sep 14 13:18:38.220451 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.220451 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-credential-socket\") pod \"ba2a8136-484e-44db-9b90-e02984478f87\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " Sep 14 13:18:38.220713 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.220510 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v26nb\" (UniqueName: \"kubernetes.io/projected/ba2a8136-484e-44db-9b90-e02984478f87-kube-api-access-v26nb\") pod \"ba2a8136-484e-44db-9b90-e02984478f87\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " Sep 14 13:18:38.220713 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.220537 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-istio-data\") pod \"ba2a8136-484e-44db-9b90-e02984478f87\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " Sep 14 13:18:38.220713 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.220587 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-istio-envoy\") pod \"ba2a8136-484e-44db-9b90-e02984478f87\" (UID: \"ba2a8136-484e-44db-9b90-e02984478f87\") " Sep 14 13:18:38.220713 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.220681 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ba2a8136-484e-44db-9b90-e02984478f87" (UID: "ba2a8136-484e-44db-9b90-e02984478f87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:18:38.220918 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.220726 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ba2a8136-484e-44db-9b90-e02984478f87" (UID: "ba2a8136-484e-44db-9b90-e02984478f87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:18:38.220918 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.220761 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ba2a8136-484e-44db-9b90-e02984478f87" (UID: "ba2a8136-484e-44db-9b90-e02984478f87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:18:38.220918 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.220803 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba2a8136-484e-44db-9b90-e02984478f87-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ba2a8136-484e-44db-9b90-e02984478f87" (UID: "ba2a8136-484e-44db-9b90-e02984478f87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:18:38.221280 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.221015 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-istio-data" (OuterVolumeSpecName: "istio-data") pod "ba2a8136-484e-44db-9b90-e02984478f87" (UID: "ba2a8136-484e-44db-9b90-e02984478f87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:18:38.221280 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.221101 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba2a8136-484e-44db-9b90-e02984478f87-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:18:38.221280 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.221115 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:18:38.221280 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.221124 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:18:38.221280 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.221131 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:18:38.221280 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.221139 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:18:38.222743 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.222709 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba2a8136-484e-44db-9b90-e02984478f87-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ba2a8136-484e-44db-9b90-e02984478f87" (UID: "ba2a8136-484e-44db-9b90-e02984478f87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 13:18:38.222852 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.222822 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba2a8136-484e-44db-9b90-e02984478f87-istio-token" (OuterVolumeSpecName: "istio-token") pod "ba2a8136-484e-44db-9b90-e02984478f87" (UID: "ba2a8136-484e-44db-9b90-e02984478f87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:18:38.223278 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.223256 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba2a8136-484e-44db-9b90-e02984478f87-kube-api-access-v26nb" (OuterVolumeSpecName: "kube-api-access-v26nb") pod "ba2a8136-484e-44db-9b90-e02984478f87" (UID: "ba2a8136-484e-44db-9b90-e02984478f87"). InnerVolumeSpecName "kube-api-access-v26nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:18:38.223278 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.223266 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ba2a8136-484e-44db-9b90-e02984478f87" (UID: "ba2a8136-484e-44db-9b90-e02984478f87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:18:38.322547 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.322515 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba2a8136-484e-44db-9b90-e02984478f87-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:18:38.322547 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.322540 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba2a8136-484e-44db-9b90-e02984478f87-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:18:38.322547 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.322550 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v26nb\" (UniqueName: \"kubernetes.io/projected/ba2a8136-484e-44db-9b90-e02984478f87-kube-api-access-v26nb\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:18:38.322774 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.322560 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba2a8136-484e-44db-9b90-e02984478f87-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:18:38.683944 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.683901 2564 generic.go:356] "Generic (PLEG): container finished" podID="ba2a8136-484e-44db-9b90-e02984478f87" containerID="02f7101b4ae498fcd6f83c2cb657faf68286aaf3ddeb623822622c9e1247b9bf" exitCode=0 Sep 14 13:18:38.684112 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.683966 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" Sep 14 13:18:38.684112 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.683962 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" event={"ID":"ba2a8136-484e-44db-9b90-e02984478f87","Type":"ContainerDied","Data":"02f7101b4ae498fcd6f83c2cb657faf68286aaf3ddeb623822622c9e1247b9bf"} Sep 14 13:18:38.684112 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.684097 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq" event={"ID":"ba2a8136-484e-44db-9b90-e02984478f87","Type":"ContainerDied","Data":"7123555bd70bbabeea18575e5ddde48aab6806a935801a5b7755af77392ff051"} Sep 14 13:18:38.684225 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.684119 2564 scope.go:117] "RemoveContainer" containerID="02f7101b4ae498fcd6f83c2cb657faf68286aaf3ddeb623822622c9e1247b9bf" Sep 14 13:18:38.694605 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.694588 2564 scope.go:117] "RemoveContainer" containerID="90ec4371213c183c13f301110cd291173d910cbc581a28f08678a90bc07128f3" Sep 14 13:18:38.703694 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.703674 2564 scope.go:117] "RemoveContainer" containerID="02f7101b4ae498fcd6f83c2cb657faf68286aaf3ddeb623822622c9e1247b9bf" Sep 14 13:18:38.703985 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:18:38.703963 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02f7101b4ae498fcd6f83c2cb657faf68286aaf3ddeb623822622c9e1247b9bf\": container with ID starting with 02f7101b4ae498fcd6f83c2cb657faf68286aaf3ddeb623822622c9e1247b9bf not found: ID does not exist" containerID="02f7101b4ae498fcd6f83c2cb657faf68286aaf3ddeb623822622c9e1247b9bf" Sep 14 13:18:38.704054 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.703995 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f7101b4ae498fcd6f83c2cb657faf68286aaf3ddeb623822622c9e1247b9bf"} err="failed to get container status \"02f7101b4ae498fcd6f83c2cb657faf68286aaf3ddeb623822622c9e1247b9bf\": rpc error: code = NotFound desc = could not find container \"02f7101b4ae498fcd6f83c2cb657faf68286aaf3ddeb623822622c9e1247b9bf\": container with ID starting with 02f7101b4ae498fcd6f83c2cb657faf68286aaf3ddeb623822622c9e1247b9bf not found: ID does not exist" Sep 14 13:18:38.704054 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.704011 2564 scope.go:117] "RemoveContainer" containerID="90ec4371213c183c13f301110cd291173d910cbc581a28f08678a90bc07128f3" Sep 14 13:18:38.704270 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:18:38.704252 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90ec4371213c183c13f301110cd291173d910cbc581a28f08678a90bc07128f3\": container with ID starting with 90ec4371213c183c13f301110cd291173d910cbc581a28f08678a90bc07128f3 not found: ID does not exist" containerID="90ec4371213c183c13f301110cd291173d910cbc581a28f08678a90bc07128f3" Sep 14 13:18:38.704324 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.704280 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90ec4371213c183c13f301110cd291173d910cbc581a28f08678a90bc07128f3"} err="failed to get container status \"90ec4371213c183c13f301110cd291173d910cbc581a28f08678a90bc07128f3\": rpc error: code = NotFound desc = could not find container \"90ec4371213c183c13f301110cd291173d910cbc581a28f08678a90bc07128f3\": container with ID starting with 90ec4371213c183c13f301110cd291173d910cbc581a28f08678a90bc07128f3 not found: ID does not exist" Sep 14 13:18:38.710503 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.710483 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq"] Sep 14 13:18:38.715363 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:38.715341 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-ace1c6-openshift-default-87545cf7b-gdfjq"] Sep 14 13:18:39.815297 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:39.815266 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba2a8136-484e-44db-9b90-e02984478f87" path="/var/lib/kubelet/pods/ba2a8136-484e-44db-9b90-e02984478f87/volumes" Sep 14 13:18:49.927421 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:49.927386 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb"] Sep 14 13:18:49.928065 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:49.928043 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba2a8136-484e-44db-9b90-e02984478f87" containerName="istio-proxy" Sep 14 13:18:49.928065 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:49.928066 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba2a8136-484e-44db-9b90-e02984478f87" containerName="istio-proxy" Sep 14 13:18:49.928171 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:49.928081 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba2a8136-484e-44db-9b90-e02984478f87" containerName="istio-proxy" Sep 14 13:18:49.928171 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:49.928088 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba2a8136-484e-44db-9b90-e02984478f87" containerName="istio-proxy" Sep 14 13:18:49.928251 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:49.928175 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba2a8136-484e-44db-9b90-e02984478f87" containerName="istio-proxy" Sep 14 13:18:49.928251 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:49.928188 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba2a8136-484e-44db-9b90-e02984478f87" containerName="istio-proxy" Sep 14 13:18:49.930777 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:49.930755 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:49.935082 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:49.935064 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-336ccc18-openshift-default-dockercfg-5r75j\"" Sep 14 13:18:49.951242 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:49.951218 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb"] Sep 14 13:18:50.049249 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.049203 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-workload-certs\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.049430 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.049302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-workload-socket\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.049430 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.049342 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-credential-socket\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.049430 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.049382 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfcmp\" (UniqueName: \"kubernetes.io/projected/b1131fb1-a049-418b-be10-9c7c052f7c01-kube-api-access-xfcmp\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.049689 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.049431 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1131fb1-a049-418b-be10-9c7c052f7c01-istiod-ca-cert\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.049689 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.049455 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-podinfo\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.049689 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.049480 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-data\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.049689 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.049521 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-envoy\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.049689 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.049579 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-token\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.150814 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.150782 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-envoy\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.150986 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.150833 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-token\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.150986 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.150876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-workload-certs\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.150986 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.150948 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-workload-socket\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.151161 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.150993 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-credential-socket\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.151161 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.151035 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfcmp\" (UniqueName: \"kubernetes.io/projected/b1131fb1-a049-418b-be10-9c7c052f7c01-kube-api-access-xfcmp\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.151161 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.151087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1131fb1-a049-418b-be10-9c7c052f7c01-istiod-ca-cert\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.151314 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.151301 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-workload-certs\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.151371 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.151320 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-podinfo\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.151371 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.151361 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-data\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.151655 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.151627 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-credential-socket\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.151784 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.151668 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-workload-socket\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.151784 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.151701 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1131fb1-a049-418b-be10-9c7c052f7c01-istiod-ca-cert\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.151917 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.151896 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-data\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.153296 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.153276 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-envoy\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.154038 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.154018 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-podinfo\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.160896 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.160873 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-token\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.162261 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.162241 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfcmp\" (UniqueName: \"kubernetes.io/projected/b1131fb1-a049-418b-be10-9c7c052f7c01-kube-api-access-xfcmp\") pod \"e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.240902 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.240872 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:50.534393 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.534364 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb"] Sep 14 13:18:50.535455 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:18:50.535425 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1131fb1_a049_418b_be10_9c7c052f7c01.slice/crio-d31cce034755669f53eb1be5a093d81349c5c8b919558b30cd17997eaf0012e9 WatchSource:0}: Error finding container d31cce034755669f53eb1be5a093d81349c5c8b919558b30cd17997eaf0012e9: Status 404 returned error can't find the container with id d31cce034755669f53eb1be5a093d81349c5c8b919558b30cd17997eaf0012e9 Sep 14 13:18:50.538015 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.537983 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:18:50.538096 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.538048 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:18:50.538096 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.538079 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:18:50.762718 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.762664 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" event={"ID":"b1131fb1-a049-418b-be10-9c7c052f7c01","Type":"ContainerStarted","Data":"c133aa6b3e1e3c33192c3acbe35f8e67573c775ce4c84dd3b5ef86f0c8781f67"} Sep 14 13:18:50.762994 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.762741 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" event={"ID":"b1131fb1-a049-418b-be10-9c7c052f7c01","Type":"ContainerStarted","Data":"d31cce034755669f53eb1be5a093d81349c5c8b919558b30cd17997eaf0012e9"} Sep 14 13:18:50.785054 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:50.784963 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" podStartSLOduration=1.784947622 podStartE2EDuration="1.784947622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:18:50.781868493 +0000 UTC m=+1231.444764125" watchObservedRunningTime="2026-09-14 13:18:50.784947622 +0000 UTC m=+1231.447843252" Sep 14 13:18:51.241303 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:51.241255 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:51.246988 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:51.246960 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:51.768485 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:51.768447 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:51.769590 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:51.769569 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:18:53.782320 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:53.782284 2564 generic.go:356] "Generic (PLEG): container finished" podID="484e65bc-00ba-458b-b212-edda1a7a97e3" containerID="8494d7354344aba6233b6f881199a78dc5e0b1372f2ea76bd3fca8b148eadf88" exitCode=0 Sep 14 13:18:53.782746 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:53.782356 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" event={"ID":"484e65bc-00ba-458b-b212-edda1a7a97e3","Type":"ContainerDied","Data":"8494d7354344aba6233b6f881199a78dc5e0b1372f2ea76bd3fca8b148eadf88"} Sep 14 13:18:53.782843 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:53.782829 2564 scope.go:117] "RemoveContainer" containerID="8494d7354344aba6233b6f881199a78dc5e0b1372f2ea76bd3fca8b148eadf88" Sep 14 13:18:53.784008 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:53.783978 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:18:53.784122 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:53.784056 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:18:53.784122 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:53.784104 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:18:54.790794 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:54.790759 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" event={"ID":"484e65bc-00ba-458b-b212-edda1a7a97e3","Type":"ContainerStarted","Data":"e1c7b215f88f6a2dc02cc25ff595238161d549c26053a0f2c5f56d2ac10b13db"} Sep 14 13:18:55.393939 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:55.393883 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:18:55.394254 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:55.394229 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 14 13:18:55.394358 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:55.394286 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" podUID="484e65bc-00ba-458b-b212-edda1a7a97e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 14 13:18:56.393567 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:56.393534 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 14 13:18:56.394066 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:56.393607 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" podUID="484e65bc-00ba-458b-b212-edda1a7a97e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 14 13:18:57.394031 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:57.393989 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 14 13:18:57.394433 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:57.394041 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" podUID="484e65bc-00ba-458b-b212-edda1a7a97e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 14 13:18:58.017432 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:58.017391 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2"] Sep 14 13:18:58.017659 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:58.017621 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" podUID="0fd30d61-45b8-4586-8612-2b2caeb8a1ec" containerName="istio-proxy" containerID="cri-o://0bb0da257e51dc27f0b0d8ce9b92d8f0ce719fa35a24aed2b4986378c82ea43d" gracePeriod=30 Sep 14 13:18:58.394390 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:58.394356 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 14 13:18:58.394805 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:58.394428 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" podUID="484e65bc-00ba-458b-b212-edda1a7a97e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 14 13:18:59.393517 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:59.393476 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 14 13:18:59.393712 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:18:59.393546 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" podUID="484e65bc-00ba-458b-b212-edda1a7a97e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 14 13:19:00.394535 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:00.394497 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Sep 14 13:19:00.395077 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:00.394691 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" podUID="484e65bc-00ba-458b-b212-edda1a7a97e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Sep 14 13:19:01.424649 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:01.424613 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:19:01.425403 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:01.425374 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:19:01.436461 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:01.436421 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:19:03.275068 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.275042 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:19:03.313506 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.313427 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-workload-certs\") pod \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " Sep 14 13:19:03.313506 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.313482 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-data\") pod \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " Sep 14 13:19:03.313725 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.313547 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-workload-socket\") pod \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " Sep 14 13:19:03.313725 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.313577 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istiod-ca-cert\") pod \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " Sep 14 13:19:03.313725 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.313623 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvlwl\" (UniqueName: \"kubernetes.io/projected/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-kube-api-access-hvlwl\") pod \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " Sep 14 13:19:03.313725 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.313648 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-envoy\") pod \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " Sep 14 13:19:03.313725 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.313666 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0fd30d61-45b8-4586-8612-2b2caeb8a1ec" (UID: "0fd30d61-45b8-4586-8612-2b2caeb8a1ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:19:03.313965 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.313755 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-data" (OuterVolumeSpecName: "istio-data") pod "0fd30d61-45b8-4586-8612-2b2caeb8a1ec" (UID: "0fd30d61-45b8-4586-8612-2b2caeb8a1ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:19:03.313965 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.313802 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-token\") pod \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " Sep 14 13:19:03.313965 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.313847 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0fd30d61-45b8-4586-8612-2b2caeb8a1ec" (UID: "0fd30d61-45b8-4586-8612-2b2caeb8a1ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:19:03.313965 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.313856 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-podinfo\") pod \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " Sep 14 13:19:03.313965 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.313913 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-credential-socket\") pod \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\" (UID: \"0fd30d61-45b8-4586-8612-2b2caeb8a1ec\") " Sep 14 13:19:03.314239 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.314073 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0fd30d61-45b8-4586-8612-2b2caeb8a1ec" (UID: "0fd30d61-45b8-4586-8612-2b2caeb8a1ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:19:03.314300 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.314232 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0fd30d61-45b8-4586-8612-2b2caeb8a1ec" (UID: "0fd30d61-45b8-4586-8612-2b2caeb8a1ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:19:03.314457 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.314429 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:19:03.314457 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.314454 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:19:03.314765 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.314468 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:19:03.314765 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.314481 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:19:03.314765 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.314558 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:19:03.316348 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.316321 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-kube-api-access-hvlwl" (OuterVolumeSpecName: "kube-api-access-hvlwl") pod "0fd30d61-45b8-4586-8612-2b2caeb8a1ec" (UID: "0fd30d61-45b8-4586-8612-2b2caeb8a1ec"). InnerVolumeSpecName "kube-api-access-hvlwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:19:03.316444 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.316356 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-token" (OuterVolumeSpecName: "istio-token") pod "0fd30d61-45b8-4586-8612-2b2caeb8a1ec" (UID: "0fd30d61-45b8-4586-8612-2b2caeb8a1ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:19:03.316793 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.316772 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0fd30d61-45b8-4586-8612-2b2caeb8a1ec" (UID: "0fd30d61-45b8-4586-8612-2b2caeb8a1ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:19:03.316854 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.316791 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0fd30d61-45b8-4586-8612-2b2caeb8a1ec" (UID: "0fd30d61-45b8-4586-8612-2b2caeb8a1ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 13:19:03.415489 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.415452 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvlwl\" (UniqueName: \"kubernetes.io/projected/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-kube-api-access-hvlwl\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:19:03.415489 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.415486 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:19:03.415489 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.415495 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:19:03.415726 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.415502 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fd30d61-45b8-4586-8612-2b2caeb8a1ec-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:19:03.853700 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.853668 2564 generic.go:356] "Generic (PLEG): container finished" podID="0fd30d61-45b8-4586-8612-2b2caeb8a1ec" containerID="0bb0da257e51dc27f0b0d8ce9b92d8f0ce719fa35a24aed2b4986378c82ea43d" exitCode=0 Sep 14 13:19:03.853895 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.853717 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" Sep 14 13:19:03.853895 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.853753 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" event={"ID":"0fd30d61-45b8-4586-8612-2b2caeb8a1ec","Type":"ContainerDied","Data":"0bb0da257e51dc27f0b0d8ce9b92d8f0ce719fa35a24aed2b4986378c82ea43d"} Sep 14 13:19:03.853895 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.853818 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2" event={"ID":"0fd30d61-45b8-4586-8612-2b2caeb8a1ec","Type":"ContainerDied","Data":"d7d5c0579ca58ede4df397e10e6ef26152d39010029e6ac5ab2d542e8a64b4a2"} Sep 14 13:19:03.853895 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.853848 2564 scope.go:117] "RemoveContainer" containerID="0bb0da257e51dc27f0b0d8ce9b92d8f0ce719fa35a24aed2b4986378c82ea43d" Sep 14 13:19:03.863446 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.863423 2564 scope.go:117] "RemoveContainer" containerID="8ba43d861dda923ac88a1324931e7b1fac21bd170adbb536f178aa00c4125ba8" Sep 14 13:19:03.872163 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.872146 2564 scope.go:117] "RemoveContainer" containerID="0bb0da257e51dc27f0b0d8ce9b92d8f0ce719fa35a24aed2b4986378c82ea43d" Sep 14 13:19:03.872449 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:19:03.872430 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb0da257e51dc27f0b0d8ce9b92d8f0ce719fa35a24aed2b4986378c82ea43d\": container with ID starting with 0bb0da257e51dc27f0b0d8ce9b92d8f0ce719fa35a24aed2b4986378c82ea43d not found: ID does not exist" containerID="0bb0da257e51dc27f0b0d8ce9b92d8f0ce719fa35a24aed2b4986378c82ea43d" Sep 14 13:19:03.872525 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.872461 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb0da257e51dc27f0b0d8ce9b92d8f0ce719fa35a24aed2b4986378c82ea43d"} err="failed to get container status \"0bb0da257e51dc27f0b0d8ce9b92d8f0ce719fa35a24aed2b4986378c82ea43d\": rpc error: code = NotFound desc = could not find container \"0bb0da257e51dc27f0b0d8ce9b92d8f0ce719fa35a24aed2b4986378c82ea43d\": container with ID starting with 0bb0da257e51dc27f0b0d8ce9b92d8f0ce719fa35a24aed2b4986378c82ea43d not found: ID does not exist" Sep 14 13:19:03.872525 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.872482 2564 scope.go:117] "RemoveContainer" containerID="8ba43d861dda923ac88a1324931e7b1fac21bd170adbb536f178aa00c4125ba8" Sep 14 13:19:03.872762 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:19:03.872744 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba43d861dda923ac88a1324931e7b1fac21bd170adbb536f178aa00c4125ba8\": container with ID starting with 8ba43d861dda923ac88a1324931e7b1fac21bd170adbb536f178aa00c4125ba8 not found: ID does not exist" containerID="8ba43d861dda923ac88a1324931e7b1fac21bd170adbb536f178aa00c4125ba8" Sep 14 13:19:03.872805 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.872770 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba43d861dda923ac88a1324931e7b1fac21bd170adbb536f178aa00c4125ba8"} err="failed to get container status \"8ba43d861dda923ac88a1324931e7b1fac21bd170adbb536f178aa00c4125ba8\": rpc error: code = NotFound desc = could not find container \"8ba43d861dda923ac88a1324931e7b1fac21bd170adbb536f178aa00c4125ba8\": container with ID starting with 8ba43d861dda923ac88a1324931e7b1fac21bd170adbb536f178aa00c4125ba8 not found: ID does not exist" Sep 14 13:19:03.880569 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.880544 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2"] Sep 14 13:19:03.887110 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:03.887091 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cadd9c-openshift-default-7df5ccf54d-vqjp2"] Sep 14 13:19:05.812796 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:05.812765 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fd30d61-45b8-4586-8612-2b2caeb8a1ec" path="/var/lib/kubelet/pods/0fd30d61-45b8-4586-8612-2b2caeb8a1ec/volumes" Sep 14 13:19:17.937616 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:17.937529 2564 generic.go:356] "Generic (PLEG): container finished" podID="b1131fb1-a049-418b-be10-9c7c052f7c01" containerID="c133aa6b3e1e3c33192c3acbe35f8e67573c775ce4c84dd3b5ef86f0c8781f67" exitCode=0 Sep 14 13:19:17.937616 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:17.937584 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" event={"ID":"b1131fb1-a049-418b-be10-9c7c052f7c01","Type":"ContainerDied","Data":"c133aa6b3e1e3c33192c3acbe35f8e67573c775ce4c84dd3b5ef86f0c8781f67"} Sep 14 13:19:17.938098 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:17.938035 2564 scope.go:117] "RemoveContainer" containerID="c133aa6b3e1e3c33192c3acbe35f8e67573c775ce4c84dd3b5ef86f0c8781f67" Sep 14 13:19:17.939254 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:17.939221 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:19:17.939352 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:17.939301 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:19:17.939352 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:17.939338 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:19:18.946606 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:18.946551 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" event={"ID":"b1131fb1-a049-418b-be10-9c7c052f7c01","Type":"ContainerStarted","Data":"36512dc773c793ec13df2137e7994331cbd10e1704006525450433d54df6613a"} Sep 14 13:19:19.241290 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:19.241193 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:19:19.242171 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:19.242144 2564 patch_prober.go:28] interesting pod/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 14 13:19:19.242277 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:19.242225 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" podUID="b1131fb1-a049-418b-be10-9c7c052f7c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 14 13:19:20.241314 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:20.241225 2564 patch_prober.go:28] interesting pod/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 14 13:19:20.241314 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:20.241282 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" podUID="b1131fb1-a049-418b-be10-9c7c052f7c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 14 13:19:21.241578 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:21.241520 2564 patch_prober.go:28] interesting pod/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 14 13:19:21.242107 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:21.241609 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" podUID="b1131fb1-a049-418b-be10-9c7c052f7c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 14 13:19:21.768736 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:21.768704 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:19:22.241557 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:22.241504 2564 patch_prober.go:28] interesting pod/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 14 13:19:22.241773 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:22.241573 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" podUID="b1131fb1-a049-418b-be10-9c7c052f7c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 14 13:19:23.241565 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:23.241527 2564 patch_prober.go:28] interesting pod/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 14 13:19:23.241762 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:23.241584 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" podUID="b1131fb1-a049-418b-be10-9c7c052f7c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 14 13:19:24.241640 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:24.241609 2564 patch_prober.go:28] interesting pod/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Sep 14 13:19:24.242064 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:24.241660 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" podUID="b1131fb1-a049-418b-be10-9c7c052f7c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Sep 14 13:19:25.246062 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:25.246030 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:19:25.247174 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:25.247147 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:19:27.883803 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:27.883758 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm"] Sep 14 13:19:27.884455 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:27.884435 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fd30d61-45b8-4586-8612-2b2caeb8a1ec" containerName="istio-proxy" Sep 14 13:19:27.884549 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:27.884458 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fd30d61-45b8-4586-8612-2b2caeb8a1ec" containerName="istio-proxy" Sep 14 13:19:27.884549 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:27.884471 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fd30d61-45b8-4586-8612-2b2caeb8a1ec" containerName="istio-proxy" Sep 14 13:19:27.884549 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:27.884478 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fd30d61-45b8-4586-8612-2b2caeb8a1ec" containerName="istio-proxy" Sep 14 13:19:27.884713 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:27.884589 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fd30d61-45b8-4586-8612-2b2caeb8a1ec" containerName="istio-proxy" Sep 14 13:19:27.888083 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:27.888062 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:27.890942 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:27.890909 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-eff5e8-openshift-default-dockercfg-hbl77\"" Sep 14 13:19:27.907937 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:27.907901 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm"] Sep 14 13:19:27.986908 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:27.986871 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-workload-certs\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:27.987370 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:27.987346 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-credential-socket\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:27.987498 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:27.987481 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e444894f-628c-4794-8a59-886c3ccb0674-istiod-ca-cert\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:27.987632 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:27.987615 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e444894f-628c-4794-8a59-886c3ccb0674-istio-token\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:27.987786 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:27.987763 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-istio-envoy\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:27.987908 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:27.987896 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e444894f-628c-4794-8a59-886c3ccb0674-istio-podinfo\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:27.988039 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:27.988023 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-istio-data\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:27.988149 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:27.988134 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2n78\" (UniqueName: \"kubernetes.io/projected/e444894f-628c-4794-8a59-886c3ccb0674-kube-api-access-j2n78\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:27.988285 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:27.988265 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-workload-socket\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:28.089895 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.089862 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-workload-certs\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:28.090118 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.089909 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-credential-socket\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:28.090118 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.089958 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e444894f-628c-4794-8a59-886c3ccb0674-istiod-ca-cert\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:28.090118 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.089992 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e444894f-628c-4794-8a59-886c3ccb0674-istio-token\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:28.090118 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.090017 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-istio-envoy\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:28.090118 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.090035 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e444894f-628c-4794-8a59-886c3ccb0674-istio-podinfo\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:28.090411 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.090200 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-istio-data\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:28.090411 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.090254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2n78\" (UniqueName: \"kubernetes.io/projected/e444894f-628c-4794-8a59-886c3ccb0674-kube-api-access-j2n78\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:28.090411 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.090286 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-workload-socket\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:28.090411 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.090333 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-credential-socket\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:28.090621 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.090583 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-istio-data\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:28.090681 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.090669 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-workload-socket\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:28.090738 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.090702 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-workload-certs\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:28.091115 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.091091 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e444894f-628c-4794-8a59-886c3ccb0674-istiod-ca-cert\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:28.092438 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.092415 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-istio-envoy\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:28.092642 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.092626 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e444894f-628c-4794-8a59-886c3ccb0674-istio-podinfo\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:28.098226 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.098203 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e444894f-628c-4794-8a59-886c3ccb0674-istio-token\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:28.098463 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.098445 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2n78\" (UniqueName: \"kubernetes.io/projected/e444894f-628c-4794-8a59-886c3ccb0674-kube-api-access-j2n78\") pod \"e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:28.199238 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.199147 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:28.478409 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.478377 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm"] Sep 14 13:19:28.480084 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:19:28.480056 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode444894f_628c_4794_8a59_886c3ccb0674.slice/crio-f01f80f429c42206dab43f485c18e6be7d280706d87491d7646ca5085278b685 WatchSource:0}: Error finding container f01f80f429c42206dab43f485c18e6be7d280706d87491d7646ca5085278b685: Status 404 returned error can't find the container with id f01f80f429c42206dab43f485c18e6be7d280706d87491d7646ca5085278b685 Sep 14 13:19:28.482509 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.482476 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:19:28.482594 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.482538 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:19:28.482594 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:28.482564 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:19:29.021155 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:29.021121 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" event={"ID":"e444894f-628c-4794-8a59-886c3ccb0674","Type":"ContainerStarted","Data":"3f5115815e08ffff87641c2bfcb44bcc0210afcb70addd06c04785f3560773a8"} Sep 14 13:19:29.021625 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:29.021178 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" event={"ID":"e444894f-628c-4794-8a59-886c3ccb0674","Type":"ContainerStarted","Data":"f01f80f429c42206dab43f485c18e6be7d280706d87491d7646ca5085278b685"} Sep 14 13:19:29.046161 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:29.046116 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" podStartSLOduration=2.046098892 podStartE2EDuration="2.046098892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:19:29.043656246 +0000 UTC m=+1269.706551914" watchObservedRunningTime="2026-09-14 13:19:29.046098892 +0000 UTC m=+1269.708994522" Sep 14 13:19:29.200159 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:29.200128 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:29.205609 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:29.205573 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:30.026877 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:30.026852 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:30.027825 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:30.027809 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:19:36.206981 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:36.206950 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt"] Sep 14 13:19:36.207496 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:36.207477 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fd30d61-45b8-4586-8612-2b2caeb8a1ec" containerName="istio-proxy" Sep 14 13:19:36.210538 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:36.210512 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" Sep 14 13:19:36.213700 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:36.213680 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-eff5e8-serving-cert\"" Sep 14 13:19:36.224091 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:36.224067 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt"] Sep 14 13:19:36.280464 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:36.280430 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt\" (UID: \"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" Sep 14 13:19:36.280641 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:36.280499 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt\" (UID: \"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" Sep 14 13:19:36.280641 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:36.280565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sj7d\" (UniqueName: \"kubernetes.io/projected/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-kube-api-access-6sj7d\") pod \"maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt\" (UID: \"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" Sep 14 13:19:36.280729 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:36.280647 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt\" (UID: \"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" Sep 14 13:19:36.381972 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:36.381904 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt\" (UID: \"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" Sep 14 13:19:36.382173 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:36.382033 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt\" (UID: \"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" Sep 14 13:19:36.382173 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:36.382062 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sj7d\" (UniqueName: \"kubernetes.io/projected/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-kube-api-access-6sj7d\") pod \"maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt\" (UID: \"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" Sep 14 13:19:36.382173 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:36.382115 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt\" (UID: \"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" Sep 14 13:19:36.382576 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:36.382555 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt\" (UID: \"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" Sep 14 13:19:36.384600 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:36.384573 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt\" (UID: \"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" Sep 14 13:19:36.384779 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:36.384760 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt\" (UID: \"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" Sep 14 13:19:36.393876 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:36.393854 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sj7d\" (UniqueName: \"kubernetes.io/projected/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-kube-api-access-6sj7d\") pod \"maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt\" (UID: \"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" Sep 14 13:19:36.522691 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:36.522594 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" Sep 14 13:19:36.814782 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:36.814756 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt"] Sep 14 13:19:36.815939 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:19:36.815891 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c0a848_fb3c_4b6b_94c5_eb0b7f249bea.slice/crio-1c0542bf036d6c4aa518ef05f0d6a3b8663f609e6fdb8258be58f12a3bef6a96 WatchSource:0}: Error finding container 1c0542bf036d6c4aa518ef05f0d6a3b8663f609e6fdb8258be58f12a3bef6a96: Status 404 returned error can't find the container with id 1c0542bf036d6c4aa518ef05f0d6a3b8663f609e6fdb8258be58f12a3bef6a96 Sep 14 13:19:36.817393 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:36.817373 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 13:19:37.074321 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:37.074220 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" event={"ID":"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea","Type":"ContainerStarted","Data":"1c0542bf036d6c4aa518ef05f0d6a3b8663f609e6fdb8258be58f12a3bef6a96"} Sep 14 13:19:38.081849 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:38.081811 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" event={"ID":"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea","Type":"ContainerStarted","Data":"74a9f28fdac065f54bf7f3ec95dc79efecf828cc4f11ec643924d8b27fff1035"} Sep 14 13:19:38.082260 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:38.081948 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" Sep 14 13:19:38.115514 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:38.115459 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" podStartSLOduration=1.7349117779999998 podStartE2EDuration="2.11544395s" totalImagesPullingTime="380.532172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:19:38.11230063 +0000 UTC m=+1278.775196259" watchObservedRunningTime="2026-09-14 13:19:38.11544395 +0000 UTC m=+1278.778339580" Sep 14 13:19:44.093745 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:19:44.093715 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" Sep 14 13:20:01.068087 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:01.068042 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 14 13:20:01.068646 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:01.068109 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" podUID="e444894f-628c-4794-8a59-886c3ccb0674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 14 13:20:02.929564 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:02.929521 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs"] Sep 14 13:20:02.930051 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:02.929791 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" podUID="e39e4e21-c72a-4608-990a-b2fe34e05856" containerName="istio-proxy" containerID="cri-o://7289e0dbcc47aed67042b3c7371320cb04f092e262f8cfccd31cf319ea0b3db3" gracePeriod=30 Sep 14 13:20:03.240970 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:03.240897 2564 generic.go:356] "Generic (PLEG): container finished" podID="e444894f-628c-4794-8a59-886c3ccb0674" containerID="3f5115815e08ffff87641c2bfcb44bcc0210afcb70addd06c04785f3560773a8" exitCode=0 Sep 14 13:20:03.241148 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:03.240973 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" event={"ID":"e444894f-628c-4794-8a59-886c3ccb0674","Type":"ContainerDied","Data":"3f5115815e08ffff87641c2bfcb44bcc0210afcb70addd06c04785f3560773a8"} Sep 14 13:20:03.241617 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:03.241598 2564 scope.go:117] "RemoveContainer" containerID="3f5115815e08ffff87641c2bfcb44bcc0210afcb70addd06c04785f3560773a8" Sep 14 13:20:03.242891 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:03.242863 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:20:03.243003 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:03.242945 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:20:03.243003 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:03.242981 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:20:04.249364 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:04.249332 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" event={"ID":"e444894f-628c-4794-8a59-886c3ccb0674","Type":"ContainerStarted","Data":"522f61cf898600564adea088a800d0b2ef794d41916f4f858d0caff5b9c17691"} Sep 14 13:20:05.199494 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:05.199455 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:20:05.199720 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:05.199698 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Sep 14 13:20:05.199767 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:05.199746 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" podUID="e444894f-628c-4794-8a59-886c3ccb0674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Sep 14 13:20:06.200175 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:06.200136 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Sep 14 13:20:06.200582 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:06.200210 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" podUID="e444894f-628c-4794-8a59-886c3ccb0674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Sep 14 13:20:06.678410 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:06.678357 2564 patch_prober.go:28] interesting pod/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 13:20:06.678729 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:06.678683 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" podUID="e39e4e21-c72a-4608-990a-b2fe34e05856" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 13:20:07.200270 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:07.200228 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Sep 14 13:20:07.200678 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:07.200289 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" podUID="e444894f-628c-4794-8a59-886c3ccb0674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Sep 14 13:20:08.200396 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.200368 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Sep 14 13:20:08.200717 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.200434 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" podUID="e444894f-628c-4794-8a59-886c3ccb0674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Sep 14 13:20:08.216864 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.216839 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:20:08.291561 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.291523 2564 generic.go:356] "Generic (PLEG): container finished" podID="e39e4e21-c72a-4608-990a-b2fe34e05856" containerID="7289e0dbcc47aed67042b3c7371320cb04f092e262f8cfccd31cf319ea0b3db3" exitCode=0 Sep 14 13:20:08.291760 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.291585 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" Sep 14 13:20:08.291760 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.291604 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" event={"ID":"e39e4e21-c72a-4608-990a-b2fe34e05856","Type":"ContainerDied","Data":"7289e0dbcc47aed67042b3c7371320cb04f092e262f8cfccd31cf319ea0b3db3"} Sep 14 13:20:08.291760 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.291666 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs" event={"ID":"e39e4e21-c72a-4608-990a-b2fe34e05856","Type":"ContainerDied","Data":"c435e78a7421013b4bdbe69ca39b2b702dba5eaab1a724829c50ed95fe3fb164"} Sep 14 13:20:08.291760 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.291699 2564 scope.go:117] "RemoveContainer" containerID="7289e0dbcc47aed67042b3c7371320cb04f092e262f8cfccd31cf319ea0b3db3" Sep 14 13:20:08.302109 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.302086 2564 scope.go:117] "RemoveContainer" containerID="7289e0dbcc47aed67042b3c7371320cb04f092e262f8cfccd31cf319ea0b3db3" Sep 14 13:20:08.302490 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:20:08.302469 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7289e0dbcc47aed67042b3c7371320cb04f092e262f8cfccd31cf319ea0b3db3\": container with ID starting with 7289e0dbcc47aed67042b3c7371320cb04f092e262f8cfccd31cf319ea0b3db3 not found: ID does not exist" containerID="7289e0dbcc47aed67042b3c7371320cb04f092e262f8cfccd31cf319ea0b3db3" Sep 14 13:20:08.302597 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.302495 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7289e0dbcc47aed67042b3c7371320cb04f092e262f8cfccd31cf319ea0b3db3"} err="failed to get container status \"7289e0dbcc47aed67042b3c7371320cb04f092e262f8cfccd31cf319ea0b3db3\": rpc error: code = NotFound desc = could not find container \"7289e0dbcc47aed67042b3c7371320cb04f092e262f8cfccd31cf319ea0b3db3\": container with ID starting with 7289e0dbcc47aed67042b3c7371320cb04f092e262f8cfccd31cf319ea0b3db3 not found: ID does not exist" Sep 14 13:20:08.332243 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.332212 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-data\") pod \"e39e4e21-c72a-4608-990a-b2fe34e05856\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " Sep 14 13:20:08.332243 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.332252 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-credential-socket\") pod \"e39e4e21-c72a-4608-990a-b2fe34e05856\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " Sep 14 13:20:08.332515 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.332284 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39e4e21-c72a-4608-990a-b2fe34e05856-istiod-ca-cert\") pod \"e39e4e21-c72a-4608-990a-b2fe34e05856\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " Sep 14 13:20:08.332515 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.332349 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-workload-certs\") pod \"e39e4e21-c72a-4608-990a-b2fe34e05856\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " Sep 14 13:20:08.332515 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.332379 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-token\") pod \"e39e4e21-c72a-4608-990a-b2fe34e05856\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " Sep 14 13:20:08.332674 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.332506 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-envoy\") pod \"e39e4e21-c72a-4608-990a-b2fe34e05856\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " Sep 14 13:20:08.332674 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.332587 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-podinfo\") pod \"e39e4e21-c72a-4608-990a-b2fe34e05856\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " Sep 14 13:20:08.332674 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.332583 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e39e4e21-c72a-4608-990a-b2fe34e05856" (UID: "e39e4e21-c72a-4608-990a-b2fe34e05856"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:20:08.332674 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.332592 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-data" (OuterVolumeSpecName: "istio-data") pod "e39e4e21-c72a-4608-990a-b2fe34e05856" (UID: "e39e4e21-c72a-4608-990a-b2fe34e05856"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:20:08.332894 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.332691 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-workload-socket\") pod \"e39e4e21-c72a-4608-990a-b2fe34e05856\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " Sep 14 13:20:08.332894 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.332676 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e39e4e21-c72a-4608-990a-b2fe34e05856" (UID: "e39e4e21-c72a-4608-990a-b2fe34e05856"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:20:08.332894 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.332728 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dq7t8\" (UniqueName: \"kubernetes.io/projected/e39e4e21-c72a-4608-990a-b2fe34e05856-kube-api-access-dq7t8\") pod \"e39e4e21-c72a-4608-990a-b2fe34e05856\" (UID: \"e39e4e21-c72a-4608-990a-b2fe34e05856\") " Sep 14 13:20:08.332894 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.332761 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39e4e21-c72a-4608-990a-b2fe34e05856-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e39e4e21-c72a-4608-990a-b2fe34e05856" (UID: "e39e4e21-c72a-4608-990a-b2fe34e05856"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:20:08.333314 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.332999 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e39e4e21-c72a-4608-990a-b2fe34e05856" (UID: "e39e4e21-c72a-4608-990a-b2fe34e05856"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:20:08.333314 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.333238 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:20:08.333314 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.333260 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:20:08.333511 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.333349 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:20:08.333511 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.333452 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39e4e21-c72a-4608-990a-b2fe34e05856-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:20:08.333511 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.333466 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:20:08.334830 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.334799 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e39e4e21-c72a-4608-990a-b2fe34e05856" (UID: "e39e4e21-c72a-4608-990a-b2fe34e05856"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:20:08.334982 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.334846 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-token" (OuterVolumeSpecName: "istio-token") pod "e39e4e21-c72a-4608-990a-b2fe34e05856" (UID: "e39e4e21-c72a-4608-990a-b2fe34e05856"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:20:08.334982 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.334887 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e39e4e21-c72a-4608-990a-b2fe34e05856" (UID: "e39e4e21-c72a-4608-990a-b2fe34e05856"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 13:20:08.335084 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.334974 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39e4e21-c72a-4608-990a-b2fe34e05856-kube-api-access-dq7t8" (OuterVolumeSpecName: "kube-api-access-dq7t8") pod "e39e4e21-c72a-4608-990a-b2fe34e05856" (UID: "e39e4e21-c72a-4608-990a-b2fe34e05856"). InnerVolumeSpecName "kube-api-access-dq7t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:20:08.434452 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.434412 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq7t8\" (UniqueName: \"kubernetes.io/projected/e39e4e21-c72a-4608-990a-b2fe34e05856-kube-api-access-dq7t8\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:20:08.434452 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.434445 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:20:08.434452 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.434458 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:20:08.434691 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.434469 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39e4e21-c72a-4608-990a-b2fe34e05856-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:20:08.616941 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.616883 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs"] Sep 14 13:20:08.625952 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:08.621774 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-4c0b93-openshift-default-574684cd66-nh8gs"] Sep 14 13:20:09.199720 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:09.199679 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Sep 14 13:20:09.199908 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:09.199750 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" podUID="e444894f-628c-4794-8a59-886c3ccb0674" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Sep 14 13:20:09.813170 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:09.813133 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e39e4e21-c72a-4608-990a-b2fe34e05856" path="/var/lib/kubelet/pods/e39e4e21-c72a-4608-990a-b2fe34e05856/volumes" Sep 14 13:20:10.203722 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:10.203694 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:20:10.204059 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:10.204030 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:20:10.204716 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:20:10.204698 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:23:19.932482 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:23:19.932445 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/1.log" Sep 14 13:23:19.937098 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:23:19.937077 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/1.log" Sep 14 13:26:28.120426 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:28.120391 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb"] Sep 14 13:26:28.122147 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:28.120590 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" podUID="b1131fb1-a049-418b-be10-9c7c052f7c01" containerName="istio-proxy" containerID="cri-o://36512dc773c793ec13df2137e7994331cbd10e1704006525450433d54df6613a" gracePeriod=30 Sep 14 13:26:33.365584 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.365559 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:26:33.417296 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.417207 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1131fb1-a049-418b-be10-9c7c052f7c01-istiod-ca-cert\") pod \"b1131fb1-a049-418b-be10-9c7c052f7c01\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " Sep 14 13:26:33.417296 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.417273 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-podinfo\") pod \"b1131fb1-a049-418b-be10-9c7c052f7c01\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " Sep 14 13:26:33.417516 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.417350 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-data\") pod \"b1131fb1-a049-418b-be10-9c7c052f7c01\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " Sep 14 13:26:33.417516 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.417373 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-workload-certs\") pod \"b1131fb1-a049-418b-be10-9c7c052f7c01\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " Sep 14 13:26:33.417516 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.417407 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-credential-socket\") pod \"b1131fb1-a049-418b-be10-9c7c052f7c01\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " Sep 14 13:26:33.417516 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.417504 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-envoy\") pod \"b1131fb1-a049-418b-be10-9c7c052f7c01\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " Sep 14 13:26:33.417726 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.417571 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-workload-socket\") pod \"b1131fb1-a049-418b-be10-9c7c052f7c01\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " Sep 14 13:26:33.417726 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.417610 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-token\") pod \"b1131fb1-a049-418b-be10-9c7c052f7c01\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " Sep 14 13:26:33.417726 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.417627 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-data" (OuterVolumeSpecName: "istio-data") pod "b1131fb1-a049-418b-be10-9c7c052f7c01" (UID: "b1131fb1-a049-418b-be10-9c7c052f7c01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:26:33.417726 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.417643 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfcmp\" (UniqueName: \"kubernetes.io/projected/b1131fb1-a049-418b-be10-9c7c052f7c01-kube-api-access-xfcmp\") pod \"b1131fb1-a049-418b-be10-9c7c052f7c01\" (UID: \"b1131fb1-a049-418b-be10-9c7c052f7c01\") " Sep 14 13:26:33.417726 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.417685 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b1131fb1-a049-418b-be10-9c7c052f7c01" (UID: "b1131fb1-a049-418b-be10-9c7c052f7c01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:26:33.417726 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.417707 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1131fb1-a049-418b-be10-9c7c052f7c01-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b1131fb1-a049-418b-be10-9c7c052f7c01" (UID: "b1131fb1-a049-418b-be10-9c7c052f7c01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:26:33.418068 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.418050 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:33.418132 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.418072 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:33.418132 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.418085 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1131fb1-a049-418b-be10-9c7c052f7c01-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:33.418383 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.418353 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b1131fb1-a049-418b-be10-9c7c052f7c01" (UID: "b1131fb1-a049-418b-be10-9c7c052f7c01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:26:33.418453 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.418425 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b1131fb1-a049-418b-be10-9c7c052f7c01" (UID: "b1131fb1-a049-418b-be10-9c7c052f7c01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:26:33.419757 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.419736 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b1131fb1-a049-418b-be10-9c7c052f7c01" (UID: "b1131fb1-a049-418b-be10-9c7c052f7c01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 13:26:33.420357 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.420322 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1131fb1-a049-418b-be10-9c7c052f7c01-kube-api-access-xfcmp" (OuterVolumeSpecName: "kube-api-access-xfcmp") pod "b1131fb1-a049-418b-be10-9c7c052f7c01" (UID: "b1131fb1-a049-418b-be10-9c7c052f7c01"). InnerVolumeSpecName "kube-api-access-xfcmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:26:33.420471 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.420391 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b1131fb1-a049-418b-be10-9c7c052f7c01" (UID: "b1131fb1-a049-418b-be10-9c7c052f7c01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:26:33.420471 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.420411 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-token" (OuterVolumeSpecName: "istio-token") pod "b1131fb1-a049-418b-be10-9c7c052f7c01" (UID: "b1131fb1-a049-418b-be10-9c7c052f7c01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:26:33.518674 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.518645 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:33.518674 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.518671 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:33.518674 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.518683 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1131fb1-a049-418b-be10-9c7c052f7c01-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:33.518945 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.518691 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:33.518945 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.518700 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfcmp\" (UniqueName: \"kubernetes.io/projected/b1131fb1-a049-418b-be10-9c7c052f7c01-kube-api-access-xfcmp\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:33.518945 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.518708 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1131fb1-a049-418b-be10-9c7c052f7c01-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:33.662478 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.662446 2564 generic.go:356] "Generic (PLEG): container finished" podID="b1131fb1-a049-418b-be10-9c7c052f7c01" containerID="36512dc773c793ec13df2137e7994331cbd10e1704006525450433d54df6613a" exitCode=0 Sep 14 13:26:33.662666 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.662493 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" Sep 14 13:26:33.662666 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.662533 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" event={"ID":"b1131fb1-a049-418b-be10-9c7c052f7c01","Type":"ContainerDied","Data":"36512dc773c793ec13df2137e7994331cbd10e1704006525450433d54df6613a"} Sep 14 13:26:33.662666 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.662593 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb" event={"ID":"b1131fb1-a049-418b-be10-9c7c052f7c01","Type":"ContainerDied","Data":"d31cce034755669f53eb1be5a093d81349c5c8b919558b30cd17997eaf0012e9"} Sep 14 13:26:33.662666 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.662624 2564 scope.go:117] "RemoveContainer" containerID="36512dc773c793ec13df2137e7994331cbd10e1704006525450433d54df6613a" Sep 14 13:26:33.672881 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.672862 2564 scope.go:117] "RemoveContainer" containerID="c133aa6b3e1e3c33192c3acbe35f8e67573c775ce4c84dd3b5ef86f0c8781f67" Sep 14 13:26:33.681846 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.681830 2564 scope.go:117] "RemoveContainer" containerID="36512dc773c793ec13df2137e7994331cbd10e1704006525450433d54df6613a" Sep 14 13:26:33.682140 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:26:33.682122 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36512dc773c793ec13df2137e7994331cbd10e1704006525450433d54df6613a\": container with ID starting with 36512dc773c793ec13df2137e7994331cbd10e1704006525450433d54df6613a not found: ID does not exist" containerID="36512dc773c793ec13df2137e7994331cbd10e1704006525450433d54df6613a" Sep 14 13:26:33.682201 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.682149 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36512dc773c793ec13df2137e7994331cbd10e1704006525450433d54df6613a"} err="failed to get container status \"36512dc773c793ec13df2137e7994331cbd10e1704006525450433d54df6613a\": rpc error: code = NotFound desc = could not find container \"36512dc773c793ec13df2137e7994331cbd10e1704006525450433d54df6613a\": container with ID starting with 36512dc773c793ec13df2137e7994331cbd10e1704006525450433d54df6613a not found: ID does not exist" Sep 14 13:26:33.682201 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.682177 2564 scope.go:117] "RemoveContainer" containerID="c133aa6b3e1e3c33192c3acbe35f8e67573c775ce4c84dd3b5ef86f0c8781f67" Sep 14 13:26:33.682434 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:26:33.682409 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c133aa6b3e1e3c33192c3acbe35f8e67573c775ce4c84dd3b5ef86f0c8781f67\": container with ID starting with c133aa6b3e1e3c33192c3acbe35f8e67573c775ce4c84dd3b5ef86f0c8781f67 not found: ID does not exist" containerID="c133aa6b3e1e3c33192c3acbe35f8e67573c775ce4c84dd3b5ef86f0c8781f67" Sep 14 13:26:33.682497 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.682432 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c133aa6b3e1e3c33192c3acbe35f8e67573c775ce4c84dd3b5ef86f0c8781f67"} err="failed to get container status \"c133aa6b3e1e3c33192c3acbe35f8e67573c775ce4c84dd3b5ef86f0c8781f67\": rpc error: code = NotFound desc = could not find container \"c133aa6b3e1e3c33192c3acbe35f8e67573c775ce4c84dd3b5ef86f0c8781f67\": container with ID starting with c133aa6b3e1e3c33192c3acbe35f8e67573c775ce4c84dd3b5ef86f0c8781f67 not found: ID does not exist" Sep 14 13:26:33.686862 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.686838 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb"] Sep 14 13:26:33.689992 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.689972 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-336ccc18-openshift-default-b45ffcd9b-zgwmb"] Sep 14 13:26:33.818750 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:33.814195 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1131fb1-a049-418b-be10-9c7c052f7c01" path="/var/lib/kubelet/pods/b1131fb1-a049-418b-be10-9c7c052f7c01/volumes" Sep 14 13:26:34.311866 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.311837 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln"] Sep 14 13:26:34.312108 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.312078 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" podUID="e2da29be-f347-409b-9f93-f723ceb9deba" containerName="maas-api" containerID="cri-o://e6a714e7a86398317a832b91534d8032ec40da2024e30e72fb9ba21218e2d470" gracePeriod=30 Sep 14 13:26:34.564449 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.564383 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" Sep 14 13:26:34.631512 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.631481 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76fjq\" (UniqueName: \"kubernetes.io/projected/e2da29be-f347-409b-9f93-f723ceb9deba-kube-api-access-76fjq\") pod \"e2da29be-f347-409b-9f93-f723ceb9deba\" (UID: \"e2da29be-f347-409b-9f93-f723ceb9deba\") " Sep 14 13:26:34.631673 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.631526 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2da29be-f347-409b-9f93-f723ceb9deba-maas-api-tls\") pod \"e2da29be-f347-409b-9f93-f723ceb9deba\" (UID: \"e2da29be-f347-409b-9f93-f723ceb9deba\") " Sep 14 13:26:34.631673 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.631574 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2da29be-f347-409b-9f93-f723ceb9deba-openshift-service-ca\") pod \"e2da29be-f347-409b-9f93-f723ceb9deba\" (UID: \"e2da29be-f347-409b-9f93-f723ceb9deba\") " Sep 14 13:26:34.631673 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.631625 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2da29be-f347-409b-9f93-f723ceb9deba-metrics-certs\") pod \"e2da29be-f347-409b-9f93-f723ceb9deba\" (UID: \"e2da29be-f347-409b-9f93-f723ceb9deba\") " Sep 14 13:26:34.632010 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.631984 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2da29be-f347-409b-9f93-f723ceb9deba-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e2da29be-f347-409b-9f93-f723ceb9deba" (UID: "e2da29be-f347-409b-9f93-f723ceb9deba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:26:34.633889 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.633826 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2da29be-f347-409b-9f93-f723ceb9deba-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e2da29be-f347-409b-9f93-f723ceb9deba" (UID: "e2da29be-f347-409b-9f93-f723ceb9deba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:26:34.633889 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.633844 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2da29be-f347-409b-9f93-f723ceb9deba-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e2da29be-f347-409b-9f93-f723ceb9deba" (UID: "e2da29be-f347-409b-9f93-f723ceb9deba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:26:34.634068 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.633887 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2da29be-f347-409b-9f93-f723ceb9deba-kube-api-access-76fjq" (OuterVolumeSpecName: "kube-api-access-76fjq") pod "e2da29be-f347-409b-9f93-f723ceb9deba" (UID: "e2da29be-f347-409b-9f93-f723ceb9deba"). InnerVolumeSpecName "kube-api-access-76fjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:26:34.670953 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.670895 2564 generic.go:356] "Generic (PLEG): container finished" podID="e2da29be-f347-409b-9f93-f723ceb9deba" containerID="e6a714e7a86398317a832b91534d8032ec40da2024e30e72fb9ba21218e2d470" exitCode=0 Sep 14 13:26:34.671084 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.670968 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" Sep 14 13:26:34.671084 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.671019 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" event={"ID":"e2da29be-f347-409b-9f93-f723ceb9deba","Type":"ContainerDied","Data":"e6a714e7a86398317a832b91534d8032ec40da2024e30e72fb9ba21218e2d470"} Sep 14 13:26:34.671084 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.671070 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln" event={"ID":"e2da29be-f347-409b-9f93-f723ceb9deba","Type":"ContainerDied","Data":"1561fdf69181a1cc898032ec9d22f97feceb5be91458b32e41990908fb7b4c40"} Sep 14 13:26:34.671228 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.671098 2564 scope.go:117] "RemoveContainer" containerID="e6a714e7a86398317a832b91534d8032ec40da2024e30e72fb9ba21218e2d470" Sep 14 13:26:34.682050 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.682016 2564 scope.go:117] "RemoveContainer" containerID="e6a714e7a86398317a832b91534d8032ec40da2024e30e72fb9ba21218e2d470" Sep 14 13:26:34.682378 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:26:34.682356 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6a714e7a86398317a832b91534d8032ec40da2024e30e72fb9ba21218e2d470\": container with ID starting with e6a714e7a86398317a832b91534d8032ec40da2024e30e72fb9ba21218e2d470 not found: ID does not exist" containerID="e6a714e7a86398317a832b91534d8032ec40da2024e30e72fb9ba21218e2d470" Sep 14 13:26:34.682464 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.682386 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6a714e7a86398317a832b91534d8032ec40da2024e30e72fb9ba21218e2d470"} err="failed to get container status \"e6a714e7a86398317a832b91534d8032ec40da2024e30e72fb9ba21218e2d470\": rpc error: code = NotFound desc = could not find container \"e6a714e7a86398317a832b91534d8032ec40da2024e30e72fb9ba21218e2d470\": container with ID starting with e6a714e7a86398317a832b91534d8032ec40da2024e30e72fb9ba21218e2d470 not found: ID does not exist" Sep 14 13:26:34.702914 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.702882 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln"] Sep 14 13:26:34.708532 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.708497 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-bc851d-6657b9c4d8-p9mln"] Sep 14 13:26:34.734517 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.734466 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76fjq\" (UniqueName: \"kubernetes.io/projected/e2da29be-f347-409b-9f93-f723ceb9deba-kube-api-access-76fjq\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:34.734517 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.734512 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2da29be-f347-409b-9f93-f723ceb9deba-maas-api-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:34.734790 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.734529 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2da29be-f347-409b-9f93-f723ceb9deba-openshift-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:34.734790 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:34.734545 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2da29be-f347-409b-9f93-f723ceb9deba-metrics-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:35.812658 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:35.812627 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2da29be-f347-409b-9f93-f723ceb9deba" path="/var/lib/kubelet/pods/e2da29be-f347-409b-9f93-f723ceb9deba/volumes" Sep 14 13:26:37.152901 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.152861 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7422-62518b"] Sep 14 13:26:37.153555 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.153535 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2da29be-f347-409b-9f93-f723ceb9deba" containerName="maas-api" Sep 14 13:26:37.153655 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.153558 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2da29be-f347-409b-9f93-f723ceb9deba" containerName="maas-api" Sep 14 13:26:37.153655 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.153582 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1131fb1-a049-418b-be10-9c7c052f7c01" containerName="istio-proxy" Sep 14 13:26:37.153655 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.153590 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1131fb1-a049-418b-be10-9c7c052f7c01" containerName="istio-proxy" Sep 14 13:26:37.153655 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.153607 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e39e4e21-c72a-4608-990a-b2fe34e05856" containerName="istio-proxy" Sep 14 13:26:37.153655 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.153615 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e39e4e21-c72a-4608-990a-b2fe34e05856" containerName="istio-proxy" Sep 14 13:26:37.153655 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.153631 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1131fb1-a049-418b-be10-9c7c052f7c01" containerName="istio-proxy" Sep 14 13:26:37.153655 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.153641 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1131fb1-a049-418b-be10-9c7c052f7c01" containerName="istio-proxy" Sep 14 13:26:37.154041 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.153740 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e39e4e21-c72a-4608-990a-b2fe34e05856" containerName="istio-proxy" Sep 14 13:26:37.154041 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.153755 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1131fb1-a049-418b-be10-9c7c052f7c01" containerName="istio-proxy" Sep 14 13:26:37.154041 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.153764 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1131fb1-a049-418b-be10-9c7c052f7c01" containerName="istio-proxy" Sep 14 13:26:37.154041 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.153778 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2da29be-f347-409b-9f93-f723ceb9deba" containerName="maas-api" Sep 14 13:26:37.157259 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.157228 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-62518b" Sep 14 13:26:37.160502 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.160483 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vtxmg\"" Sep 14 13:26:37.168519 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.168499 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-62518b"] Sep 14 13:26:37.264003 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.263965 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gxdh\" (UniqueName: \"kubernetes.io/projected/35454c83-886a-4210-b181-66a7ebf30d85-kube-api-access-7gxdh\") pod \"test-curl-7422-62518b\" (UID: \"35454c83-886a-4210-b181-66a7ebf30d85\") " pod="opendatahub/test-curl-7422-62518b" Sep 14 13:26:37.364975 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.364939 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gxdh\" (UniqueName: \"kubernetes.io/projected/35454c83-886a-4210-b181-66a7ebf30d85-kube-api-access-7gxdh\") pod \"test-curl-7422-62518b\" (UID: \"35454c83-886a-4210-b181-66a7ebf30d85\") " pod="opendatahub/test-curl-7422-62518b" Sep 14 13:26:37.373335 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.373311 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gxdh\" (UniqueName: \"kubernetes.io/projected/35454c83-886a-4210-b181-66a7ebf30d85-kube-api-access-7gxdh\") pod \"test-curl-7422-62518b\" (UID: \"35454c83-886a-4210-b181-66a7ebf30d85\") " pod="opendatahub/test-curl-7422-62518b" Sep 14 13:26:37.467783 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.467695 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-62518b" Sep 14 13:26:37.744022 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.743997 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7422-62518b"] Sep 14 13:26:37.744958 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:26:37.744897 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35454c83_886a_4210_b181_66a7ebf30d85.slice/crio-54f747018ee3c2aa09b94016c2306df0c8013e94c6b0b8cdf3f42ef8ad83d985 WatchSource:0}: Error finding container 54f747018ee3c2aa09b94016c2306df0c8013e94c6b0b8cdf3f42ef8ad83d985: Status 404 returned error can't find the container with id 54f747018ee3c2aa09b94016c2306df0c8013e94c6b0b8cdf3f42ef8ad83d985 Sep 14 13:26:37.746847 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:37.746829 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 13:26:38.699969 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:38.699933 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-62518b" event={"ID":"35454c83-886a-4210-b181-66a7ebf30d85","Type":"ContainerStarted","Data":"cb8fb3235cf1da5c369c42e1c3e2e7cdb751c6e2d166bb897e72451799ecfbe4"} Sep 14 13:26:38.700363 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:38.699979 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7422-62518b" event={"ID":"35454c83-886a-4210-b181-66a7ebf30d85","Type":"ContainerStarted","Data":"54f747018ee3c2aa09b94016c2306df0c8013e94c6b0b8cdf3f42ef8ad83d985"} Sep 14 13:26:38.715768 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:38.715723 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7422-62518b" podStartSLOduration=0.979881865 podStartE2EDuration="1.715709479s" totalImagesPullingTime="735.827614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:26:38.715023554 +0000 UTC m=+1699.377919184" watchObservedRunningTime="2026-09-14 13:26:38.715709479 +0000 UTC m=+1699.378605108" Sep 14 13:26:39.307884 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:39.307852 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7422-62518b"] Sep 14 13:26:39.311520 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:39.311489 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7422-62518b"] Sep 14 13:26:39.706012 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:26:39.705981 2564 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7422-62518b" container="test-curl-7422-62518b" Sep 14 13:26:40.712683 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:40.712645 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7422-62518b" podUID="35454c83-886a-4210-b181-66a7ebf30d85" containerName="test-curl-7422-62518b" containerID="cri-o://cb8fb3235cf1da5c369c42e1c3e2e7cdb751c6e2d166bb897e72451799ecfbe4" gracePeriod=2 Sep 14 13:26:42.893999 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:42.893976 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-62518b" Sep 14 13:26:43.026281 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:43.026198 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gxdh\" (UniqueName: \"kubernetes.io/projected/35454c83-886a-4210-b181-66a7ebf30d85-kube-api-access-7gxdh\") pod \"35454c83-886a-4210-b181-66a7ebf30d85\" (UID: \"35454c83-886a-4210-b181-66a7ebf30d85\") " Sep 14 13:26:43.028547 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:43.028514 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35454c83-886a-4210-b181-66a7ebf30d85-kube-api-access-7gxdh" (OuterVolumeSpecName: "kube-api-access-7gxdh") pod "35454c83-886a-4210-b181-66a7ebf30d85" (UID: "35454c83-886a-4210-b181-66a7ebf30d85"). InnerVolumeSpecName "kube-api-access-7gxdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:26:43.127413 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:43.127382 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gxdh\" (UniqueName: \"kubernetes.io/projected/35454c83-886a-4210-b181-66a7ebf30d85-kube-api-access-7gxdh\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:43.521574 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:43.521542 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml"] Sep 14 13:26:43.521808 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:43.521782 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" podUID="6ad76071-ccde-4279-a47a-3753c2a4b0e9" containerName="istio-proxy" containerID="cri-o://c7ffafb0444e93a66177dc12132b903e864070e8932d18f1828512e433d41f25" gracePeriod=30 Sep 14 13:26:43.732773 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:43.732722 2564 generic.go:356] "Generic (PLEG): container finished" podID="35454c83-886a-4210-b181-66a7ebf30d85" containerID="cb8fb3235cf1da5c369c42e1c3e2e7cdb751c6e2d166bb897e72451799ecfbe4" exitCode=137 Sep 14 13:26:43.732773 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:43.732769 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7422-62518b" Sep 14 13:26:43.733064 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:43.732829 2564 scope.go:117] "RemoveContainer" containerID="cb8fb3235cf1da5c369c42e1c3e2e7cdb751c6e2d166bb897e72451799ecfbe4" Sep 14 13:26:43.745136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:43.743240 2564 scope.go:117] "RemoveContainer" containerID="cb8fb3235cf1da5c369c42e1c3e2e7cdb751c6e2d166bb897e72451799ecfbe4" Sep 14 13:26:43.745136 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:26:43.743714 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7422-62518b\" is forbidden: User \"system:node:ip-10-0-136-96.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-96.ec2.internal' and this object" podUID="35454c83-886a-4210-b181-66a7ebf30d85" pod="opendatahub/test-curl-7422-62518b" Sep 14 13:26:43.745588 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:26:43.745567 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8fb3235cf1da5c369c42e1c3e2e7cdb751c6e2d166bb897e72451799ecfbe4\": container with ID starting with cb8fb3235cf1da5c369c42e1c3e2e7cdb751c6e2d166bb897e72451799ecfbe4 not found: ID does not exist" containerID="cb8fb3235cf1da5c369c42e1c3e2e7cdb751c6e2d166bb897e72451799ecfbe4" Sep 14 13:26:43.745646 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:43.745597 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8fb3235cf1da5c369c42e1c3e2e7cdb751c6e2d166bb897e72451799ecfbe4"} err="failed to get container status \"cb8fb3235cf1da5c369c42e1c3e2e7cdb751c6e2d166bb897e72451799ecfbe4\": rpc error: code = NotFound desc = could not find container \"cb8fb3235cf1da5c369c42e1c3e2e7cdb751c6e2d166bb897e72451799ecfbe4\": container with ID starting with cb8fb3235cf1da5c369c42e1c3e2e7cdb751c6e2d166bb897e72451799ecfbe4 not found: ID does not exist" Sep 14 13:26:43.812478 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:43.812398 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35454c83-886a-4210-b181-66a7ebf30d85" path="/var/lib/kubelet/pods/35454c83-886a-4210-b181-66a7ebf30d85/volumes" Sep 14 13:26:44.029394 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.029361 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r"] Sep 14 13:26:44.029837 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.029604 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" podUID="45f9dea1-c3e6-44b4-85fa-45009d40b328" containerName="maas-api" containerID="cri-o://f6e77b622b7a8985371463138e607c509dde3d8fed7d5b1d20539ae9de841df6" gracePeriod=30 Sep 14 13:26:44.282303 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.282274 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" Sep 14 13:26:44.442680 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.442602 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45f9dea1-c3e6-44b4-85fa-45009d40b328-openshift-service-ca\") pod \"45f9dea1-c3e6-44b4-85fa-45009d40b328\" (UID: \"45f9dea1-c3e6-44b4-85fa-45009d40b328\") " Sep 14 13:26:44.442680 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.442641 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45f9dea1-c3e6-44b4-85fa-45009d40b328-metrics-certs\") pod \"45f9dea1-c3e6-44b4-85fa-45009d40b328\" (UID: \"45f9dea1-c3e6-44b4-85fa-45009d40b328\") " Sep 14 13:26:44.442850 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.442760 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8s8j\" (UniqueName: \"kubernetes.io/projected/45f9dea1-c3e6-44b4-85fa-45009d40b328-kube-api-access-f8s8j\") pod \"45f9dea1-c3e6-44b4-85fa-45009d40b328\" (UID: \"45f9dea1-c3e6-44b4-85fa-45009d40b328\") " Sep 14 13:26:44.442850 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.442814 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45f9dea1-c3e6-44b4-85fa-45009d40b328-maas-api-tls\") pod \"45f9dea1-c3e6-44b4-85fa-45009d40b328\" (UID: \"45f9dea1-c3e6-44b4-85fa-45009d40b328\") " Sep 14 13:26:44.443023 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.442998 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f9dea1-c3e6-44b4-85fa-45009d40b328-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "45f9dea1-c3e6-44b4-85fa-45009d40b328" (UID: "45f9dea1-c3e6-44b4-85fa-45009d40b328"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:26:44.443285 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.443262 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45f9dea1-c3e6-44b4-85fa-45009d40b328-openshift-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:44.445053 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.445028 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45f9dea1-c3e6-44b4-85fa-45009d40b328-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "45f9dea1-c3e6-44b4-85fa-45009d40b328" (UID: "45f9dea1-c3e6-44b4-85fa-45009d40b328"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:26:44.445154 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.445071 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45f9dea1-c3e6-44b4-85fa-45009d40b328-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "45f9dea1-c3e6-44b4-85fa-45009d40b328" (UID: "45f9dea1-c3e6-44b4-85fa-45009d40b328"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:26:44.445154 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.445088 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f9dea1-c3e6-44b4-85fa-45009d40b328-kube-api-access-f8s8j" (OuterVolumeSpecName: "kube-api-access-f8s8j") pod "45f9dea1-c3e6-44b4-85fa-45009d40b328" (UID: "45f9dea1-c3e6-44b4-85fa-45009d40b328"). InnerVolumeSpecName "kube-api-access-f8s8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:26:44.544307 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.544264 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45f9dea1-c3e6-44b4-85fa-45009d40b328-metrics-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:44.544307 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.544304 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8s8j\" (UniqueName: \"kubernetes.io/projected/45f9dea1-c3e6-44b4-85fa-45009d40b328-kube-api-access-f8s8j\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:44.544307 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.544315 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45f9dea1-c3e6-44b4-85fa-45009d40b328-maas-api-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:44.739990 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.739881 2564 generic.go:356] "Generic (PLEG): container finished" podID="45f9dea1-c3e6-44b4-85fa-45009d40b328" containerID="f6e77b622b7a8985371463138e607c509dde3d8fed7d5b1d20539ae9de841df6" exitCode=0 Sep 14 13:26:44.739990 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.739943 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" Sep 14 13:26:44.739990 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.739972 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" event={"ID":"45f9dea1-c3e6-44b4-85fa-45009d40b328","Type":"ContainerDied","Data":"f6e77b622b7a8985371463138e607c509dde3d8fed7d5b1d20539ae9de841df6"} Sep 14 13:26:44.740264 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.740031 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r" event={"ID":"45f9dea1-c3e6-44b4-85fa-45009d40b328","Type":"ContainerDied","Data":"4812074965af9a56e8db91154bdb2b35041b36ccd5b097b7639018123ada6f2a"} Sep 14 13:26:44.740264 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.740055 2564 scope.go:117] "RemoveContainer" containerID="f6e77b622b7a8985371463138e607c509dde3d8fed7d5b1d20539ae9de841df6" Sep 14 13:26:44.750155 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.750133 2564 scope.go:117] "RemoveContainer" containerID="f6e77b622b7a8985371463138e607c509dde3d8fed7d5b1d20539ae9de841df6" Sep 14 13:26:44.750482 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:26:44.750458 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e77b622b7a8985371463138e607c509dde3d8fed7d5b1d20539ae9de841df6\": container with ID starting with f6e77b622b7a8985371463138e607c509dde3d8fed7d5b1d20539ae9de841df6 not found: ID does not exist" containerID="f6e77b622b7a8985371463138e607c509dde3d8fed7d5b1d20539ae9de841df6" Sep 14 13:26:44.750580 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.750489 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e77b622b7a8985371463138e607c509dde3d8fed7d5b1d20539ae9de841df6"} err="failed to get container status \"f6e77b622b7a8985371463138e607c509dde3d8fed7d5b1d20539ae9de841df6\": rpc error: code = NotFound desc = could not find container \"f6e77b622b7a8985371463138e607c509dde3d8fed7d5b1d20539ae9de841df6\": container with ID starting with f6e77b622b7a8985371463138e607c509dde3d8fed7d5b1d20539ae9de841df6 not found: ID does not exist" Sep 14 13:26:44.763419 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.763396 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r"] Sep 14 13:26:44.766656 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:44.766636 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-b3d6c5-644446d7b8-rlj5r"] Sep 14 13:26:45.743149 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:45.743108 2564 patch_prober.go:28] interesting pod/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 13:26:45.743574 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:45.743179 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" podUID="6ad76071-ccde-4279-a47a-3753c2a4b0e9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 13:26:45.811781 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:45.811746 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45f9dea1-c3e6-44b4-85fa-45009d40b328" path="/var/lib/kubelet/pods/45f9dea1-c3e6-44b4-85fa-45009d40b328/volumes" Sep 14 13:26:48.769117 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.769085 2564 generic.go:356] "Generic (PLEG): container finished" podID="6ad76071-ccde-4279-a47a-3753c2a4b0e9" containerID="c7ffafb0444e93a66177dc12132b903e864070e8932d18f1828512e433d41f25" exitCode=0 Sep 14 13:26:48.769456 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.769149 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" event={"ID":"6ad76071-ccde-4279-a47a-3753c2a4b0e9","Type":"ContainerDied","Data":"c7ffafb0444e93a66177dc12132b903e864070e8932d18f1828512e433d41f25"} Sep 14 13:26:48.769456 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.769208 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" event={"ID":"6ad76071-ccde-4279-a47a-3753c2a4b0e9","Type":"ContainerDied","Data":"2d7cd9bcb66ab4b5940b9d6ff1968f3deae6ca7ea86f227aa1c421a59227f5cb"} Sep 14 13:26:48.769456 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.769224 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d7cd9bcb66ab4b5940b9d6ff1968f3deae6ca7ea86f227aa1c421a59227f5cb" Sep 14 13:26:48.769456 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.769237 2564 scope.go:117] "RemoveContainer" containerID="1eb004706eddd0311e56e93d76bd106d2b2db21b1f8a65e044f3dbaf42cf70cb" Sep 14 13:26:48.781841 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.781820 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:26:48.891221 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.891197 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-podinfo\") pod \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " Sep 14 13:26:48.891411 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.891240 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istiod-ca-cert\") pod \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " Sep 14 13:26:48.891411 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.891329 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-token\") pod \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " Sep 14 13:26:48.891411 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.891351 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-data\") pod \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " Sep 14 13:26:48.891411 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.891405 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-envoy\") pod \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " Sep 14 13:26:48.891635 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.891442 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-credential-socket\") pod \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " Sep 14 13:26:48.891635 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.891478 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-workload-socket\") pod \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " Sep 14 13:26:48.891635 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.891500 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-workload-certs\") pod \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " Sep 14 13:26:48.891635 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.891520 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5k8q2\" (UniqueName: \"kubernetes.io/projected/6ad76071-ccde-4279-a47a-3753c2a4b0e9-kube-api-access-5k8q2\") pod \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\" (UID: \"6ad76071-ccde-4279-a47a-3753c2a4b0e9\") " Sep 14 13:26:48.891854 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.891684 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-data" (OuterVolumeSpecName: "istio-data") pod "6ad76071-ccde-4279-a47a-3753c2a4b0e9" (UID: "6ad76071-ccde-4279-a47a-3753c2a4b0e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:26:48.891854 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.891740 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6ad76071-ccde-4279-a47a-3753c2a4b0e9" (UID: "6ad76071-ccde-4279-a47a-3753c2a4b0e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:26:48.891854 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.891771 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6ad76071-ccde-4279-a47a-3753c2a4b0e9" (UID: "6ad76071-ccde-4279-a47a-3753c2a4b0e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:26:48.891854 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.891816 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6ad76071-ccde-4279-a47a-3753c2a4b0e9" (UID: "6ad76071-ccde-4279-a47a-3753c2a4b0e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:26:48.892098 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.891969 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6ad76071-ccde-4279-a47a-3753c2a4b0e9" (UID: "6ad76071-ccde-4279-a47a-3753c2a4b0e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:26:48.892279 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.892253 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:48.892391 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.892285 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:48.892391 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.892301 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:48.892391 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.892314 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:48.892391 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.892327 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:48.893651 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.893630 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6ad76071-ccde-4279-a47a-3753c2a4b0e9" (UID: "6ad76071-ccde-4279-a47a-3753c2a4b0e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 13:26:48.893740 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.893712 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6ad76071-ccde-4279-a47a-3753c2a4b0e9" (UID: "6ad76071-ccde-4279-a47a-3753c2a4b0e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:26:48.894293 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.894265 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad76071-ccde-4279-a47a-3753c2a4b0e9-kube-api-access-5k8q2" (OuterVolumeSpecName: "kube-api-access-5k8q2") pod "6ad76071-ccde-4279-a47a-3753c2a4b0e9" (UID: "6ad76071-ccde-4279-a47a-3753c2a4b0e9"). InnerVolumeSpecName "kube-api-access-5k8q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:26:48.894380 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.894319 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-token" (OuterVolumeSpecName: "istio-token") pod "6ad76071-ccde-4279-a47a-3753c2a4b0e9" (UID: "6ad76071-ccde-4279-a47a-3753c2a4b0e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:26:48.992877 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.992846 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:48.992877 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.992873 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:48.992877 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.992882 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k8q2\" (UniqueName: \"kubernetes.io/projected/6ad76071-ccde-4279-a47a-3753c2a4b0e9-kube-api-access-5k8q2\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:48.993143 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:48.992891 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ad76071-ccde-4279-a47a-3753c2a4b0e9-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:26:49.775839 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:49.775810 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml" Sep 14 13:26:49.813754 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:49.813728 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml"] Sep 14 13:26:49.820456 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:49.820431 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-bc851d-openshift-default-86c8786974-47jml"] Sep 14 13:26:51.812600 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:26:51.812569 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ad76071-ccde-4279-a47a-3753c2a4b0e9" path="/var/lib/kubelet/pods/6ad76071-ccde-4279-a47a-3753c2a4b0e9/volumes" Sep 14 13:27:20.074677 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:20.074644 2564 scope.go:117] "RemoveContainer" containerID="c7ffafb0444e93a66177dc12132b903e864070e8932d18f1828512e433d41f25" Sep 14 13:27:51.010387 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.010348 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg"] Sep 14 13:27:51.011032 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.010994 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad76071-ccde-4279-a47a-3753c2a4b0e9" containerName="istio-proxy" Sep 14 13:27:51.011032 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.011013 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad76071-ccde-4279-a47a-3753c2a4b0e9" containerName="istio-proxy" Sep 14 13:27:51.011247 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.011045 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45f9dea1-c3e6-44b4-85fa-45009d40b328" containerName="maas-api" Sep 14 13:27:51.011247 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.011054 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="45f9dea1-c3e6-44b4-85fa-45009d40b328" containerName="maas-api" Sep 14 13:27:51.011247 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.011077 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35454c83-886a-4210-b181-66a7ebf30d85" containerName="test-curl-7422-62518b" Sep 14 13:27:51.011247 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.011086 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="35454c83-886a-4210-b181-66a7ebf30d85" containerName="test-curl-7422-62518b" Sep 14 13:27:51.011247 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.011198 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad76071-ccde-4279-a47a-3753c2a4b0e9" containerName="istio-proxy" Sep 14 13:27:51.011247 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.011212 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="35454c83-886a-4210-b181-66a7ebf30d85" containerName="test-curl-7422-62518b" Sep 14 13:27:51.011247 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.011222 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ad76071-ccde-4279-a47a-3753c2a4b0e9" containerName="istio-proxy" Sep 14 13:27:51.011247 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.011232 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="45f9dea1-c3e6-44b4-85fa-45009d40b328" containerName="maas-api" Sep 14 13:27:51.013808 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.013788 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" Sep 14 13:27:51.016618 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.016597 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1cf25e-serving-cert\"" Sep 14 13:27:51.025961 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.025915 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg"] Sep 14 13:27:51.112147 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.112039 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qns2b\" (UniqueName: \"kubernetes.io/projected/66a5d825-7731-41b3-94fa-5eca3547d281-kube-api-access-qns2b\") pod \"maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg\" (UID: \"66a5d825-7731-41b3-94fa-5eca3547d281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" Sep 14 13:27:51.117001 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.115437 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66a5d825-7731-41b3-94fa-5eca3547d281-openshift-service-ca\") pod \"maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg\" (UID: \"66a5d825-7731-41b3-94fa-5eca3547d281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" Sep 14 13:27:51.117001 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.115589 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a5d825-7731-41b3-94fa-5eca3547d281-metrics-certs\") pod \"maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg\" (UID: \"66a5d825-7731-41b3-94fa-5eca3547d281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" Sep 14 13:27:51.117001 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.115659 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66a5d825-7731-41b3-94fa-5eca3547d281-maas-api-tls\") pod \"maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg\" (UID: \"66a5d825-7731-41b3-94fa-5eca3547d281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" Sep 14 13:27:51.217005 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.216962 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a5d825-7731-41b3-94fa-5eca3547d281-metrics-certs\") pod \"maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg\" (UID: \"66a5d825-7731-41b3-94fa-5eca3547d281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" Sep 14 13:27:51.217215 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.217110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66a5d825-7731-41b3-94fa-5eca3547d281-maas-api-tls\") pod \"maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg\" (UID: \"66a5d825-7731-41b3-94fa-5eca3547d281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" Sep 14 13:27:51.217215 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:27:51.217135 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-serving-cert: secret "maas-api-e2e-api-b-1cf25e-serving-cert" not found Sep 14 13:27:51.217215 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.217173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qns2b\" (UniqueName: \"kubernetes.io/projected/66a5d825-7731-41b3-94fa-5eca3547d281-kube-api-access-qns2b\") pod \"maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg\" (UID: \"66a5d825-7731-41b3-94fa-5eca3547d281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" Sep 14 13:27:51.217409 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:27:51.217220 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66a5d825-7731-41b3-94fa-5eca3547d281-maas-api-tls podName:66a5d825-7731-41b3-94fa-5eca3547d281 nodeName:}" failed. No retries permitted until 2026-09-14 13:27:51.717199682 +0000 UTC m=+1772.380095307 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66a5d825-7731-41b3-94fa-5eca3547d281-maas-api-tls") pod "maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" (UID: "66a5d825-7731-41b3-94fa-5eca3547d281") : secret "maas-api-e2e-api-b-1cf25e-serving-cert" not found Sep 14 13:27:51.217409 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.217268 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66a5d825-7731-41b3-94fa-5eca3547d281-openshift-service-ca\") pod \"maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg\" (UID: \"66a5d825-7731-41b3-94fa-5eca3547d281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" Sep 14 13:27:51.218087 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.218064 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66a5d825-7731-41b3-94fa-5eca3547d281-openshift-service-ca\") pod \"maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg\" (UID: \"66a5d825-7731-41b3-94fa-5eca3547d281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" Sep 14 13:27:51.219641 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.219625 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a5d825-7731-41b3-94fa-5eca3547d281-metrics-certs\") pod \"maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg\" (UID: \"66a5d825-7731-41b3-94fa-5eca3547d281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" Sep 14 13:27:51.225748 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.225722 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qns2b\" (UniqueName: \"kubernetes.io/projected/66a5d825-7731-41b3-94fa-5eca3547d281-kube-api-access-qns2b\") pod \"maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg\" (UID: \"66a5d825-7731-41b3-94fa-5eca3547d281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" Sep 14 13:27:51.723481 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.723446 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66a5d825-7731-41b3-94fa-5eca3547d281-maas-api-tls\") pod \"maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg\" (UID: \"66a5d825-7731-41b3-94fa-5eca3547d281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" Sep 14 13:27:51.725988 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.725960 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66a5d825-7731-41b3-94fa-5eca3547d281-maas-api-tls\") pod \"maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg\" (UID: \"66a5d825-7731-41b3-94fa-5eca3547d281\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" Sep 14 13:27:51.925848 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:51.925796 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" Sep 14 13:27:52.202787 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:52.202754 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg"] Sep 14 13:27:52.203776 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:27:52.203753 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a5d825_7731_41b3_94fa_5eca3547d281.slice/crio-a746b075d710e73231c24d93c4eada04cca7a55fb238e7b60545584321fdf762 WatchSource:0}: Error finding container a746b075d710e73231c24d93c4eada04cca7a55fb238e7b60545584321fdf762: Status 404 returned error can't find the container with id a746b075d710e73231c24d93c4eada04cca7a55fb238e7b60545584321fdf762 Sep 14 13:27:53.172214 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:53.172177 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" event={"ID":"66a5d825-7731-41b3-94fa-5eca3547d281","Type":"ContainerStarted","Data":"49f0a4a62555a3da7e53bd4a06881c6d9e94b7f2e14165e572ba75d959948e3f"} Sep 14 13:27:53.172425 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:53.172234 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" event={"ID":"66a5d825-7731-41b3-94fa-5eca3547d281","Type":"ContainerStarted","Data":"a746b075d710e73231c24d93c4eada04cca7a55fb238e7b60545584321fdf762"} Sep 14 13:27:53.172425 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:53.172331 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" Sep 14 13:27:53.194450 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:53.194404 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" podStartSLOduration=2.837303757 podStartE2EDuration="3.194387292s" totalImagesPullingTime="357.083535ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:27:53.193952272 +0000 UTC m=+1773.856847893" watchObservedRunningTime="2026-09-14 13:27:53.194387292 +0000 UTC m=+1773.857282951" Sep 14 13:27:59.184504 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:27:59.184470 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" Sep 14 13:28:19.976818 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:28:19.976790 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/1.log" Sep 14 13:28:19.980940 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:28:19.980896 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/1.log" Sep 14 13:28:59.997806 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:28:59.997761 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg"] Sep 14 13:28:59.998417 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:28:59.998019 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" podUID="66a5d825-7731-41b3-94fa-5eca3547d281" containerName="maas-api" containerID="cri-o://49f0a4a62555a3da7e53bd4a06881c6d9e94b7f2e14165e572ba75d959948e3f" gracePeriod=30 Sep 14 13:29:00.255033 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.254966 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" Sep 14 13:29:00.343736 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.343692 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a5d825-7731-41b3-94fa-5eca3547d281-metrics-certs\") pod \"66a5d825-7731-41b3-94fa-5eca3547d281\" (UID: \"66a5d825-7731-41b3-94fa-5eca3547d281\") " Sep 14 13:29:00.343964 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.343780 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66a5d825-7731-41b3-94fa-5eca3547d281-openshift-service-ca\") pod \"66a5d825-7731-41b3-94fa-5eca3547d281\" (UID: \"66a5d825-7731-41b3-94fa-5eca3547d281\") " Sep 14 13:29:00.343964 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.343805 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66a5d825-7731-41b3-94fa-5eca3547d281-maas-api-tls\") pod \"66a5d825-7731-41b3-94fa-5eca3547d281\" (UID: \"66a5d825-7731-41b3-94fa-5eca3547d281\") " Sep 14 13:29:00.343964 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.343851 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qns2b\" (UniqueName: \"kubernetes.io/projected/66a5d825-7731-41b3-94fa-5eca3547d281-kube-api-access-qns2b\") pod \"66a5d825-7731-41b3-94fa-5eca3547d281\" (UID: \"66a5d825-7731-41b3-94fa-5eca3547d281\") " Sep 14 13:29:00.344209 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.344172 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66a5d825-7731-41b3-94fa-5eca3547d281-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "66a5d825-7731-41b3-94fa-5eca3547d281" (UID: "66a5d825-7731-41b3-94fa-5eca3547d281"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:29:00.344289 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.344272 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66a5d825-7731-41b3-94fa-5eca3547d281-openshift-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:00.346276 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.346240 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66a5d825-7731-41b3-94fa-5eca3547d281-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "66a5d825-7731-41b3-94fa-5eca3547d281" (UID: "66a5d825-7731-41b3-94fa-5eca3547d281"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:29:00.346405 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.346304 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66a5d825-7731-41b3-94fa-5eca3547d281-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "66a5d825-7731-41b3-94fa-5eca3547d281" (UID: "66a5d825-7731-41b3-94fa-5eca3547d281"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:29:00.346405 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.346386 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a5d825-7731-41b3-94fa-5eca3547d281-kube-api-access-qns2b" (OuterVolumeSpecName: "kube-api-access-qns2b") pod "66a5d825-7731-41b3-94fa-5eca3547d281" (UID: "66a5d825-7731-41b3-94fa-5eca3547d281"). InnerVolumeSpecName "kube-api-access-qns2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:29:00.445033 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.445004 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66a5d825-7731-41b3-94fa-5eca3547d281-metrics-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:00.445033 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.445028 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66a5d825-7731-41b3-94fa-5eca3547d281-maas-api-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:00.445033 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.445037 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qns2b\" (UniqueName: \"kubernetes.io/projected/66a5d825-7731-41b3-94fa-5eca3547d281-kube-api-access-qns2b\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:00.585447 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.585358 2564 generic.go:356] "Generic (PLEG): container finished" podID="66a5d825-7731-41b3-94fa-5eca3547d281" containerID="49f0a4a62555a3da7e53bd4a06881c6d9e94b7f2e14165e572ba75d959948e3f" exitCode=0 Sep 14 13:29:00.585447 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.585417 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" Sep 14 13:29:00.585648 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.585444 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" event={"ID":"66a5d825-7731-41b3-94fa-5eca3547d281","Type":"ContainerDied","Data":"49f0a4a62555a3da7e53bd4a06881c6d9e94b7f2e14165e572ba75d959948e3f"} Sep 14 13:29:00.585648 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.585521 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg" event={"ID":"66a5d825-7731-41b3-94fa-5eca3547d281","Type":"ContainerDied","Data":"a746b075d710e73231c24d93c4eada04cca7a55fb238e7b60545584321fdf762"} Sep 14 13:29:00.585648 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.585543 2564 scope.go:117] "RemoveContainer" containerID="49f0a4a62555a3da7e53bd4a06881c6d9e94b7f2e14165e572ba75d959948e3f" Sep 14 13:29:00.595496 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.595477 2564 scope.go:117] "RemoveContainer" containerID="49f0a4a62555a3da7e53bd4a06881c6d9e94b7f2e14165e572ba75d959948e3f" Sep 14 13:29:00.595794 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:29:00.595770 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49f0a4a62555a3da7e53bd4a06881c6d9e94b7f2e14165e572ba75d959948e3f\": container with ID starting with 49f0a4a62555a3da7e53bd4a06881c6d9e94b7f2e14165e572ba75d959948e3f not found: ID does not exist" containerID="49f0a4a62555a3da7e53bd4a06881c6d9e94b7f2e14165e572ba75d959948e3f" Sep 14 13:29:00.595905 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.595800 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49f0a4a62555a3da7e53bd4a06881c6d9e94b7f2e14165e572ba75d959948e3f"} err="failed to get container status \"49f0a4a62555a3da7e53bd4a06881c6d9e94b7f2e14165e572ba75d959948e3f\": rpc error: code = NotFound desc = could not find container \"49f0a4a62555a3da7e53bd4a06881c6d9e94b7f2e14165e572ba75d959948e3f\": container with ID starting with 49f0a4a62555a3da7e53bd4a06881c6d9e94b7f2e14165e572ba75d959948e3f not found: ID does not exist" Sep 14 13:29:00.608692 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.608670 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg"] Sep 14 13:29:00.616691 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:00.616667 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1cf25e-65dbd77d96-fxwfg"] Sep 14 13:29:01.811918 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:01.811889 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66a5d825-7731-41b3-94fa-5eca3547d281" path="/var/lib/kubelet/pods/66a5d825-7731-41b3-94fa-5eca3547d281/volumes" Sep 14 13:29:18.525363 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:18.525330 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq"] Sep 14 13:29:18.525976 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:18.525560 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" podUID="4a94e70e-b37d-4e4b-8b46-a746ea37849f" containerName="maas-api" containerID="cri-o://e65e16907910cf1164e6cd30b036777dfaa53f50a76b55018eca698c561fad86" gracePeriod=30 Sep 14 13:29:18.697357 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:18.697321 2564 generic.go:356] "Generic (PLEG): container finished" podID="4a94e70e-b37d-4e4b-8b46-a746ea37849f" containerID="e65e16907910cf1164e6cd30b036777dfaa53f50a76b55018eca698c561fad86" exitCode=0 Sep 14 13:29:18.697520 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:18.697409 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" event={"ID":"4a94e70e-b37d-4e4b-8b46-a746ea37849f","Type":"ContainerDied","Data":"e65e16907910cf1164e6cd30b036777dfaa53f50a76b55018eca698c561fad86"} Sep 14 13:29:18.783735 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:18.783673 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" Sep 14 13:29:18.833359 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:18.831488 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a94e70e-b37d-4e4b-8b46-a746ea37849f-maas-api-tls\") pod \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\" (UID: \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\") " Sep 14 13:29:18.833359 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:18.831608 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q99t\" (UniqueName: \"kubernetes.io/projected/4a94e70e-b37d-4e4b-8b46-a746ea37849f-kube-api-access-6q99t\") pod \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\" (UID: \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\") " Sep 14 13:29:18.833359 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:18.831686 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a94e70e-b37d-4e4b-8b46-a746ea37849f-openshift-service-ca\") pod \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\" (UID: \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\") " Sep 14 13:29:18.833359 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:18.831754 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a94e70e-b37d-4e4b-8b46-a746ea37849f-metrics-certs\") pod \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\" (UID: \"4a94e70e-b37d-4e4b-8b46-a746ea37849f\") " Sep 14 13:29:18.834697 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:18.834642 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a94e70e-b37d-4e4b-8b46-a746ea37849f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4a94e70e-b37d-4e4b-8b46-a746ea37849f" (UID: "4a94e70e-b37d-4e4b-8b46-a746ea37849f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:29:18.837140 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:18.837099 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a94e70e-b37d-4e4b-8b46-a746ea37849f-kube-api-access-6q99t" (OuterVolumeSpecName: "kube-api-access-6q99t") pod "4a94e70e-b37d-4e4b-8b46-a746ea37849f" (UID: "4a94e70e-b37d-4e4b-8b46-a746ea37849f"). InnerVolumeSpecName "kube-api-access-6q99t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:29:18.837679 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:18.837656 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a94e70e-b37d-4e4b-8b46-a746ea37849f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4a94e70e-b37d-4e4b-8b46-a746ea37849f" (UID: "4a94e70e-b37d-4e4b-8b46-a746ea37849f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:29:18.839797 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:18.839762 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a94e70e-b37d-4e4b-8b46-a746ea37849f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4a94e70e-b37d-4e4b-8b46-a746ea37849f" (UID: "4a94e70e-b37d-4e4b-8b46-a746ea37849f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:29:18.933850 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:18.933811 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q99t\" (UniqueName: \"kubernetes.io/projected/4a94e70e-b37d-4e4b-8b46-a746ea37849f-kube-api-access-6q99t\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:18.933850 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:18.933853 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a94e70e-b37d-4e4b-8b46-a746ea37849f-openshift-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:18.934089 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:18.933869 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a94e70e-b37d-4e4b-8b46-a746ea37849f-metrics-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:18.934089 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:18.933883 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a94e70e-b37d-4e4b-8b46-a746ea37849f-maas-api-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:19.704546 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:19.704521 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" Sep 14 13:29:19.705011 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:19.704555 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq" event={"ID":"4a94e70e-b37d-4e4b-8b46-a746ea37849f","Type":"ContainerDied","Data":"bf33d2ddf93a5270d92f9c3ce95f5d12229ddf2cae545097a910bc80eeb19f5e"} Sep 14 13:29:19.705011 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:19.704610 2564 scope.go:117] "RemoveContainer" containerID="e65e16907910cf1164e6cd30b036777dfaa53f50a76b55018eca698c561fad86" Sep 14 13:29:19.726806 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:19.726783 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq"] Sep 14 13:29:19.731034 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:19.731013 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-31063a-598b6dc64-x6nbq"] Sep 14 13:29:19.811656 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:19.811624 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a94e70e-b37d-4e4b-8b46-a746ea37849f" path="/var/lib/kubelet/pods/4a94e70e-b37d-4e4b-8b46-a746ea37849f/volumes" Sep 14 13:29:25.990944 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:25.990900 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw"] Sep 14 13:29:25.991518 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:25.991174 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" podUID="be11482a-7155-4cec-b70f-0456b0566843" containerName="istio-proxy" containerID="cri-o://702e3ae6c8293e47032e0e39c746a2b97f7ba2f907fb2060e700b5bee9d51c1f" gracePeriod=30 Sep 14 13:29:29.321478 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:29.321436 2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 13:29:29.321955 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:29.321524 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" podUID="be11482a-7155-4cec-b70f-0456b0566843" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 13:29:31.243683 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.243660 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:29:31.364575 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.364552 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-istio-envoy\") pod \"be11482a-7155-4cec-b70f-0456b0566843\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " Sep 14 13:29:31.364752 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.364585 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be11482a-7155-4cec-b70f-0456b0566843-istio-podinfo\") pod \"be11482a-7155-4cec-b70f-0456b0566843\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " Sep 14 13:29:31.364752 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.364603 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-workload-certs\") pod \"be11482a-7155-4cec-b70f-0456b0566843\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " Sep 14 13:29:31.364752 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.364618 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be11482a-7155-4cec-b70f-0456b0566843-istio-token\") pod \"be11482a-7155-4cec-b70f-0456b0566843\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " Sep 14 13:29:31.364752 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.364652 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhrlr\" (UniqueName: \"kubernetes.io/projected/be11482a-7155-4cec-b70f-0456b0566843-kube-api-access-zhrlr\") pod \"be11482a-7155-4cec-b70f-0456b0566843\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " Sep 14 13:29:31.365015 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.364773 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-credential-socket\") pod \"be11482a-7155-4cec-b70f-0456b0566843\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " Sep 14 13:29:31.365015 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.364827 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be11482a-7155-4cec-b70f-0456b0566843-istiod-ca-cert\") pod \"be11482a-7155-4cec-b70f-0456b0566843\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " Sep 14 13:29:31.365015 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.364906 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "be11482a-7155-4cec-b70f-0456b0566843" (UID: "be11482a-7155-4cec-b70f-0456b0566843"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:29:31.365015 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.364960 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-workload-socket\") pod \"be11482a-7155-4cec-b70f-0456b0566843\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " Sep 14 13:29:31.365224 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.365018 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-istio-data\") pod \"be11482a-7155-4cec-b70f-0456b0566843\" (UID: \"be11482a-7155-4cec-b70f-0456b0566843\") " Sep 14 13:29:31.365277 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.365256 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "be11482a-7155-4cec-b70f-0456b0566843" (UID: "be11482a-7155-4cec-b70f-0456b0566843"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:29:31.365331 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.365300 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be11482a-7155-4cec-b70f-0456b0566843-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "be11482a-7155-4cec-b70f-0456b0566843" (UID: "be11482a-7155-4cec-b70f-0456b0566843"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:29:31.365531 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.365506 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-istio-data" (OuterVolumeSpecName: "istio-data") pod "be11482a-7155-4cec-b70f-0456b0566843" (UID: "be11482a-7155-4cec-b70f-0456b0566843"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:29:31.365616 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.365561 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "be11482a-7155-4cec-b70f-0456b0566843" (UID: "be11482a-7155-4cec-b70f-0456b0566843"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:29:31.365729 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.365710 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:31.365763 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.365736 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:31.365763 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.365752 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:31.365826 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.365767 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:31.365826 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.365780 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be11482a-7155-4cec-b70f-0456b0566843-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:31.367061 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.367040 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be11482a-7155-4cec-b70f-0456b0566843-kube-api-access-zhrlr" (OuterVolumeSpecName: "kube-api-access-zhrlr") pod "be11482a-7155-4cec-b70f-0456b0566843" (UID: "be11482a-7155-4cec-b70f-0456b0566843"). InnerVolumeSpecName "kube-api-access-zhrlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:29:31.367061 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.367053 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "be11482a-7155-4cec-b70f-0456b0566843" (UID: "be11482a-7155-4cec-b70f-0456b0566843"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:29:31.367464 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.367449 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be11482a-7155-4cec-b70f-0456b0566843-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "be11482a-7155-4cec-b70f-0456b0566843" (UID: "be11482a-7155-4cec-b70f-0456b0566843"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 13:29:31.367511 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.367476 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be11482a-7155-4cec-b70f-0456b0566843-istio-token" (OuterVolumeSpecName: "istio-token") pod "be11482a-7155-4cec-b70f-0456b0566843" (UID: "be11482a-7155-4cec-b70f-0456b0566843"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:29:31.466349 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.466319 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be11482a-7155-4cec-b70f-0456b0566843-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:31.466349 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.466344 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhrlr\" (UniqueName: \"kubernetes.io/projected/be11482a-7155-4cec-b70f-0456b0566843-kube-api-access-zhrlr\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:31.466349 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.466354 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be11482a-7155-4cec-b70f-0456b0566843-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:31.466567 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.466363 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be11482a-7155-4cec-b70f-0456b0566843-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:31.784002 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.783895 2564 generic.go:356] "Generic (PLEG): container finished" podID="be11482a-7155-4cec-b70f-0456b0566843" containerID="702e3ae6c8293e47032e0e39c746a2b97f7ba2f907fb2060e700b5bee9d51c1f" exitCode=0 Sep 14 13:29:31.784002 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.783964 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" Sep 14 13:29:31.784002 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.783964 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" event={"ID":"be11482a-7155-4cec-b70f-0456b0566843","Type":"ContainerDied","Data":"702e3ae6c8293e47032e0e39c746a2b97f7ba2f907fb2060e700b5bee9d51c1f"} Sep 14 13:29:31.784275 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.784022 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw" event={"ID":"be11482a-7155-4cec-b70f-0456b0566843","Type":"ContainerDied","Data":"89d9558bc2ee1b7001758af9c199d6fcb760ca22a947df8cae9ec07fb17aff11"} Sep 14 13:29:31.784275 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.784046 2564 scope.go:117] "RemoveContainer" containerID="702e3ae6c8293e47032e0e39c746a2b97f7ba2f907fb2060e700b5bee9d51c1f" Sep 14 13:29:31.794938 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.794906 2564 scope.go:117] "RemoveContainer" containerID="c52727df75243ddcf885d166e8335d9c8f5408d6f88cfb77303788f7b2594074" Sep 14 13:29:31.803593 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.803574 2564 scope.go:117] "RemoveContainer" containerID="702e3ae6c8293e47032e0e39c746a2b97f7ba2f907fb2060e700b5bee9d51c1f" Sep 14 13:29:31.803862 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:29:31.803844 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"702e3ae6c8293e47032e0e39c746a2b97f7ba2f907fb2060e700b5bee9d51c1f\": container with ID starting with 702e3ae6c8293e47032e0e39c746a2b97f7ba2f907fb2060e700b5bee9d51c1f not found: ID does not exist" containerID="702e3ae6c8293e47032e0e39c746a2b97f7ba2f907fb2060e700b5bee9d51c1f" Sep 14 13:29:31.803959 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.803874 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"702e3ae6c8293e47032e0e39c746a2b97f7ba2f907fb2060e700b5bee9d51c1f"} err="failed to get container status \"702e3ae6c8293e47032e0e39c746a2b97f7ba2f907fb2060e700b5bee9d51c1f\": rpc error: code = NotFound desc = could not find container \"702e3ae6c8293e47032e0e39c746a2b97f7ba2f907fb2060e700b5bee9d51c1f\": container with ID starting with 702e3ae6c8293e47032e0e39c746a2b97f7ba2f907fb2060e700b5bee9d51c1f not found: ID does not exist" Sep 14 13:29:31.803959 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.803898 2564 scope.go:117] "RemoveContainer" containerID="c52727df75243ddcf885d166e8335d9c8f5408d6f88cfb77303788f7b2594074" Sep 14 13:29:31.804200 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:29:31.804181 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c52727df75243ddcf885d166e8335d9c8f5408d6f88cfb77303788f7b2594074\": container with ID starting with c52727df75243ddcf885d166e8335d9c8f5408d6f88cfb77303788f7b2594074 not found: ID does not exist" containerID="c52727df75243ddcf885d166e8335d9c8f5408d6f88cfb77303788f7b2594074" Sep 14 13:29:31.804237 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.804207 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c52727df75243ddcf885d166e8335d9c8f5408d6f88cfb77303788f7b2594074"} err="failed to get container status \"c52727df75243ddcf885d166e8335d9c8f5408d6f88cfb77303788f7b2594074\": rpc error: code = NotFound desc = could not find container \"c52727df75243ddcf885d166e8335d9c8f5408d6f88cfb77303788f7b2594074\": container with ID starting with c52727df75243ddcf885d166e8335d9c8f5408d6f88cfb77303788f7b2594074 not found: ID does not exist" Sep 14 13:29:31.812718 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.812697 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw"] Sep 14 13:29:31.815533 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:31.815512 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-31063a-openshift-default-d976fc474-8ngzw"] Sep 14 13:29:33.811678 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:33.811649 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be11482a-7155-4cec-b70f-0456b0566843" path="/var/lib/kubelet/pods/be11482a-7155-4cec-b70f-0456b0566843/volumes" Sep 14 13:29:45.129996 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:45.129889 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7"] Sep 14 13:29:45.130432 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:45.130262 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" podUID="484e65bc-00ba-458b-b212-edda1a7a97e3" containerName="istio-proxy" containerID="cri-o://e1c7b215f88f6a2dc02cc25ff595238161d549c26053a0f2c5f56d2ac10b13db" gracePeriod=30 Sep 14 13:29:46.428066 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:46.428028 2564 patch_prober.go:28] interesting pod/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 13:29:46.428446 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:46.428089 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" podUID="484e65bc-00ba-458b-b212-edda1a7a97e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 13:29:50.390831 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.390798 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:29:50.460080 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.459997 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-credential-socket\") pod \"484e65bc-00ba-458b-b212-edda1a7a97e3\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " Sep 14 13:29:50.460080 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.460031 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49wpj\" (UniqueName: \"kubernetes.io/projected/484e65bc-00ba-458b-b212-edda1a7a97e3-kube-api-access-49wpj\") pod \"484e65bc-00ba-458b-b212-edda1a7a97e3\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " Sep 14 13:29:50.460265 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.460119 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-podinfo\") pod \"484e65bc-00ba-458b-b212-edda1a7a97e3\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " Sep 14 13:29:50.460265 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.460233 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-workload-socket\") pod \"484e65bc-00ba-458b-b212-edda1a7a97e3\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " Sep 14 13:29:50.460356 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.460286 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-workload-certs\") pod \"484e65bc-00ba-458b-b212-edda1a7a97e3\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " Sep 14 13:29:50.460356 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.460293 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "484e65bc-00ba-458b-b212-edda1a7a97e3" (UID: "484e65bc-00ba-458b-b212-edda1a7a97e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:29:50.460356 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.460316 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-envoy\") pod \"484e65bc-00ba-458b-b212-edda1a7a97e3\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " Sep 14 13:29:50.460520 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.460353 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-token\") pod \"484e65bc-00ba-458b-b212-edda1a7a97e3\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " Sep 14 13:29:50.460520 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.460382 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-data\") pod \"484e65bc-00ba-458b-b212-edda1a7a97e3\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " Sep 14 13:29:50.460520 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.460425 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484e65bc-00ba-458b-b212-edda1a7a97e3-istiod-ca-cert\") pod \"484e65bc-00ba-458b-b212-edda1a7a97e3\" (UID: \"484e65bc-00ba-458b-b212-edda1a7a97e3\") " Sep 14 13:29:50.460520 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.460492 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "484e65bc-00ba-458b-b212-edda1a7a97e3" (UID: "484e65bc-00ba-458b-b212-edda1a7a97e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:29:50.460754 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.460509 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "484e65bc-00ba-458b-b212-edda1a7a97e3" (UID: "484e65bc-00ba-458b-b212-edda1a7a97e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:29:50.460754 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.460576 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-data" (OuterVolumeSpecName: "istio-data") pod "484e65bc-00ba-458b-b212-edda1a7a97e3" (UID: "484e65bc-00ba-458b-b212-edda1a7a97e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:29:50.460867 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.460849 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/484e65bc-00ba-458b-b212-edda1a7a97e3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "484e65bc-00ba-458b-b212-edda1a7a97e3" (UID: "484e65bc-00ba-458b-b212-edda1a7a97e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:29:50.461065 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.461006 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:50.461065 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.461031 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:50.461065 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.461044 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:50.461065 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.461056 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:50.462527 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.462501 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "484e65bc-00ba-458b-b212-edda1a7a97e3" (UID: "484e65bc-00ba-458b-b212-edda1a7a97e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 13:29:50.462636 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.462607 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "484e65bc-00ba-458b-b212-edda1a7a97e3" (UID: "484e65bc-00ba-458b-b212-edda1a7a97e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:29:50.462636 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.462607 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484e65bc-00ba-458b-b212-edda1a7a97e3-kube-api-access-49wpj" (OuterVolumeSpecName: "kube-api-access-49wpj") pod "484e65bc-00ba-458b-b212-edda1a7a97e3" (UID: "484e65bc-00ba-458b-b212-edda1a7a97e3"). InnerVolumeSpecName "kube-api-access-49wpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:29:50.462636 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.462618 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-token" (OuterVolumeSpecName: "istio-token") pod "484e65bc-00ba-458b-b212-edda1a7a97e3" (UID: "484e65bc-00ba-458b-b212-edda1a7a97e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:29:50.562436 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.562399 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:50.562436 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.562428 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:50.562436 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.562436 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/484e65bc-00ba-458b-b212-edda1a7a97e3-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:50.562436 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.562445 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49wpj\" (UniqueName: \"kubernetes.io/projected/484e65bc-00ba-458b-b212-edda1a7a97e3-kube-api-access-49wpj\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:50.562705 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.562454 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/484e65bc-00ba-458b-b212-edda1a7a97e3-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:29:50.898009 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.897969 2564 generic.go:356] "Generic (PLEG): container finished" podID="484e65bc-00ba-458b-b212-edda1a7a97e3" containerID="e1c7b215f88f6a2dc02cc25ff595238161d549c26053a0f2c5f56d2ac10b13db" exitCode=0 Sep 14 13:29:50.898209 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.898028 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" Sep 14 13:29:50.898209 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.898064 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" event={"ID":"484e65bc-00ba-458b-b212-edda1a7a97e3","Type":"ContainerDied","Data":"e1c7b215f88f6a2dc02cc25ff595238161d549c26053a0f2c5f56d2ac10b13db"} Sep 14 13:29:50.898209 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.898124 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7" event={"ID":"484e65bc-00ba-458b-b212-edda1a7a97e3","Type":"ContainerDied","Data":"cab36624f7e378b7af9e9fff51b513802e1e57739de55113048fe2842034c418"} Sep 14 13:29:50.898209 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.898150 2564 scope.go:117] "RemoveContainer" containerID="e1c7b215f88f6a2dc02cc25ff595238161d549c26053a0f2c5f56d2ac10b13db" Sep 14 13:29:50.908520 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.908489 2564 scope.go:117] "RemoveContainer" containerID="8494d7354344aba6233b6f881199a78dc5e0b1372f2ea76bd3fca8b148eadf88" Sep 14 13:29:50.917212 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.917182 2564 scope.go:117] "RemoveContainer" containerID="e1c7b215f88f6a2dc02cc25ff595238161d549c26053a0f2c5f56d2ac10b13db" Sep 14 13:29:50.917481 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:29:50.917465 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c7b215f88f6a2dc02cc25ff595238161d549c26053a0f2c5f56d2ac10b13db\": container with ID starting with e1c7b215f88f6a2dc02cc25ff595238161d549c26053a0f2c5f56d2ac10b13db not found: ID does not exist" containerID="e1c7b215f88f6a2dc02cc25ff595238161d549c26053a0f2c5f56d2ac10b13db" Sep 14 13:29:50.917532 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.917489 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c7b215f88f6a2dc02cc25ff595238161d549c26053a0f2c5f56d2ac10b13db"} err="failed to get container status \"e1c7b215f88f6a2dc02cc25ff595238161d549c26053a0f2c5f56d2ac10b13db\": rpc error: code = NotFound desc = could not find container \"e1c7b215f88f6a2dc02cc25ff595238161d549c26053a0f2c5f56d2ac10b13db\": container with ID starting with e1c7b215f88f6a2dc02cc25ff595238161d549c26053a0f2c5f56d2ac10b13db not found: ID does not exist" Sep 14 13:29:50.917532 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.917504 2564 scope.go:117] "RemoveContainer" containerID="8494d7354344aba6233b6f881199a78dc5e0b1372f2ea76bd3fca8b148eadf88" Sep 14 13:29:50.917728 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:29:50.917714 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8494d7354344aba6233b6f881199a78dc5e0b1372f2ea76bd3fca8b148eadf88\": container with ID starting with 8494d7354344aba6233b6f881199a78dc5e0b1372f2ea76bd3fca8b148eadf88 not found: ID does not exist" containerID="8494d7354344aba6233b6f881199a78dc5e0b1372f2ea76bd3fca8b148eadf88" Sep 14 13:29:50.917773 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.917733 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8494d7354344aba6233b6f881199a78dc5e0b1372f2ea76bd3fca8b148eadf88"} err="failed to get container status \"8494d7354344aba6233b6f881199a78dc5e0b1372f2ea76bd3fca8b148eadf88\": rpc error: code = NotFound desc = could not find container \"8494d7354344aba6233b6f881199a78dc5e0b1372f2ea76bd3fca8b148eadf88\": container with ID starting with 8494d7354344aba6233b6f881199a78dc5e0b1372f2ea76bd3fca8b148eadf88 not found: ID does not exist" Sep 14 13:29:50.930174 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.927472 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7"] Sep 14 13:29:50.933780 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:50.933751 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b8be88-openshift-default-764ffd8cbb-gbql7"] Sep 14 13:29:51.811889 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:29:51.811854 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="484e65bc-00ba-458b-b212-edda1a7a97e3" path="/var/lib/kubelet/pods/484e65bc-00ba-458b-b212-edda1a7a97e3/volumes" Sep 14 13:30:24.858861 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.858829 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6"] Sep 14 13:30:24.859301 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.859287 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be11482a-7155-4cec-b70f-0456b0566843" containerName="istio-proxy" Sep 14 13:30:24.859349 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.859303 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="be11482a-7155-4cec-b70f-0456b0566843" containerName="istio-proxy" Sep 14 13:30:24.859349 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.859313 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="484e65bc-00ba-458b-b212-edda1a7a97e3" containerName="istio-proxy" Sep 14 13:30:24.859349 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.859319 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="484e65bc-00ba-458b-b212-edda1a7a97e3" containerName="istio-proxy" Sep 14 13:30:24.859349 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.859328 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="484e65bc-00ba-458b-b212-edda1a7a97e3" containerName="istio-proxy" Sep 14 13:30:24.859349 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.859335 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="484e65bc-00ba-458b-b212-edda1a7a97e3" containerName="istio-proxy" Sep 14 13:30:24.859505 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.859350 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be11482a-7155-4cec-b70f-0456b0566843" containerName="istio-proxy" Sep 14 13:30:24.859505 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.859359 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="be11482a-7155-4cec-b70f-0456b0566843" containerName="istio-proxy" Sep 14 13:30:24.859505 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.859369 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ad76071-ccde-4279-a47a-3753c2a4b0e9" containerName="istio-proxy" Sep 14 13:30:24.859505 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.859374 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ad76071-ccde-4279-a47a-3753c2a4b0e9" containerName="istio-proxy" Sep 14 13:30:24.859505 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.859389 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66a5d825-7731-41b3-94fa-5eca3547d281" containerName="maas-api" Sep 14 13:30:24.859505 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.859394 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="66a5d825-7731-41b3-94fa-5eca3547d281" containerName="maas-api" Sep 14 13:30:24.859505 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.859399 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a94e70e-b37d-4e4b-8b46-a746ea37849f" containerName="maas-api" Sep 14 13:30:24.859505 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.859405 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a94e70e-b37d-4e4b-8b46-a746ea37849f" containerName="maas-api" Sep 14 13:30:24.859505 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.859471 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a94e70e-b37d-4e4b-8b46-a746ea37849f" containerName="maas-api" Sep 14 13:30:24.859505 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.859480 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="be11482a-7155-4cec-b70f-0456b0566843" containerName="istio-proxy" Sep 14 13:30:24.859505 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.859486 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="484e65bc-00ba-458b-b212-edda1a7a97e3" containerName="istio-proxy" Sep 14 13:30:24.859505 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.859492 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="be11482a-7155-4cec-b70f-0456b0566843" containerName="istio-proxy" Sep 14 13:30:24.859505 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.859500 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="484e65bc-00ba-458b-b212-edda1a7a97e3" containerName="istio-proxy" Sep 14 13:30:24.859505 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.859509 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="66a5d825-7731-41b3-94fa-5eca3547d281" containerName="maas-api" Sep 14 13:30:24.862630 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.862614 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" Sep 14 13:30:24.865408 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.865380 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c87a95-serving-cert\"" Sep 14 13:30:24.877121 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.877099 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6"] Sep 14 13:30:24.895814 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.895785 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ce657a-287c-47c7-816f-e78c2b7aa291-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6\" (UID: \"f3ce657a-287c-47c7-816f-e78c2b7aa291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" Sep 14 13:30:24.895986 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.895822 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3ce657a-287c-47c7-816f-e78c2b7aa291-metrics-certs\") pod \"maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6\" (UID: \"f3ce657a-287c-47c7-816f-e78c2b7aa291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" Sep 14 13:30:24.895986 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.895936 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3ce657a-287c-47c7-816f-e78c2b7aa291-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6\" (UID: \"f3ce657a-287c-47c7-816f-e78c2b7aa291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" Sep 14 13:30:24.896128 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.896103 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96bcq\" (UniqueName: \"kubernetes.io/projected/f3ce657a-287c-47c7-816f-e78c2b7aa291-kube-api-access-96bcq\") pod \"maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6\" (UID: \"f3ce657a-287c-47c7-816f-e78c2b7aa291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" Sep 14 13:30:24.997334 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.997294 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96bcq\" (UniqueName: \"kubernetes.io/projected/f3ce657a-287c-47c7-816f-e78c2b7aa291-kube-api-access-96bcq\") pod \"maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6\" (UID: \"f3ce657a-287c-47c7-816f-e78c2b7aa291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" Sep 14 13:30:24.997334 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.997343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ce657a-287c-47c7-816f-e78c2b7aa291-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6\" (UID: \"f3ce657a-287c-47c7-816f-e78c2b7aa291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" Sep 14 13:30:24.997568 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.997362 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3ce657a-287c-47c7-816f-e78c2b7aa291-metrics-certs\") pod \"maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6\" (UID: \"f3ce657a-287c-47c7-816f-e78c2b7aa291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" Sep 14 13:30:24.997568 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.997398 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3ce657a-287c-47c7-816f-e78c2b7aa291-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6\" (UID: \"f3ce657a-287c-47c7-816f-e78c2b7aa291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" Sep 14 13:30:24.997568 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:30:24.997501 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-serving-cert: secret "maas-api-e2e-ipp-a-c87a95-serving-cert" not found Sep 14 13:30:24.997568 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:30:24.997568 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3ce657a-287c-47c7-816f-e78c2b7aa291-maas-api-tls podName:f3ce657a-287c-47c7-816f-e78c2b7aa291 nodeName:}" failed. No retries permitted until 2026-09-14 13:30:25.497550194 +0000 UTC m=+1926.160445808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3ce657a-287c-47c7-816f-e78c2b7aa291-maas-api-tls") pod "maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" (UID: "f3ce657a-287c-47c7-816f-e78c2b7aa291") : secret "maas-api-e2e-ipp-a-c87a95-serving-cert" not found Sep 14 13:30:24.998111 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:24.998093 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ce657a-287c-47c7-816f-e78c2b7aa291-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6\" (UID: \"f3ce657a-287c-47c7-816f-e78c2b7aa291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" Sep 14 13:30:25.000161 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:25.000133 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3ce657a-287c-47c7-816f-e78c2b7aa291-metrics-certs\") pod \"maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6\" (UID: \"f3ce657a-287c-47c7-816f-e78c2b7aa291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" Sep 14 13:30:25.006700 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:25.006669 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96bcq\" (UniqueName: \"kubernetes.io/projected/f3ce657a-287c-47c7-816f-e78c2b7aa291-kube-api-access-96bcq\") pod \"maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6\" (UID: \"f3ce657a-287c-47c7-816f-e78c2b7aa291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" Sep 14 13:30:25.502673 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:25.502634 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3ce657a-287c-47c7-816f-e78c2b7aa291-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6\" (UID: \"f3ce657a-287c-47c7-816f-e78c2b7aa291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" Sep 14 13:30:25.505339 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:25.505315 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3ce657a-287c-47c7-816f-e78c2b7aa291-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6\" (UID: \"f3ce657a-287c-47c7-816f-e78c2b7aa291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" Sep 14 13:30:25.773548 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:25.773450 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" Sep 14 13:30:26.048938 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:26.048894 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6"] Sep 14 13:30:26.050145 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:30:26.050117 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ce657a_287c_47c7_816f_e78c2b7aa291.slice/crio-3fee80659aa385a5ae7515e9066a7f4a91fb12d4c29cb88c5bc0edfa245ab36e WatchSource:0}: Error finding container 3fee80659aa385a5ae7515e9066a7f4a91fb12d4c29cb88c5bc0edfa245ab36e: Status 404 returned error can't find the container with id 3fee80659aa385a5ae7515e9066a7f4a91fb12d4c29cb88c5bc0edfa245ab36e Sep 14 13:30:26.104414 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:26.104385 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" event={"ID":"f3ce657a-287c-47c7-816f-e78c2b7aa291","Type":"ContainerStarted","Data":"3fee80659aa385a5ae7515e9066a7f4a91fb12d4c29cb88c5bc0edfa245ab36e"} Sep 14 13:30:27.112182 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:27.112140 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" event={"ID":"f3ce657a-287c-47c7-816f-e78c2b7aa291","Type":"ContainerStarted","Data":"923b5971e1bde00f0b15f81f2646b0df50abea63a234ebc700561093f9d283d7"} Sep 14 13:30:27.112590 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:27.112229 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" Sep 14 13:30:27.134769 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:27.134710 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" podStartSLOduration=2.662512638 podStartE2EDuration="3.134684365s" totalImagesPullingTime="472.171727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:30:27.13250033 +0000 UTC m=+1927.795395960" watchObservedRunningTime="2026-09-14 13:30:27.134684365 +0000 UTC m=+1927.797579995" Sep 14 13:30:33.125515 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:33.125486 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" Sep 14 13:30:54.553277 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:54.553240 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc"] Sep 14 13:30:54.556780 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:54.556756 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" Sep 14 13:30:54.560169 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:54.560148 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-27ae76-serving-cert\"" Sep 14 13:30:54.567049 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:54.567003 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc"] Sep 14 13:30:54.695819 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:54.695778 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfb3b76-47b2-49d9-aed2-ad99befab7df-metrics-certs\") pod \"maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc\" (UID: \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" Sep 14 13:30:54.696036 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:54.695870 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79js6\" (UniqueName: \"kubernetes.io/projected/bbfb3b76-47b2-49d9-aed2-ad99befab7df-kube-api-access-79js6\") pod \"maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc\" (UID: \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" Sep 14 13:30:54.696036 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:54.695892 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbfb3b76-47b2-49d9-aed2-ad99befab7df-maas-api-tls\") pod \"maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc\" (UID: \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" Sep 14 13:30:54.696036 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:54.695911 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbfb3b76-47b2-49d9-aed2-ad99befab7df-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc\" (UID: \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" Sep 14 13:30:54.796993 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:54.796949 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfb3b76-47b2-49d9-aed2-ad99befab7df-metrics-certs\") pod \"maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc\" (UID: \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" Sep 14 13:30:54.797160 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:54.797100 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79js6\" (UniqueName: \"kubernetes.io/projected/bbfb3b76-47b2-49d9-aed2-ad99befab7df-kube-api-access-79js6\") pod \"maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc\" (UID: \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" Sep 14 13:30:54.797160 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:54.797135 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbfb3b76-47b2-49d9-aed2-ad99befab7df-maas-api-tls\") pod \"maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc\" (UID: \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" Sep 14 13:30:54.797291 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:54.797162 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbfb3b76-47b2-49d9-aed2-ad99befab7df-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc\" (UID: \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" Sep 14 13:30:54.797349 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:30:54.797291 2564 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-serving-cert: secret "maas-api-e2e-ipp-b-27ae76-serving-cert" not found Sep 14 13:30:54.797403 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:30:54.797389 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbfb3b76-47b2-49d9-aed2-ad99befab7df-maas-api-tls podName:bbfb3b76-47b2-49d9-aed2-ad99befab7df nodeName:}" failed. No retries permitted until 2026-09-14 13:30:55.297366307 +0000 UTC m=+1955.960261938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bbfb3b76-47b2-49d9-aed2-ad99befab7df-maas-api-tls") pod "maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" (UID: "bbfb3b76-47b2-49d9-aed2-ad99befab7df") : secret "maas-api-e2e-ipp-b-27ae76-serving-cert" not found Sep 14 13:30:54.797799 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:54.797778 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbfb3b76-47b2-49d9-aed2-ad99befab7df-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc\" (UID: \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" Sep 14 13:30:54.799571 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:54.799551 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfb3b76-47b2-49d9-aed2-ad99befab7df-metrics-certs\") pod \"maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc\" (UID: \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" Sep 14 13:30:54.808241 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:54.808185 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79js6\" (UniqueName: \"kubernetes.io/projected/bbfb3b76-47b2-49d9-aed2-ad99befab7df-kube-api-access-79js6\") pod \"maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc\" (UID: \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" Sep 14 13:30:55.302306 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:55.302269 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbfb3b76-47b2-49d9-aed2-ad99befab7df-maas-api-tls\") pod \"maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc\" (UID: \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" Sep 14 13:30:55.305096 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:55.305067 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbfb3b76-47b2-49d9-aed2-ad99befab7df-maas-api-tls\") pod \"maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc\" (UID: \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" Sep 14 13:30:55.468945 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:55.468883 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" Sep 14 13:30:55.752613 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:55.752579 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc"] Sep 14 13:30:55.754126 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:30:55.754103 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbfb3b76_47b2_49d9_aed2_ad99befab7df.slice/crio-a9470973002dd419d19b88266b4fb0b82f1bfe50918098e90cce8552e6ccc461 WatchSource:0}: Error finding container a9470973002dd419d19b88266b4fb0b82f1bfe50918098e90cce8552e6ccc461: Status 404 returned error can't find the container with id a9470973002dd419d19b88266b4fb0b82f1bfe50918098e90cce8552e6ccc461 Sep 14 13:30:56.286198 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:56.286157 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" event={"ID":"bbfb3b76-47b2-49d9-aed2-ad99befab7df","Type":"ContainerStarted","Data":"012440aeb29751ce4add5b856190e19b55bb7b1d45bae574affc08f8a7091dbd"} Sep 14 13:30:56.286667 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:56.286217 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" event={"ID":"bbfb3b76-47b2-49d9-aed2-ad99befab7df","Type":"ContainerStarted","Data":"a9470973002dd419d19b88266b4fb0b82f1bfe50918098e90cce8552e6ccc461"} Sep 14 13:30:56.286667 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:56.286245 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" Sep 14 13:30:56.306663 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:30:56.306611 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" podStartSLOduration=1.954260381 podStartE2EDuration="2.306595235s" totalImagesPullingTime="352.334854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:30:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:30:56.305446595 +0000 UTC m=+1956.968342217" watchObservedRunningTime="2026-09-14 13:30:56.306595235 +0000 UTC m=+1956.969490909" Sep 14 13:31:02.297352 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:02.297324 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" Sep 14 13:31:58.872395 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:58.872350 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s"] Sep 14 13:31:58.876534 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:58.876514 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:58.879611 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:58.879596 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-8e2f31-openshift-default-dockercfg-vd695\"" Sep 14 13:31:58.888323 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:58.888302 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s"] Sep 14 13:31:58.914978 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:58.914945 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-workload-socket\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:58.914978 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:58.914982 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:58.915156 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:58.915002 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-workload-certs\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:58.915156 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:58.915097 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-token\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:58.915156 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:58.915143 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-credential-socket\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:58.915295 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:58.915164 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-884tk\" (UniqueName: \"kubernetes.io/projected/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-kube-api-access-884tk\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:58.915295 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:58.915192 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-envoy\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:58.915399 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:58.915379 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-data\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:58.915459 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:58.915433 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-podinfo\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:59.016525 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.016495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-envoy\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:59.016708 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.016548 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-data\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:59.016708 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.016572 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-podinfo\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:59.016708 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.016626 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-workload-socket\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:59.016708 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.016645 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:59.016708 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.016668 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-workload-certs\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:59.017027 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.016714 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-token\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:59.017027 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.016747 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-credential-socket\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:59.017027 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.016764 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-884tk\" (UniqueName: \"kubernetes.io/projected/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-kube-api-access-884tk\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:59.017027 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.016999 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-data\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:59.017237 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.017104 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-workload-certs\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:59.017237 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.017165 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-workload-socket\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:59.017325 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.017285 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-credential-socket\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:59.017378 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.017366 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istiod-ca-cert\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:59.019115 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.019088 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-envoy\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:59.019292 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.019276 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-podinfo\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:59.026261 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.026215 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-token\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:59.027442 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.027405 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-884tk\" (UniqueName: \"kubernetes.io/projected/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-kube-api-access-884tk\") pod \"e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:59.188131 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.188046 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:31:59.471187 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.471151 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s"] Sep 14 13:31:59.471839 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:31:59.471811 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5a7b78_5ca3_427d_b66d_be84180b6dd1.slice/crio-f44e94fe9b10b45ee7b9fd3f83ff60bd29f7eaac64113bd567fc4cf92121ab22 WatchSource:0}: Error finding container f44e94fe9b10b45ee7b9fd3f83ff60bd29f7eaac64113bd567fc4cf92121ab22: Status 404 returned error can't find the container with id f44e94fe9b10b45ee7b9fd3f83ff60bd29f7eaac64113bd567fc4cf92121ab22 Sep 14 13:31:59.474287 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.474256 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:31:59.474373 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.474320 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:31:59.474373 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.474355 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:31:59.650785 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.650736 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" event={"ID":"3b5a7b78-5ca3-427d-b66d-be84180b6dd1","Type":"ContainerStarted","Data":"13b5068a5d325c3b2172b757bf9181b4c17a70f489c91d501938f58c8ec6c85d"} Sep 14 13:31:59.651001 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.650812 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" event={"ID":"3b5a7b78-5ca3-427d-b66d-be84180b6dd1","Type":"ContainerStarted","Data":"f44e94fe9b10b45ee7b9fd3f83ff60bd29f7eaac64113bd567fc4cf92121ab22"} Sep 14 13:31:59.674756 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:31:59.674699 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" podStartSLOduration=1.674679921 podStartE2EDuration="1.674679921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:31:59.674300913 +0000 UTC m=+2020.337196555" watchObservedRunningTime="2026-09-14 13:31:59.674679921 +0000 UTC m=+2020.337575556" Sep 14 13:32:00.188583 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:00.188550 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:32:00.194226 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:00.194202 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:32:00.656392 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:00.656356 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:32:00.657187 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:00.657171 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:32:21.778748 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:21.778714 2564 generic.go:356] "Generic (PLEG): container finished" podID="3b5a7b78-5ca3-427d-b66d-be84180b6dd1" containerID="13b5068a5d325c3b2172b757bf9181b4c17a70f489c91d501938f58c8ec6c85d" exitCode=0 Sep 14 13:32:21.779191 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:21.778790 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" event={"ID":"3b5a7b78-5ca3-427d-b66d-be84180b6dd1","Type":"ContainerDied","Data":"13b5068a5d325c3b2172b757bf9181b4c17a70f489c91d501938f58c8ec6c85d"} Sep 14 13:32:21.779240 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:21.779195 2564 scope.go:117] "RemoveContainer" containerID="13b5068a5d325c3b2172b757bf9181b4c17a70f489c91d501938f58c8ec6c85d" Sep 14 13:32:21.780370 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:21.780346 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:32:21.780440 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:21.780414 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:32:21.780479 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:21.780443 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"} Sep 14 13:32:22.787125 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:22.787088 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" event={"ID":"3b5a7b78-5ca3-427d-b66d-be84180b6dd1","Type":"ContainerStarted","Data":"2438d6f820f916013563c8ed20876e97a79e5bb8dfdcd39115606ac4cc3543a4"} Sep 14 13:32:23.188603 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:23.188570 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:32:23.188789 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:23.188748 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 14 13:32:23.188834 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:23.188806 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" podUID="3b5a7b78-5ca3-427d-b66d-be84180b6dd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 14 13:32:24.189496 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:24.189461 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 14 13:32:24.189903 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:24.189521 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" podUID="3b5a7b78-5ca3-427d-b66d-be84180b6dd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 14 13:32:25.188683 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:25.188648 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 14 13:32:25.188877 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:25.188703 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" podUID="3b5a7b78-5ca3-427d-b66d-be84180b6dd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 14 13:32:26.188484 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:26.188444 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 14 13:32:26.189004 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:26.188513 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" podUID="3b5a7b78-5ca3-427d-b66d-be84180b6dd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 14 13:32:27.189177 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:27.189142 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Sep 14 13:32:27.189592 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:27.189204 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" podUID="3b5a7b78-5ca3-427d-b66d-be84180b6dd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Sep 14 13:32:28.193532 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:28.193501 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:32:28.193906 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:28.193777 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:32:28.194526 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:28.194510 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:32:40.242673 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:40.242640 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s"] Sep 14 13:32:40.243138 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:40.242854 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" podUID="3b5a7b78-5ca3-427d-b66d-be84180b6dd1" containerName="istio-proxy" containerID="cri-o://2438d6f820f916013563c8ed20876e97a79e5bb8dfdcd39115606ac4cc3543a4" gracePeriod=30 Sep 14 13:32:43.197209 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:43.197109 2564 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 14 13:32:43.197209 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:43.197191 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" podUID="3b5a7b78-5ca3-427d-b66d-be84180b6dd1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 14 13:32:45.506032 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.505979 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:32:45.621037 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.620987 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-884tk\" (UniqueName: \"kubernetes.io/projected/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-kube-api-access-884tk\") pod \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " Sep 14 13:32:45.621037 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.621041 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-credential-socket\") pod \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " Sep 14 13:32:45.621266 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.621078 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-workload-certs\") pod \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " Sep 14 13:32:45.621266 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.621170 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-token\") pod \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " Sep 14 13:32:45.621266 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.621214 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-envoy\") pod \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " Sep 14 13:32:45.621266 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.621261 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-workload-socket\") pod \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " Sep 14 13:32:45.621464 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.621288 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-podinfo\") pod \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " Sep 14 13:32:45.621464 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.621316 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-data\") pod \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " Sep 14 13:32:45.621464 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.621357 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istiod-ca-cert\") pod \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\" (UID: \"3b5a7b78-5ca3-427d-b66d-be84180b6dd1\") " Sep 14 13:32:45.621464 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.621376 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3b5a7b78-5ca3-427d-b66d-be84180b6dd1" (UID: "3b5a7b78-5ca3-427d-b66d-be84180b6dd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:32:45.621464 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.621390 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3b5a7b78-5ca3-427d-b66d-be84180b6dd1" (UID: "3b5a7b78-5ca3-427d-b66d-be84180b6dd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:32:45.621734 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.621574 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3b5a7b78-5ca3-427d-b66d-be84180b6dd1" (UID: "3b5a7b78-5ca3-427d-b66d-be84180b6dd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:32:45.621734 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.621611 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-data" (OuterVolumeSpecName: "istio-data") pod "3b5a7b78-5ca3-427d-b66d-be84180b6dd1" (UID: "3b5a7b78-5ca3-427d-b66d-be84180b6dd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:32:45.621879 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.621841 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:32:45.622114 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.621884 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3b5a7b78-5ca3-427d-b66d-be84180b6dd1" (UID: "3b5a7b78-5ca3-427d-b66d-be84180b6dd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:32:45.622114 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.621911 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:32:45.622114 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.622077 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:32:45.622114 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.622091 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:32:45.623503 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.623478 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-kube-api-access-884tk" (OuterVolumeSpecName: "kube-api-access-884tk") pod "3b5a7b78-5ca3-427d-b66d-be84180b6dd1" (UID: "3b5a7b78-5ca3-427d-b66d-be84180b6dd1"). InnerVolumeSpecName "kube-api-access-884tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:32:45.623503 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.623484 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3b5a7b78-5ca3-427d-b66d-be84180b6dd1" (UID: "3b5a7b78-5ca3-427d-b66d-be84180b6dd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:32:45.623651 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.623525 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-token" (OuterVolumeSpecName: "istio-token") pod "3b5a7b78-5ca3-427d-b66d-be84180b6dd1" (UID: "3b5a7b78-5ca3-427d-b66d-be84180b6dd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:32:45.623651 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.623575 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3b5a7b78-5ca3-427d-b66d-be84180b6dd1" (UID: "3b5a7b78-5ca3-427d-b66d-be84180b6dd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 13:32:45.723218 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.723169 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:32:45.723218 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.723210 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:32:45.723218 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.723220 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:32:45.723218 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.723227 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-884tk\" (UniqueName: \"kubernetes.io/projected/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-kube-api-access-884tk\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:32:45.723218 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.723236 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b5a7b78-5ca3-427d-b66d-be84180b6dd1-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:32:45.931348 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.931301 2564 generic.go:356] "Generic (PLEG): container finished" podID="3b5a7b78-5ca3-427d-b66d-be84180b6dd1" containerID="2438d6f820f916013563c8ed20876e97a79e5bb8dfdcd39115606ac4cc3543a4" exitCode=0 Sep 14 13:32:45.931554 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.931398 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" event={"ID":"3b5a7b78-5ca3-427d-b66d-be84180b6dd1","Type":"ContainerDied","Data":"2438d6f820f916013563c8ed20876e97a79e5bb8dfdcd39115606ac4cc3543a4"} Sep 14 13:32:45.931554 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.931479 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" Sep 14 13:32:45.931554 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.931507 2564 scope.go:117] "RemoveContainer" containerID="2438d6f820f916013563c8ed20876e97a79e5bb8dfdcd39115606ac4cc3543a4" Sep 14 13:32:45.931739 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.931576 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s" event={"ID":"3b5a7b78-5ca3-427d-b66d-be84180b6dd1","Type":"ContainerDied","Data":"f44e94fe9b10b45ee7b9fd3f83ff60bd29f7eaac64113bd567fc4cf92121ab22"} Sep 14 13:32:45.941710 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.941693 2564 scope.go:117] "RemoveContainer" containerID="13b5068a5d325c3b2172b757bf9181b4c17a70f489c91d501938f58c8ec6c85d" Sep 14 13:32:45.949784 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.949756 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s"] Sep 14 13:32:45.951180 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.951158 2564 scope.go:117] "RemoveContainer" containerID="2438d6f820f916013563c8ed20876e97a79e5bb8dfdcd39115606ac4cc3543a4" Sep 14 13:32:45.951692 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:32:45.951664 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2438d6f820f916013563c8ed20876e97a79e5bb8dfdcd39115606ac4cc3543a4\": container with ID starting with 2438d6f820f916013563c8ed20876e97a79e5bb8dfdcd39115606ac4cc3543a4 not found: ID does not exist" containerID="2438d6f820f916013563c8ed20876e97a79e5bb8dfdcd39115606ac4cc3543a4" Sep 14 13:32:45.951784 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.951705 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2438d6f820f916013563c8ed20876e97a79e5bb8dfdcd39115606ac4cc3543a4"} err="failed to get container status \"2438d6f820f916013563c8ed20876e97a79e5bb8dfdcd39115606ac4cc3543a4\": rpc error: code = NotFound desc = could not find container \"2438d6f820f916013563c8ed20876e97a79e5bb8dfdcd39115606ac4cc3543a4\": container with ID starting with 2438d6f820f916013563c8ed20876e97a79e5bb8dfdcd39115606ac4cc3543a4 not found: ID does not exist" Sep 14 13:32:45.951784 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.951730 2564 scope.go:117] "RemoveContainer" containerID="13b5068a5d325c3b2172b757bf9181b4c17a70f489c91d501938f58c8ec6c85d" Sep 14 13:32:45.952075 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:32:45.952053 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13b5068a5d325c3b2172b757bf9181b4c17a70f489c91d501938f58c8ec6c85d\": container with ID starting with 13b5068a5d325c3b2172b757bf9181b4c17a70f489c91d501938f58c8ec6c85d not found: ID does not exist" containerID="13b5068a5d325c3b2172b757bf9181b4c17a70f489c91d501938f58c8ec6c85d" Sep 14 13:32:45.952172 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.952080 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13b5068a5d325c3b2172b757bf9181b4c17a70f489c91d501938f58c8ec6c85d"} err="failed to get container status \"13b5068a5d325c3b2172b757bf9181b4c17a70f489c91d501938f58c8ec6c85d\": rpc error: code = NotFound desc = could not find container \"13b5068a5d325c3b2172b757bf9181b4c17a70f489c91d501938f58c8ec6c85d\": container with ID starting with 13b5068a5d325c3b2172b757bf9181b4c17a70f489c91d501938f58c8ec6c85d not found: ID does not exist" Sep 14 13:32:45.954098 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:45.954077 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-8e2f31-openshift-default-6dc5c6fb6b-mtj8s"] Sep 14 13:32:47.812264 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:47.812227 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b5a7b78-5ca3-427d-b66d-be84180b6dd1" path="/var/lib/kubelet/pods/3b5a7b78-5ca3-427d-b66d-be84180b6dd1/volumes" Sep 14 13:32:59.171004 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:59.170972 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6"] Sep 14 13:32:59.171597 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:59.171267 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" podUID="f3ce657a-287c-47c7-816f-e78c2b7aa291" containerName="maas-api" containerID="cri-o://923b5971e1bde00f0b15f81f2646b0df50abea63a234ebc700561093f9d283d7" gracePeriod=30 Sep 14 13:32:59.425459 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:59.425388 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" Sep 14 13:32:59.464388 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:59.464356 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ce657a-287c-47c7-816f-e78c2b7aa291-openshift-service-ca\") pod \"f3ce657a-287c-47c7-816f-e78c2b7aa291\" (UID: \"f3ce657a-287c-47c7-816f-e78c2b7aa291\") " Sep 14 13:32:59.464595 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:59.464438 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3ce657a-287c-47c7-816f-e78c2b7aa291-maas-api-tls\") pod \"f3ce657a-287c-47c7-816f-e78c2b7aa291\" (UID: \"f3ce657a-287c-47c7-816f-e78c2b7aa291\") " Sep 14 13:32:59.464595 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:59.464486 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3ce657a-287c-47c7-816f-e78c2b7aa291-metrics-certs\") pod \"f3ce657a-287c-47c7-816f-e78c2b7aa291\" (UID: \"f3ce657a-287c-47c7-816f-e78c2b7aa291\") " Sep 14 13:32:59.464595 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:59.464545 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96bcq\" (UniqueName: \"kubernetes.io/projected/f3ce657a-287c-47c7-816f-e78c2b7aa291-kube-api-access-96bcq\") pod \"f3ce657a-287c-47c7-816f-e78c2b7aa291\" (UID: \"f3ce657a-287c-47c7-816f-e78c2b7aa291\") " Sep 14 13:32:59.465078 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:59.464824 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3ce657a-287c-47c7-816f-e78c2b7aa291-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f3ce657a-287c-47c7-816f-e78c2b7aa291" (UID: "f3ce657a-287c-47c7-816f-e78c2b7aa291"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:32:59.465078 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:59.464960 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3ce657a-287c-47c7-816f-e78c2b7aa291-openshift-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:32:59.468109 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:59.468075 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3ce657a-287c-47c7-816f-e78c2b7aa291-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f3ce657a-287c-47c7-816f-e78c2b7aa291" (UID: "f3ce657a-287c-47c7-816f-e78c2b7aa291"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:32:59.468109 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:59.468092 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3ce657a-287c-47c7-816f-e78c2b7aa291-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f3ce657a-287c-47c7-816f-e78c2b7aa291" (UID: "f3ce657a-287c-47c7-816f-e78c2b7aa291"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:32:59.468254 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:59.468137 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3ce657a-287c-47c7-816f-e78c2b7aa291-kube-api-access-96bcq" (OuterVolumeSpecName: "kube-api-access-96bcq") pod "f3ce657a-287c-47c7-816f-e78c2b7aa291" (UID: "f3ce657a-287c-47c7-816f-e78c2b7aa291"). InnerVolumeSpecName "kube-api-access-96bcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:32:59.565892 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:59.565860 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3ce657a-287c-47c7-816f-e78c2b7aa291-maas-api-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:32:59.565892 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:59.565889 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3ce657a-287c-47c7-816f-e78c2b7aa291-metrics-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:32:59.565892 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:32:59.565899 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96bcq\" (UniqueName: \"kubernetes.io/projected/f3ce657a-287c-47c7-816f-e78c2b7aa291-kube-api-access-96bcq\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:33:00.018522 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:00.018485 2564 generic.go:356] "Generic (PLEG): container finished" podID="f3ce657a-287c-47c7-816f-e78c2b7aa291" containerID="923b5971e1bde00f0b15f81f2646b0df50abea63a234ebc700561093f9d283d7" exitCode=0 Sep 14 13:33:00.018698 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:00.018540 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" Sep 14 13:33:00.018698 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:00.018575 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" event={"ID":"f3ce657a-287c-47c7-816f-e78c2b7aa291","Type":"ContainerDied","Data":"923b5971e1bde00f0b15f81f2646b0df50abea63a234ebc700561093f9d283d7"} Sep 14 13:33:00.018698 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:00.018630 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6" event={"ID":"f3ce657a-287c-47c7-816f-e78c2b7aa291","Type":"ContainerDied","Data":"3fee80659aa385a5ae7515e9066a7f4a91fb12d4c29cb88c5bc0edfa245ab36e"} Sep 14 13:33:00.018698 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:00.018653 2564 scope.go:117] "RemoveContainer" containerID="923b5971e1bde00f0b15f81f2646b0df50abea63a234ebc700561093f9d283d7" Sep 14 13:33:00.028292 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:00.028273 2564 scope.go:117] "RemoveContainer" containerID="923b5971e1bde00f0b15f81f2646b0df50abea63a234ebc700561093f9d283d7" Sep 14 13:33:00.028573 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:33:00.028555 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"923b5971e1bde00f0b15f81f2646b0df50abea63a234ebc700561093f9d283d7\": container with ID starting with 923b5971e1bde00f0b15f81f2646b0df50abea63a234ebc700561093f9d283d7 not found: ID does not exist" containerID="923b5971e1bde00f0b15f81f2646b0df50abea63a234ebc700561093f9d283d7" Sep 14 13:33:00.028625 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:00.028581 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"923b5971e1bde00f0b15f81f2646b0df50abea63a234ebc700561093f9d283d7"} err="failed to get container status \"923b5971e1bde00f0b15f81f2646b0df50abea63a234ebc700561093f9d283d7\": rpc error: code = NotFound desc = could not find container \"923b5971e1bde00f0b15f81f2646b0df50abea63a234ebc700561093f9d283d7\": container with ID starting with 923b5971e1bde00f0b15f81f2646b0df50abea63a234ebc700561093f9d283d7 not found: ID does not exist" Sep 14 13:33:00.036364 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:00.036339 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6"] Sep 14 13:33:00.038795 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:00.038772 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c87a95-5858c8bff-879k6"] Sep 14 13:33:01.812071 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:01.812039 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3ce657a-287c-47c7-816f-e78c2b7aa291" path="/var/lib/kubelet/pods/f3ce657a-287c-47c7-816f-e78c2b7aa291/volumes" Sep 14 13:33:20.019991 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:20.019961 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/1.log" Sep 14 13:33:20.034914 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:20.034885 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/1.log" Sep 14 13:33:59.007888 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.007850 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc"] Sep 14 13:33:59.008649 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.008108 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" podUID="bbfb3b76-47b2-49d9-aed2-ad99befab7df" containerName="maas-api" containerID="cri-o://012440aeb29751ce4add5b856190e19b55bb7b1d45bae574affc08f8a7091dbd" gracePeriod=30 Sep 14 13:33:59.253243 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.253220 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" Sep 14 13:33:59.270144 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.270082 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfb3b76-47b2-49d9-aed2-ad99befab7df-metrics-certs\") pod \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\" (UID: \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\") " Sep 14 13:33:59.270267 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.270146 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbfb3b76-47b2-49d9-aed2-ad99befab7df-openshift-service-ca\") pod \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\" (UID: \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\") " Sep 14 13:33:59.270267 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.270189 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbfb3b76-47b2-49d9-aed2-ad99befab7df-maas-api-tls\") pod \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\" (UID: \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\") " Sep 14 13:33:59.270384 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.270316 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79js6\" (UniqueName: \"kubernetes.io/projected/bbfb3b76-47b2-49d9-aed2-ad99befab7df-kube-api-access-79js6\") pod \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\" (UID: \"bbfb3b76-47b2-49d9-aed2-ad99befab7df\") " Sep 14 13:33:59.270524 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.270502 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbfb3b76-47b2-49d9-aed2-ad99befab7df-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bbfb3b76-47b2-49d9-aed2-ad99befab7df" (UID: "bbfb3b76-47b2-49d9-aed2-ad99befab7df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:33:59.270629 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.270608 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbfb3b76-47b2-49d9-aed2-ad99befab7df-openshift-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:33:59.272449 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.272415 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbfb3b76-47b2-49d9-aed2-ad99befab7df-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bbfb3b76-47b2-49d9-aed2-ad99befab7df" (UID: "bbfb3b76-47b2-49d9-aed2-ad99befab7df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:33:59.272556 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.272441 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbfb3b76-47b2-49d9-aed2-ad99befab7df-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bbfb3b76-47b2-49d9-aed2-ad99befab7df" (UID: "bbfb3b76-47b2-49d9-aed2-ad99befab7df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:33:59.273122 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.272726 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbfb3b76-47b2-49d9-aed2-ad99befab7df-kube-api-access-79js6" (OuterVolumeSpecName: "kube-api-access-79js6") pod "bbfb3b76-47b2-49d9-aed2-ad99befab7df" (UID: "bbfb3b76-47b2-49d9-aed2-ad99befab7df"). InnerVolumeSpecName "kube-api-access-79js6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:33:59.359384 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.359346 2564 generic.go:356] "Generic (PLEG): container finished" podID="bbfb3b76-47b2-49d9-aed2-ad99befab7df" containerID="012440aeb29751ce4add5b856190e19b55bb7b1d45bae574affc08f8a7091dbd" exitCode=0 Sep 14 13:33:59.359541 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.359400 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" Sep 14 13:33:59.359541 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.359396 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" event={"ID":"bbfb3b76-47b2-49d9-aed2-ad99befab7df","Type":"ContainerDied","Data":"012440aeb29751ce4add5b856190e19b55bb7b1d45bae574affc08f8a7091dbd"} Sep 14 13:33:59.359541 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.359456 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc" event={"ID":"bbfb3b76-47b2-49d9-aed2-ad99befab7df","Type":"ContainerDied","Data":"a9470973002dd419d19b88266b4fb0b82f1bfe50918098e90cce8552e6ccc461"} Sep 14 13:33:59.359541 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.359480 2564 scope.go:117] "RemoveContainer" containerID="012440aeb29751ce4add5b856190e19b55bb7b1d45bae574affc08f8a7091dbd" Sep 14 13:33:59.371747 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.371720 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bbfb3b76-47b2-49d9-aed2-ad99befab7df-maas-api-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:33:59.371874 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.371749 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79js6\" (UniqueName: \"kubernetes.io/projected/bbfb3b76-47b2-49d9-aed2-ad99befab7df-kube-api-access-79js6\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:33:59.371874 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.371761 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbfb3b76-47b2-49d9-aed2-ad99befab7df-metrics-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:33:59.376768 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.376748 2564 scope.go:117] "RemoveContainer" containerID="012440aeb29751ce4add5b856190e19b55bb7b1d45bae574affc08f8a7091dbd" Sep 14 13:33:59.377144 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:33:59.377122 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"012440aeb29751ce4add5b856190e19b55bb7b1d45bae574affc08f8a7091dbd\": container with ID starting with 012440aeb29751ce4add5b856190e19b55bb7b1d45bae574affc08f8a7091dbd not found: ID does not exist" containerID="012440aeb29751ce4add5b856190e19b55bb7b1d45bae574affc08f8a7091dbd" Sep 14 13:33:59.377248 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.377152 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"012440aeb29751ce4add5b856190e19b55bb7b1d45bae574affc08f8a7091dbd"} err="failed to get container status \"012440aeb29751ce4add5b856190e19b55bb7b1d45bae574affc08f8a7091dbd\": rpc error: code = NotFound desc = could not find container \"012440aeb29751ce4add5b856190e19b55bb7b1d45bae574affc08f8a7091dbd\": container with ID starting with 012440aeb29751ce4add5b856190e19b55bb7b1d45bae574affc08f8a7091dbd not found: ID does not exist" Sep 14 13:33:59.389658 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.389581 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc"] Sep 14 13:33:59.394692 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.394663 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-27ae76-d574597c-hnpnc"] Sep 14 13:33:59.812991 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:33:59.812960 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbfb3b76-47b2-49d9-aed2-ad99befab7df" path="/var/lib/kubelet/pods/bbfb3b76-47b2-49d9-aed2-ad99befab7df/volumes" Sep 14 13:34:34.776032 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:34.776000 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt"] Sep 14 13:34:34.776622 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:34.776217 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" podUID="a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea" containerName="maas-api" containerID="cri-o://74a9f28fdac065f54bf7f3ec95dc79efecf828cc4f11ec643924d8b27fff1035" gracePeriod=30 Sep 14 13:34:35.024530 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.024501 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" Sep 14 13:34:35.121947 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.121909 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sj7d\" (UniqueName: \"kubernetes.io/projected/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-kube-api-access-6sj7d\") pod \"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea\" (UID: \"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea\") " Sep 14 13:34:35.122111 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.122019 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-openshift-service-ca\") pod \"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea\" (UID: \"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea\") " Sep 14 13:34:35.122111 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.122084 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-metrics-certs\") pod \"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea\" (UID: \"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea\") " Sep 14 13:34:35.122186 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.122110 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-maas-api-tls\") pod \"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea\" (UID: \"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea\") " Sep 14 13:34:35.122470 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.122439 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea" (UID: "a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:34:35.124377 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.124347 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea" (UID: "a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:34:35.124377 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.124372 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea" (UID: "a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:34:35.124520 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.124475 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-kube-api-access-6sj7d" (OuterVolumeSpecName: "kube-api-access-6sj7d") pod "a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea" (UID: "a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea"). InnerVolumeSpecName "kube-api-access-6sj7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:34:35.223438 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.223402 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sj7d\" (UniqueName: \"kubernetes.io/projected/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-kube-api-access-6sj7d\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:34:35.223438 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.223433 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-openshift-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:34:35.223438 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.223443 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-metrics-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:34:35.223672 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.223451 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea-maas-api-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:34:35.569758 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.569666 2564 generic.go:356] "Generic (PLEG): container finished" podID="a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea" containerID="74a9f28fdac065f54bf7f3ec95dc79efecf828cc4f11ec643924d8b27fff1035" exitCode=0 Sep 14 13:34:35.569758 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.569709 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" Sep 14 13:34:35.569758 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.569744 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" event={"ID":"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea","Type":"ContainerDied","Data":"74a9f28fdac065f54bf7f3ec95dc79efecf828cc4f11ec643924d8b27fff1035"} Sep 14 13:34:35.570028 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.569798 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt" event={"ID":"a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea","Type":"ContainerDied","Data":"1c0542bf036d6c4aa518ef05f0d6a3b8663f609e6fdb8258be58f12a3bef6a96"} Sep 14 13:34:35.570028 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.569821 2564 scope.go:117] "RemoveContainer" containerID="74a9f28fdac065f54bf7f3ec95dc79efecf828cc4f11ec643924d8b27fff1035" Sep 14 13:34:35.579752 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.579733 2564 scope.go:117] "RemoveContainer" containerID="74a9f28fdac065f54bf7f3ec95dc79efecf828cc4f11ec643924d8b27fff1035" Sep 14 13:34:35.580063 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:34:35.580045 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a9f28fdac065f54bf7f3ec95dc79efecf828cc4f11ec643924d8b27fff1035\": container with ID starting with 74a9f28fdac065f54bf7f3ec95dc79efecf828cc4f11ec643924d8b27fff1035 not found: ID does not exist" containerID="74a9f28fdac065f54bf7f3ec95dc79efecf828cc4f11ec643924d8b27fff1035" Sep 14 13:34:35.580133 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.580071 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a9f28fdac065f54bf7f3ec95dc79efecf828cc4f11ec643924d8b27fff1035"} err="failed to get container status \"74a9f28fdac065f54bf7f3ec95dc79efecf828cc4f11ec643924d8b27fff1035\": rpc error: code = NotFound desc = could not find container \"74a9f28fdac065f54bf7f3ec95dc79efecf828cc4f11ec643924d8b27fff1035\": container with ID starting with 74a9f28fdac065f54bf7f3ec95dc79efecf828cc4f11ec643924d8b27fff1035 not found: ID does not exist" Sep 14 13:34:35.592573 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.592555 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt"] Sep 14 13:34:35.594887 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.594866 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eff5e8-58589c6c75-6trvt"] Sep 14 13:34:35.811488 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:35.811455 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea" path="/var/lib/kubelet/pods/a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea/volumes" Sep 14 13:34:41.502581 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:41.502548 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm"] Sep 14 13:34:41.503119 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:41.502773 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" podUID="e444894f-628c-4794-8a59-886c3ccb0674" containerName="istio-proxy" containerID="cri-o://522f61cf898600564adea088a800d0b2ef794d41916f4f858d0caff5b9c17691" gracePeriod=30 Sep 14 13:34:46.638249 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.638215 2564 generic.go:356] "Generic (PLEG): container finished" podID="e444894f-628c-4794-8a59-886c3ccb0674" containerID="522f61cf898600564adea088a800d0b2ef794d41916f4f858d0caff5b9c17691" exitCode=0 Sep 14 13:34:46.638728 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.638290 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" event={"ID":"e444894f-628c-4794-8a59-886c3ccb0674","Type":"ContainerDied","Data":"522f61cf898600564adea088a800d0b2ef794d41916f4f858d0caff5b9c17691"} Sep 14 13:34:46.638728 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.638374 2564 scope.go:117] "RemoveContainer" containerID="3f5115815e08ffff87641c2bfcb44bcc0210afcb70addd06c04785f3560773a8" Sep 14 13:34:46.763182 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.763159 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:34:46.826523 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.826444 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-workload-certs\") pod \"e444894f-628c-4794-8a59-886c3ccb0674\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " Sep 14 13:34:46.826523 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.826486 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2n78\" (UniqueName: \"kubernetes.io/projected/e444894f-628c-4794-8a59-886c3ccb0674-kube-api-access-j2n78\") pod \"e444894f-628c-4794-8a59-886c3ccb0674\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " Sep 14 13:34:46.826780 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.826756 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e444894f-628c-4794-8a59-886c3ccb0674" (UID: "e444894f-628c-4794-8a59-886c3ccb0674"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:34:46.828853 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.828825 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e444894f-628c-4794-8a59-886c3ccb0674-kube-api-access-j2n78" (OuterVolumeSpecName: "kube-api-access-j2n78") pod "e444894f-628c-4794-8a59-886c3ccb0674" (UID: "e444894f-628c-4794-8a59-886c3ccb0674"). InnerVolumeSpecName "kube-api-access-j2n78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:34:46.926943 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.926888 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e444894f-628c-4794-8a59-886c3ccb0674-istio-podinfo\") pod \"e444894f-628c-4794-8a59-886c3ccb0674\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " Sep 14 13:34:46.926943 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.926941 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e444894f-628c-4794-8a59-886c3ccb0674-istio-token\") pod \"e444894f-628c-4794-8a59-886c3ccb0674\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " Sep 14 13:34:46.927174 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.926981 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-workload-socket\") pod \"e444894f-628c-4794-8a59-886c3ccb0674\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " Sep 14 13:34:46.927174 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.927018 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-credential-socket\") pod \"e444894f-628c-4794-8a59-886c3ccb0674\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " Sep 14 13:34:46.927174 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.927065 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-istio-envoy\") pod \"e444894f-628c-4794-8a59-886c3ccb0674\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " Sep 14 13:34:46.927174 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.927082 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e444894f-628c-4794-8a59-886c3ccb0674-istiod-ca-cert\") pod \"e444894f-628c-4794-8a59-886c3ccb0674\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " Sep 14 13:34:46.927174 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.927126 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-istio-data\") pod \"e444894f-628c-4794-8a59-886c3ccb0674\" (UID: \"e444894f-628c-4794-8a59-886c3ccb0674\") " Sep 14 13:34:46.927438 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.927288 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e444894f-628c-4794-8a59-886c3ccb0674" (UID: "e444894f-628c-4794-8a59-886c3ccb0674"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:34:46.927438 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.927309 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e444894f-628c-4794-8a59-886c3ccb0674" (UID: "e444894f-628c-4794-8a59-886c3ccb0674"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:34:46.927557 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.927487 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-credential-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:34:46.927557 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.927497 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-istio-data" (OuterVolumeSpecName: "istio-data") pod "e444894f-628c-4794-8a59-886c3ccb0674" (UID: "e444894f-628c-4794-8a59-886c3ccb0674"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:34:46.927557 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.927504 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-workload-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:34:46.927715 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.927558 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2n78\" (UniqueName: \"kubernetes.io/projected/e444894f-628c-4794-8a59-886c3ccb0674-kube-api-access-j2n78\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:34:46.927715 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.927574 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-workload-socket\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:34:46.927715 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.927607 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e444894f-628c-4794-8a59-886c3ccb0674-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e444894f-628c-4794-8a59-886c3ccb0674" (UID: "e444894f-628c-4794-8a59-886c3ccb0674"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:34:46.929278 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.929243 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e444894f-628c-4794-8a59-886c3ccb0674-istio-token" (OuterVolumeSpecName: "istio-token") pod "e444894f-628c-4794-8a59-886c3ccb0674" (UID: "e444894f-628c-4794-8a59-886c3ccb0674"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:34:46.929386 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.929324 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e444894f-628c-4794-8a59-886c3ccb0674" (UID: "e444894f-628c-4794-8a59-886c3ccb0674"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 14 13:34:46.929648 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:46.929625 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e444894f-628c-4794-8a59-886c3ccb0674-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e444894f-628c-4794-8a59-886c3ccb0674" (UID: "e444894f-628c-4794-8a59-886c3ccb0674"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 14 13:34:47.028078 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:47.028050 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e444894f-628c-4794-8a59-886c3ccb0674-istiod-ca-cert\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:34:47.028078 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:47.028074 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-istio-data\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:34:47.028078 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:47.028082 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e444894f-628c-4794-8a59-886c3ccb0674-istio-podinfo\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:34:47.028302 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:47.028090 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e444894f-628c-4794-8a59-886c3ccb0674-istio-token\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:34:47.028302 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:47.028097 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e444894f-628c-4794-8a59-886c3ccb0674-istio-envoy\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:34:47.645845 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:47.645815 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" Sep 14 13:34:47.646283 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:47.645812 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm" event={"ID":"e444894f-628c-4794-8a59-886c3ccb0674","Type":"ContainerDied","Data":"f01f80f429c42206dab43f485c18e6be7d280706d87491d7646ca5085278b685"} Sep 14 13:34:47.646283 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:47.645972 2564 scope.go:117] "RemoveContainer" containerID="522f61cf898600564adea088a800d0b2ef794d41916f4f858d0caff5b9c17691" Sep 14 13:34:47.669271 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:47.669217 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm"] Sep 14 13:34:47.672290 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:47.672267 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-eff5e8-openshift-default-bfb7c786d-vwwqm"] Sep 14 13:34:47.812992 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:34:47.812960 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e444894f-628c-4794-8a59-886c3ccb0674" path="/var/lib/kubelet/pods/e444894f-628c-4794-8a59-886c3ccb0674/volumes" Sep 14 13:38:20.068426 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:38:20.068399 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/1.log" Sep 14 13:38:20.079394 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:38:20.079372 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/1.log" Sep 14 13:40:23.047508 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.047412 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg"] Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.047829 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e444894f-628c-4794-8a59-886c3ccb0674" containerName="istio-proxy" Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.047840 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e444894f-628c-4794-8a59-886c3ccb0674" containerName="istio-proxy" Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.047848 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea" containerName="maas-api" Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.047853 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea" containerName="maas-api" Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.047862 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e444894f-628c-4794-8a59-886c3ccb0674" containerName="istio-proxy" Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.047867 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e444894f-628c-4794-8a59-886c3ccb0674" containerName="istio-proxy" Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.047875 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbfb3b76-47b2-49d9-aed2-ad99befab7df" containerName="maas-api" Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.047880 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbfb3b76-47b2-49d9-aed2-ad99befab7df" containerName="maas-api" Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.047891 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3ce657a-287c-47c7-816f-e78c2b7aa291" containerName="maas-api" Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.047896 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3ce657a-287c-47c7-816f-e78c2b7aa291" containerName="maas-api" Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.047902 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b5a7b78-5ca3-427d-b66d-be84180b6dd1" containerName="istio-proxy" Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.047907 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b5a7b78-5ca3-427d-b66d-be84180b6dd1" containerName="istio-proxy" Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.047916 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b5a7b78-5ca3-427d-b66d-be84180b6dd1" containerName="istio-proxy" Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.047941 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b5a7b78-5ca3-427d-b66d-be84180b6dd1" containerName="istio-proxy" Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.048018 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbfb3b76-47b2-49d9-aed2-ad99befab7df" containerName="maas-api" Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.048027 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b5a7b78-5ca3-427d-b66d-be84180b6dd1" containerName="istio-proxy" Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.048033 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0c0a848-fb3c-4b6b-94c5-eb0b7f249bea" containerName="maas-api" Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.048039 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e444894f-628c-4794-8a59-886c3ccb0674" containerName="istio-proxy" Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.048044 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3ce657a-287c-47c7-816f-e78c2b7aa291" containerName="maas-api" Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.048049 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e444894f-628c-4794-8a59-886c3ccb0674" containerName="istio-proxy" Sep 14 13:40:23.048136 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.048056 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b5a7b78-5ca3-427d-b66d-be84180b6dd1" containerName="istio-proxy" Sep 14 13:40:23.051251 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.051230 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg" Sep 14 13:40:23.060729 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.060704 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg"] Sep 14 13:40:23.116308 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.116270 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6289a622-f554-4e21-a783-6436d3f56144-openshift-service-ca\") pod \"maas-api-5df7d7c794-zc5cg\" (UID: \"6289a622-f554-4e21-a783-6436d3f56144\") " pod="odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg" Sep 14 13:40:23.116480 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.116345 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mfr9\" (UniqueName: \"kubernetes.io/projected/6289a622-f554-4e21-a783-6436d3f56144-kube-api-access-5mfr9\") pod \"maas-api-5df7d7c794-zc5cg\" (UID: \"6289a622-f554-4e21-a783-6436d3f56144\") " pod="odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg" Sep 14 13:40:23.116480 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.116363 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6289a622-f554-4e21-a783-6436d3f56144-metrics-certs\") pod \"maas-api-5df7d7c794-zc5cg\" (UID: \"6289a622-f554-4e21-a783-6436d3f56144\") " pod="odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg" Sep 14 13:40:23.116480 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.116386 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6289a622-f554-4e21-a783-6436d3f56144-maas-api-tls\") pod \"maas-api-5df7d7c794-zc5cg\" (UID: \"6289a622-f554-4e21-a783-6436d3f56144\") " pod="odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg" Sep 14 13:40:23.217569 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.217534 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6289a622-f554-4e21-a783-6436d3f56144-maas-api-tls\") pod \"maas-api-5df7d7c794-zc5cg\" (UID: \"6289a622-f554-4e21-a783-6436d3f56144\") " pod="odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg" Sep 14 13:40:23.217746 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.217613 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6289a622-f554-4e21-a783-6436d3f56144-openshift-service-ca\") pod \"maas-api-5df7d7c794-zc5cg\" (UID: \"6289a622-f554-4e21-a783-6436d3f56144\") " pod="odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg" Sep 14 13:40:23.217746 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.217660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mfr9\" (UniqueName: \"kubernetes.io/projected/6289a622-f554-4e21-a783-6436d3f56144-kube-api-access-5mfr9\") pod \"maas-api-5df7d7c794-zc5cg\" (UID: \"6289a622-f554-4e21-a783-6436d3f56144\") " pod="odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg" Sep 14 13:40:23.217746 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.217677 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6289a622-f554-4e21-a783-6436d3f56144-metrics-certs\") pod \"maas-api-5df7d7c794-zc5cg\" (UID: \"6289a622-f554-4e21-a783-6436d3f56144\") " pod="odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg" Sep 14 13:40:23.218314 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.218288 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6289a622-f554-4e21-a783-6436d3f56144-openshift-service-ca\") pod \"maas-api-5df7d7c794-zc5cg\" (UID: \"6289a622-f554-4e21-a783-6436d3f56144\") " pod="odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg" Sep 14 13:40:23.220221 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.220198 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6289a622-f554-4e21-a783-6436d3f56144-metrics-certs\") pod \"maas-api-5df7d7c794-zc5cg\" (UID: \"6289a622-f554-4e21-a783-6436d3f56144\") " pod="odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg" Sep 14 13:40:23.220464 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.220441 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6289a622-f554-4e21-a783-6436d3f56144-maas-api-tls\") pod \"maas-api-5df7d7c794-zc5cg\" (UID: \"6289a622-f554-4e21-a783-6436d3f56144\") " pod="odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg" Sep 14 13:40:23.226818 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.226794 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mfr9\" (UniqueName: \"kubernetes.io/projected/6289a622-f554-4e21-a783-6436d3f56144-kube-api-access-5mfr9\") pod \"maas-api-5df7d7c794-zc5cg\" (UID: \"6289a622-f554-4e21-a783-6436d3f56144\") " pod="odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg" Sep 14 13:40:23.363946 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.363909 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg" Sep 14 13:40:23.680014 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.679990 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg"] Sep 14 13:40:23.681892 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:40:23.681869 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6289a622_f554_4e21_a783_6436d3f56144.slice/crio-c0c528cc10b5a40bdf219f36a8f4a58b643f595d2941b6be239d898fe73508f0 WatchSource:0}: Error finding container c0c528cc10b5a40bdf219f36a8f4a58b643f595d2941b6be239d898fe73508f0: Status 404 returned error can't find the container with id c0c528cc10b5a40bdf219f36a8f4a58b643f595d2941b6be239d898fe73508f0 Sep 14 13:40:23.683231 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:23.683209 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 14 13:40:24.497799 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:24.497763 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg" event={"ID":"6289a622-f554-4e21-a783-6436d3f56144","Type":"ContainerStarted","Data":"fcf070c3af36c8f74cb25af7a527eaa9ca897be1c1875f5bf44d7ff2f7a90914"} Sep 14 13:40:24.498255 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:24.497810 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg" event={"ID":"6289a622-f554-4e21-a783-6436d3f56144","Type":"ContainerStarted","Data":"c0c528cc10b5a40bdf219f36a8f4a58b643f595d2941b6be239d898fe73508f0"} Sep 14 13:40:24.498255 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:24.497849 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg" Sep 14 13:40:24.517030 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:24.516988 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg" podStartSLOduration=1.109656044 podStartE2EDuration="1.516971721s" totalImagesPullingTime="407.315677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-14 13:40:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:40:24.51441844 +0000 UTC m=+2525.177314067" watchObservedRunningTime="2026-09-14 13:40:24.516971721 +0000 UTC m=+2525.179867352" Sep 14 13:40:30.508619 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:30.508588 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5df7d7c794-zc5cg" Sep 14 13:40:38.088454 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:38.088416 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf"] Sep 14 13:40:38.088900 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:38.088703 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" podUID="03c18206-36fd-43ca-93e5-866794d9360d" containerName="maas-api" containerID="cri-o://4ec72721733e05ddd2ab433764c40de57e8efeb99a4fe5103f830197209e08e9" gracePeriod=30 Sep 14 13:40:39.339769 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.339739 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" Sep 14 13:40:39.477679 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.477579 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c18206-36fd-43ca-93e5-866794d9360d-openshift-service-ca\") pod \"03c18206-36fd-43ca-93e5-866794d9360d\" (UID: \"03c18206-36fd-43ca-93e5-866794d9360d\") " Sep 14 13:40:39.477860 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.477683 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9w9mx\" (UniqueName: \"kubernetes.io/projected/03c18206-36fd-43ca-93e5-866794d9360d-kube-api-access-9w9mx\") pod \"03c18206-36fd-43ca-93e5-866794d9360d\" (UID: \"03c18206-36fd-43ca-93e5-866794d9360d\") " Sep 14 13:40:39.477860 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.477713 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03c18206-36fd-43ca-93e5-866794d9360d-maas-api-tls\") pod \"03c18206-36fd-43ca-93e5-866794d9360d\" (UID: \"03c18206-36fd-43ca-93e5-866794d9360d\") " Sep 14 13:40:39.477860 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.477788 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c18206-36fd-43ca-93e5-866794d9360d-metrics-certs\") pod \"03c18206-36fd-43ca-93e5-866794d9360d\" (UID: \"03c18206-36fd-43ca-93e5-866794d9360d\") " Sep 14 13:40:39.478062 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.478004 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c18206-36fd-43ca-93e5-866794d9360d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "03c18206-36fd-43ca-93e5-866794d9360d" (UID: "03c18206-36fd-43ca-93e5-866794d9360d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 14 13:40:39.478174 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.478153 2564 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03c18206-36fd-43ca-93e5-866794d9360d-openshift-service-ca\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:40:39.480020 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.479998 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03c18206-36fd-43ca-93e5-866794d9360d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "03c18206-36fd-43ca-93e5-866794d9360d" (UID: "03c18206-36fd-43ca-93e5-866794d9360d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:40:39.480115 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.480011 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c18206-36fd-43ca-93e5-866794d9360d-kube-api-access-9w9mx" (OuterVolumeSpecName: "kube-api-access-9w9mx") pod "03c18206-36fd-43ca-93e5-866794d9360d" (UID: "03c18206-36fd-43ca-93e5-866794d9360d"). InnerVolumeSpecName "kube-api-access-9w9mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 14 13:40:39.480175 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.480129 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03c18206-36fd-43ca-93e5-866794d9360d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "03c18206-36fd-43ca-93e5-866794d9360d" (UID: "03c18206-36fd-43ca-93e5-866794d9360d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 14 13:40:39.579289 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.579257 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c18206-36fd-43ca-93e5-866794d9360d-metrics-certs\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:40:39.579289 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.579282 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w9mx\" (UniqueName: \"kubernetes.io/projected/03c18206-36fd-43ca-93e5-866794d9360d-kube-api-access-9w9mx\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:40:39.579289 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.579291 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03c18206-36fd-43ca-93e5-866794d9360d-maas-api-tls\") on node \"ip-10-0-136-96.ec2.internal\" DevicePath \"\"" Sep 14 13:40:39.583947 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.583903 2564 generic.go:356] "Generic (PLEG): container finished" podID="03c18206-36fd-43ca-93e5-866794d9360d" containerID="4ec72721733e05ddd2ab433764c40de57e8efeb99a4fe5103f830197209e08e9" exitCode=0 Sep 14 13:40:39.584099 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.583969 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" event={"ID":"03c18206-36fd-43ca-93e5-866794d9360d","Type":"ContainerDied","Data":"4ec72721733e05ddd2ab433764c40de57e8efeb99a4fe5103f830197209e08e9"} Sep 14 13:40:39.584099 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.584025 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" event={"ID":"03c18206-36fd-43ca-93e5-866794d9360d","Type":"ContainerDied","Data":"ada5b1fb3604a829619e92612a2f8b682437b9b4d2b0a1ebd7dee52d3acacecd"} Sep 14 13:40:39.584099 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.584046 2564 scope.go:117] "RemoveContainer" containerID="4ec72721733e05ddd2ab433764c40de57e8efeb99a4fe5103f830197209e08e9" Sep 14 13:40:39.584099 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.583981 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf" Sep 14 13:40:39.594433 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.594414 2564 scope.go:117] "RemoveContainer" containerID="4ec72721733e05ddd2ab433764c40de57e8efeb99a4fe5103f830197209e08e9" Sep 14 13:40:39.594717 ip-10-0-136-96 kubenswrapper[2564]: E0914 13:40:39.594698 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec72721733e05ddd2ab433764c40de57e8efeb99a4fe5103f830197209e08e9\": container with ID starting with 4ec72721733e05ddd2ab433764c40de57e8efeb99a4fe5103f830197209e08e9 not found: ID does not exist" containerID="4ec72721733e05ddd2ab433764c40de57e8efeb99a4fe5103f830197209e08e9" Sep 14 13:40:39.594780 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.594730 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec72721733e05ddd2ab433764c40de57e8efeb99a4fe5103f830197209e08e9"} err="failed to get container status \"4ec72721733e05ddd2ab433764c40de57e8efeb99a4fe5103f830197209e08e9\": rpc error: code = NotFound desc = could not find container \"4ec72721733e05ddd2ab433764c40de57e8efeb99a4fe5103f830197209e08e9\": container with ID starting with 4ec72721733e05ddd2ab433764c40de57e8efeb99a4fe5103f830197209e08e9 not found: ID does not exist" Sep 14 13:40:39.608315 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.608281 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf"] Sep 14 13:40:39.610796 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.610772 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-577bb5997d-9mnnf"] Sep 14 13:40:39.814004 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:39.813889 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03c18206-36fd-43ca-93e5-866794d9360d" path="/var/lib/kubelet/pods/03c18206-36fd-43ca-93e5-866794d9360d/volumes" Sep 14 13:40:58.406229 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:58.406191 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rwxn9_6ee95cc6-a9d1-426d-b595-b83cc025638d/manager/0.log" Sep 14 13:40:58.517763 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:58.517735 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-697cf97c89-dwk8r_3d67c169-7edc-46cb-ba89-b753cb011b73/copy-manifests/0.log" Sep 14 13:40:58.525855 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:58.525826 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-697cf97c89-dwk8r_3d67c169-7edc-46cb-ba89-b753cb011b73/manager/0.log" Sep 14 13:40:59.223451 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:40:59.223415 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5df7d7c794-zc5cg_6289a622-f554-4e21-a783-6436d3f56144/maas-api/0.log" Sep 14 13:41:01.315050 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:01.315014 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz_bbf1706b-9e71-41e5-82ba-8b1e58997b29/istio-proxy/0.log" Sep 14 13:41:01.440780 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:01.440728 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9v8ts_f9c24a09-932f-41c9-8391-16b38a0d7ee6/discovery/0.log" Sep 14 13:41:02.112476 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:02.112433 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64988bbc7d-tnjdw_0287d99a-ca2c-45bc-8f32-a2a273401e51/router/0.log" Sep 14 13:41:02.553476 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:02.553443 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-v5wj8_f6bb71f4-9da1-40e0-8e6c-c3bcdd7924be/main/0.log" Sep 14 13:41:02.666622 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:02.666593 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-zfx95_f9fedfc1-1cbc-4c41-b99f-1f7ae335771d/main/0.log" Sep 14 13:41:02.989950 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:02.989906 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-9tf5d_c6df2760-f30e-4166-81fb-9351aa871b6e/main/0.log" Sep 14 13:41:11.278201 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:11.278176 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-72fhn_d49b2943-be96-4d9a-ab55-965845ba03f9/global-pull-secret-syncer/0.log" Sep 14 13:41:11.472790 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:11.472762 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-slcjt_00612ad7-80a8-41a4-90fd-28e632d42e8d/konnectivity-agent/0.log" Sep 14 13:41:11.539530 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:11.539445 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-96.ec2.internal_bd507eff9b3716da03d385309d794351/haproxy/0.log" Sep 14 13:41:17.738823 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:17.738791 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_74b140d0-ffc6-4819-b176-2e0204ef7397/alertmanager/0.log" Sep 14 13:41:17.761171 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:17.761142 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_74b140d0-ffc6-4819-b176-2e0204ef7397/config-reloader/0.log" Sep 14 13:41:17.781939 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:17.781901 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_74b140d0-ffc6-4819-b176-2e0204ef7397/kube-rbac-proxy-web/0.log" Sep 14 13:41:17.803087 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:17.803064 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_74b140d0-ffc6-4819-b176-2e0204ef7397/kube-rbac-proxy/0.log" Sep 14 13:41:17.824210 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:17.824180 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_74b140d0-ffc6-4819-b176-2e0204ef7397/kube-rbac-proxy-metric/0.log" Sep 14 13:41:17.842300 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:17.842272 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_74b140d0-ffc6-4819-b176-2e0204ef7397/prom-label-proxy/0.log" Sep 14 13:41:17.862537 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:17.862512 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_74b140d0-ffc6-4819-b176-2e0204ef7397/init-config-reloader/0.log" Sep 14 13:41:17.915591 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:17.915564 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-b9nh8_547938cd-bd0a-42d0-ac25-9406cc46fe95/cluster-monitoring-operator/0.log" Sep 14 13:41:18.002015 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:18.001914 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-88bd8b6b4-bf4dh_aea629de-fae3-4027-9931-197ffd9e91dc/metrics-server/0.log" Sep 14 13:41:18.038713 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:18.038689 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-j4x9t_69d15a6f-cd51-456e-a72c-5b62e13720c8/monitoring-plugin/0.log" Sep 14 13:41:18.290739 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:18.290667 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mgbcd_4b68906a-2c7d-44a3-837c-d62d9a661c79/node-exporter/0.log" Sep 14 13:41:18.323355 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:18.323328 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mgbcd_4b68906a-2c7d-44a3-837c-d62d9a661c79/kube-rbac-proxy/0.log" Sep 14 13:41:18.337276 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:18.337257 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mgbcd_4b68906a-2c7d-44a3-837c-d62d9a661c79/init-textfile/0.log" Sep 14 13:41:18.654081 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:18.654053 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76b5648bd9-6bgpj_15c77fd6-d40c-44e3-be94-2ec048db6d17/telemeter-client/0.log" Sep 14 13:41:18.680147 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:18.680118 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76b5648bd9-6bgpj_15c77fd6-d40c-44e3-be94-2ec048db6d17/reload/0.log" Sep 14 13:41:18.697452 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:18.697424 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76b5648bd9-6bgpj_15c77fd6-d40c-44e3-be94-2ec048db6d17/kube-rbac-proxy/0.log" Sep 14 13:41:20.311280 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:20.311250 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-nn8d7_1d79bb2f-3305-4485-a140-a2f5c5e1b009/networking-console-plugin/0.log" Sep 14 13:41:20.908216 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:20.908187 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/1.log" Sep 14 13:41:20.917394 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:20.917370 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-2995b_fdc65821-255d-43be-9ccc-7dffc5a49791/console-operator/2.log" Sep 14 13:41:21.442588 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.442547 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2"] Sep 14 13:41:21.443012 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.442989 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03c18206-36fd-43ca-93e5-866794d9360d" containerName="maas-api" Sep 14 13:41:21.443012 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.443002 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="03c18206-36fd-43ca-93e5-866794d9360d" containerName="maas-api" Sep 14 13:41:21.443096 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.443074 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="03c18206-36fd-43ca-93e5-866794d9360d" containerName="maas-api" Sep 14 13:41:21.447304 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.447287 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" Sep 14 13:41:21.450025 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.450003 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2dnwm\"/\"openshift-service-ca.crt\"" Sep 14 13:41:21.450119 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.450024 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2dnwm\"/\"default-dockercfg-5mfhs\"" Sep 14 13:41:21.451055 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.451036 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2dnwm\"/\"kube-root-ca.crt\"" Sep 14 13:41:21.454818 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.454797 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2"] Sep 14 13:41:21.482978 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.482955 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5fcc7ccbfd-ktknn_82e2709e-ab9a-4d23-8ade-39babbfbc8aa/console/0.log" Sep 14 13:41:21.566184 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.566145 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c849417b-aa78-4d5a-869c-70534cd460ae-sys\") pod \"perf-node-gather-daemonset-xsbz2\" (UID: \"c849417b-aa78-4d5a-869c-70534cd460ae\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" Sep 14 13:41:21.566184 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.566190 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c849417b-aa78-4d5a-869c-70534cd460ae-proc\") pod \"perf-node-gather-daemonset-xsbz2\" (UID: \"c849417b-aa78-4d5a-869c-70534cd460ae\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" Sep 14 13:41:21.566367 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.566256 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrgbd\" (UniqueName: \"kubernetes.io/projected/c849417b-aa78-4d5a-869c-70534cd460ae-kube-api-access-nrgbd\") pod \"perf-node-gather-daemonset-xsbz2\" (UID: \"c849417b-aa78-4d5a-869c-70534cd460ae\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" Sep 14 13:41:21.566367 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.566326 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c849417b-aa78-4d5a-869c-70534cd460ae-lib-modules\") pod \"perf-node-gather-daemonset-xsbz2\" (UID: \"c849417b-aa78-4d5a-869c-70534cd460ae\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" Sep 14 13:41:21.566443 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.566403 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c849417b-aa78-4d5a-869c-70534cd460ae-podres\") pod \"perf-node-gather-daemonset-xsbz2\" (UID: \"c849417b-aa78-4d5a-869c-70534cd460ae\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" Sep 14 13:41:21.667779 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.667742 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c849417b-aa78-4d5a-869c-70534cd460ae-podres\") pod \"perf-node-gather-daemonset-xsbz2\" (UID: \"c849417b-aa78-4d5a-869c-70534cd460ae\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" Sep 14 13:41:21.667982 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.667798 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c849417b-aa78-4d5a-869c-70534cd460ae-sys\") pod \"perf-node-gather-daemonset-xsbz2\" (UID: \"c849417b-aa78-4d5a-869c-70534cd460ae\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" Sep 14 13:41:21.667982 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.667819 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c849417b-aa78-4d5a-869c-70534cd460ae-proc\") pod \"perf-node-gather-daemonset-xsbz2\" (UID: \"c849417b-aa78-4d5a-869c-70534cd460ae\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" Sep 14 13:41:21.667982 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.667845 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrgbd\" (UniqueName: \"kubernetes.io/projected/c849417b-aa78-4d5a-869c-70534cd460ae-kube-api-access-nrgbd\") pod \"perf-node-gather-daemonset-xsbz2\" (UID: \"c849417b-aa78-4d5a-869c-70534cd460ae\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" Sep 14 13:41:21.667982 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.667898 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c849417b-aa78-4d5a-869c-70534cd460ae-lib-modules\") pod \"perf-node-gather-daemonset-xsbz2\" (UID: \"c849417b-aa78-4d5a-869c-70534cd460ae\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" Sep 14 13:41:21.667982 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.667914 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c849417b-aa78-4d5a-869c-70534cd460ae-podres\") pod \"perf-node-gather-daemonset-xsbz2\" (UID: \"c849417b-aa78-4d5a-869c-70534cd460ae\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" Sep 14 13:41:21.667982 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.667962 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c849417b-aa78-4d5a-869c-70534cd460ae-sys\") pod \"perf-node-gather-daemonset-xsbz2\" (UID: \"c849417b-aa78-4d5a-869c-70534cd460ae\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" Sep 14 13:41:21.667982 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.667977 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c849417b-aa78-4d5a-869c-70534cd460ae-proc\") pod \"perf-node-gather-daemonset-xsbz2\" (UID: \"c849417b-aa78-4d5a-869c-70534cd460ae\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" Sep 14 13:41:21.668385 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.668067 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c849417b-aa78-4d5a-869c-70534cd460ae-lib-modules\") pod \"perf-node-gather-daemonset-xsbz2\" (UID: \"c849417b-aa78-4d5a-869c-70534cd460ae\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" Sep 14 13:41:21.676451 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.676430 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrgbd\" (UniqueName: \"kubernetes.io/projected/c849417b-aa78-4d5a-869c-70534cd460ae-kube-api-access-nrgbd\") pod \"perf-node-gather-daemonset-xsbz2\" (UID: \"c849417b-aa78-4d5a-869c-70534cd460ae\") " pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" Sep 14 13:41:21.758611 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:21.758537 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" Sep 14 13:41:22.032392 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:22.032362 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2"] Sep 14 13:41:22.032815 ip-10-0-136-96 kubenswrapper[2564]: W0914 13:41:22.032786 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc849417b_aa78_4d5a_869c_70534cd460ae.slice/crio-b20c40e2ea3c401be1fba2fc970406696527cf639189f711a197506da93c1c97 WatchSource:0}: Error finding container b20c40e2ea3c401be1fba2fc970406696527cf639189f711a197506da93c1c97: Status 404 returned error can't find the container with id b20c40e2ea3c401be1fba2fc970406696527cf639189f711a197506da93c1c97 Sep 14 13:41:22.254218 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:22.254189 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-577p2_97519ad0-2873-4913-90da-1ba4dcfa7d9f/volume-data-source-validator/0.log" Sep 14 13:41:22.831867 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:22.831835 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" event={"ID":"c849417b-aa78-4d5a-869c-70534cd460ae","Type":"ContainerStarted","Data":"17b2cc9d0bedd0077b04baf82021ab9365c4ca44699d6fbbecba4c7848078a41"} Sep 14 13:41:22.832299 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:22.831905 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" event={"ID":"c849417b-aa78-4d5a-869c-70534cd460ae","Type":"ContainerStarted","Data":"b20c40e2ea3c401be1fba2fc970406696527cf639189f711a197506da93c1c97"} Sep 14 13:41:22.832299 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:22.831978 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" Sep 14 13:41:22.848737 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:22.848685 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" podStartSLOduration=1.8486694209999999 podStartE2EDuration="1.848669421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-14 13:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-14 13:41:22.848268758 +0000 UTC m=+2583.511164388" watchObservedRunningTime="2026-09-14 13:41:22.848669421 +0000 UTC m=+2583.511565051" Sep 14 13:41:23.427906 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:23.427876 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qr6v6_dbe829bc-1888-48fe-8416-3ae70353c8d7/dns/0.log" Sep 14 13:41:23.449410 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:23.449380 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qr6v6_dbe829bc-1888-48fe-8416-3ae70353c8d7/kube-rbac-proxy/0.log" Sep 14 13:41:23.470260 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:23.470237 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-g2k4m_0d6daf95-87b4-4f08-8058-6ad1d9ea9dec/dns-node-resolver/0.log" Sep 14 13:41:24.093977 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:24.093948 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4v75b_570dacc2-2bf1-4b1a-8f03-3abb696a2ecf/node-ca/0.log" Sep 14 13:41:25.022719 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:25.022690 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz_bbf1706b-9e71-41e5-82ba-8b1e58997b29/istio-proxy/0.log" Sep 14 13:41:25.868704 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:25.868678 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9v8ts_f9c24a09-932f-41c9-8391-16b38a0d7ee6/discovery/0.log" Sep 14 13:41:26.032283 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:26.032242 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64988bbc7d-tnjdw_0287d99a-ca2c-45bc-8f32-a2a273401e51/router/0.log" Sep 14 13:41:26.891064 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:26.891017 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dskx8_fc512c34-085d-4934-9c57-95f05e3db013/serve-healthcheck-canary/0.log" Sep 14 13:41:27.389486 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:27.389456 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-2xl57_ad5768be-fcc1-4c56-a485-2ae3f04504bc/insights-operator/1.log" Sep 14 13:41:27.396459 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:27.396417 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-2xl57_ad5768be-fcc1-4c56-a485-2ae3f04504bc/insights-operator/0.log" Sep 14 13:41:27.418701 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:27.418673 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2d9bv_e80a9489-e6d6-4afb-b04e-970dc9741e6e/kube-rbac-proxy/0.log" Sep 14 13:41:27.436671 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:27.436642 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2d9bv_e80a9489-e6d6-4afb-b04e-970dc9741e6e/exporter/0.log" Sep 14 13:41:27.455785 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:27.455736 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2d9bv_e80a9489-e6d6-4afb-b04e-970dc9741e6e/extractor/0.log" Sep 14 13:41:28.847684 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:28.847654 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2dnwm/perf-node-gather-daemonset-xsbz2" Sep 14 13:41:30.159174 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:30.159149 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-rwxn9_6ee95cc6-a9d1-426d-b595-b83cc025638d/manager/0.log" Sep 14 13:41:30.181045 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:30.181007 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-697cf97c89-dwk8r_3d67c169-7edc-46cb-ba89-b753cb011b73/manager/0.log" Sep 14 13:41:30.199032 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:30.199005 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-697cf97c89-dwk8r_3d67c169-7edc-46cb-ba89-b753cb011b73/copy-manifests/0.log" Sep 14 13:41:31.072638 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:31.072602 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz_bbf1706b-9e71-41e5-82ba-8b1e58997b29/istio-proxy/0.log" Sep 14 13:41:31.686703 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:31.686674 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6d5f86f746-xxrhs_26bec738-d51b-4da8-9eaa-2ac444c83372/manager/0.log" Sep 14 13:41:31.942454 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:31.942417 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9v8ts_f9c24a09-932f-41c9-8391-16b38a0d7ee6/discovery/0.log" Sep 14 13:41:32.083526 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:32.083493 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64988bbc7d-tnjdw_0287d99a-ca2c-45bc-8f32-a2a273401e51/router/0.log" Sep 14 13:41:33.606068 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:33.606040 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-vd8kz_bbf1706b-9e71-41e5-82ba-8b1e58997b29/istio-proxy/0.log" Sep 14 13:41:34.497044 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:34.496992 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9v8ts_f9c24a09-932f-41c9-8391-16b38a0d7ee6/discovery/0.log" Sep 14 13:41:34.619192 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:34.619162 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64988bbc7d-tnjdw_0287d99a-ca2c-45bc-8f32-a2a273401e51/router/0.log" Sep 14 13:41:37.081008 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:37.080978 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-2vff4_544ba1bf-58c6-428c-9830-a456e4940b63/kube-storage-version-migrator-operator/1.log" Sep 14 13:41:37.082808 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:37.082785 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-2vff4_544ba1bf-58c6-428c-9830-a456e4940b63/kube-storage-version-migrator-operator/0.log" Sep 14 13:41:37.996175 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:37.996149 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6h4bn_f44b8a44-25cf-4fec-a6ad-79aef4b55753/kube-multus-additional-cni-plugins/0.log" Sep 14 13:41:38.032340 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:38.032315 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6h4bn_f44b8a44-25cf-4fec-a6ad-79aef4b55753/egress-router-binary-copy/0.log" Sep 14 13:41:38.053421 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:38.053403 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6h4bn_f44b8a44-25cf-4fec-a6ad-79aef4b55753/cni-plugins/0.log" Sep 14 13:41:38.085047 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:38.085015 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6h4bn_f44b8a44-25cf-4fec-a6ad-79aef4b55753/bond-cni-plugin/0.log" Sep 14 13:41:38.121025 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:38.120998 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6h4bn_f44b8a44-25cf-4fec-a6ad-79aef4b55753/routeoverride-cni/0.log" Sep 14 13:41:38.153114 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:38.153086 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6h4bn_f44b8a44-25cf-4fec-a6ad-79aef4b55753/whereabouts-cni-bincopy/0.log" Sep 14 13:41:38.188076 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:38.188049 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6h4bn_f44b8a44-25cf-4fec-a6ad-79aef4b55753/whereabouts-cni/0.log" Sep 14 13:41:38.703218 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:38.703188 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fs7nx_921c1e9a-91e6-4f6d-846f-e146d92249ca/kube-multus/0.log" Sep 14 13:41:38.820281 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:38.820251 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kv576_ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63/network-metrics-daemon/0.log" Sep 14 13:41:38.849229 ip-10-0-136-96 kubenswrapper[2564]: I0914 13:41:38.849148 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kv576_ff51d3d8-9089-41b9-b6fe-13a3d7a8fb63/kube-rbac-proxy/0.log"