Sep 17 00:29:46.975796 ip-10-0-134-177 systemd[1]: Starting Kubernetes Kubelet... Sep 17 00:29:47.365552 ip-10-0-134-177 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 00:29:47.365552 ip-10-0-134-177 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 17 00:29:47.365552 ip-10-0-134-177 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 00:29:47.365552 ip-10-0-134-177 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 17 00:29:47.365552 ip-10-0-134-177 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 17 00:29:47.368461 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.368376 2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 17 00:29:47.374507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374482 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 00:29:47.374548 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374509 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 00:29:47.374548 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374513 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 00:29:47.374548 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374517 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 00:29:47.374548 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374519 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 00:29:47.374548 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374522 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 00:29:47.374548 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374527 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 00:29:47.374548 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374531 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 00:29:47.374548 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374534 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 00:29:47.374548 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374536 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 00:29:47.374548 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374539 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 00:29:47.374548 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374542 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 00:29:47.374548 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374545 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 00:29:47.374548 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374547 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 00:29:47.374548 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374550 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 00:29:47.374548 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374553 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 00:29:47.374548 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374555 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 00:29:47.374945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374559 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 00:29:47.374945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374561 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 00:29:47.374945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374564 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 00:29:47.374945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374566 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 00:29:47.374945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374569 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 00:29:47.374945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374571 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 00:29:47.374945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374574 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 00:29:47.374945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374576 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 00:29:47.374945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374579 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 00:29:47.374945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374581 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 00:29:47.374945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374583 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 00:29:47.374945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374586 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 00:29:47.374945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374588 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 00:29:47.374945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374591 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 00:29:47.374945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374593 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 00:29:47.374945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374595 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 00:29:47.374945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374598 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 00:29:47.374945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374600 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 00:29:47.374945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374603 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 00:29:47.374945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374605 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 00:29:47.375428 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374607 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 00:29:47.375428 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374616 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 00:29:47.375428 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374619 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 00:29:47.375428 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374621 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 00:29:47.375428 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374623 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 00:29:47.375428 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374626 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 00:29:47.375428 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374628 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 00:29:47.375428 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374631 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 00:29:47.375428 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374634 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 00:29:47.375428 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374637 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 00:29:47.375428 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374640 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 00:29:47.375428 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374642 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 00:29:47.375428 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374644 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 00:29:47.375428 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374648 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 00:29:47.375428 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374650 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 00:29:47.375428 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374653 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 00:29:47.375428 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374655 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 00:29:47.375428 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374658 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 00:29:47.375428 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374660 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 00:29:47.375428 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374663 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 00:29:47.375941 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374665 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 00:29:47.375941 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374668 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 00:29:47.375941 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374670 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 00:29:47.375941 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374673 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 00:29:47.375941 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374675 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 00:29:47.375941 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374677 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 00:29:47.375941 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374680 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 00:29:47.375941 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374682 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 00:29:47.375941 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374685 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 00:29:47.375941 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374687 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 00:29:47.375941 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374689 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 00:29:47.375941 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374692 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 00:29:47.375941 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374694 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 00:29:47.375941 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374697 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 00:29:47.375941 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374700 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 00:29:47.375941 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374703 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 00:29:47.375941 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374707 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 00:29:47.375941 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374710 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 00:29:47.375941 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374712 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 00:29:47.376513 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374715 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 00:29:47.376513 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374718 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 00:29:47.376513 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374720 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 00:29:47.376513 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374723 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 00:29:47.376513 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374725 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 00:29:47.376513 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374727 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 00:29:47.376513 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374731 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 00:29:47.376513 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374735 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 00:29:47.376513 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374738 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 00:29:47.376513 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374740 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 00:29:47.376513 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374743 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 00:29:47.376513 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374746 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 00:29:47.376513 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374748 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 00:29:47.376513 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374751 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 00:29:47.376513 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374754 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 00:29:47.376513 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374756 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 00:29:47.376513 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374782 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 00:29:47.376513 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374786 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 00:29:47.376513 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374789 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 00:29:47.376990 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374792 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 00:29:47.376990 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374795 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 00:29:47.376990 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374799 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 00:29:47.376990 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.374802 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 00:29:47.376990 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375545 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 00:29:47.376990 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375552 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 00:29:47.376990 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375556 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 00:29:47.376990 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375559 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 00:29:47.376990 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375562 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 00:29:47.376990 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375570 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 00:29:47.376990 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375573 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 00:29:47.376990 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375576 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 00:29:47.376990 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375579 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 00:29:47.376990 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375581 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 00:29:47.376990 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375584 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 00:29:47.376990 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375586 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 00:29:47.376990 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375589 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 00:29:47.376990 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375591 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 00:29:47.376990 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375594 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 00:29:47.377508 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375597 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 00:29:47.377508 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375600 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 00:29:47.377508 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375603 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 00:29:47.377508 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375605 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 00:29:47.377508 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375608 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 00:29:47.377508 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375610 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 00:29:47.377508 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375613 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 00:29:47.377508 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375615 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 00:29:47.377508 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375618 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 00:29:47.377508 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375621 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 00:29:47.377508 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375623 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 00:29:47.377508 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375626 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 00:29:47.377508 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375628 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 00:29:47.377508 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375630 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 00:29:47.377508 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375633 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 00:29:47.377508 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375635 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 00:29:47.377508 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375638 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 00:29:47.377508 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375640 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 00:29:47.377508 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375642 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 00:29:47.377508 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375645 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 00:29:47.378008 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375648 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 00:29:47.378008 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375650 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 00:29:47.378008 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375652 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 00:29:47.378008 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375661 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 00:29:47.378008 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375664 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 00:29:47.378008 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375666 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 00:29:47.378008 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375669 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 00:29:47.378008 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375673 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 00:29:47.378008 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375676 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 00:29:47.378008 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375678 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 00:29:47.378008 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375681 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 00:29:47.378008 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375683 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 00:29:47.378008 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375686 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 00:29:47.378008 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375688 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 00:29:47.378008 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375691 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 00:29:47.378008 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375695 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 00:29:47.378008 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375699 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 00:29:47.378008 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375702 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 00:29:47.378008 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375704 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 00:29:47.378484 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375707 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 00:29:47.378484 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375710 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 00:29:47.378484 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375712 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 00:29:47.378484 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375715 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 00:29:47.378484 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375717 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 00:29:47.378484 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375719 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 00:29:47.378484 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375722 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 00:29:47.378484 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375725 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 00:29:47.378484 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375728 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 00:29:47.378484 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375730 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 00:29:47.378484 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375733 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 00:29:47.378484 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375735 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 00:29:47.378484 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375737 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 00:29:47.378484 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375747 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 00:29:47.378484 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375750 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 00:29:47.378484 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375752 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 00:29:47.378484 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375755 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 00:29:47.378484 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375762 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 00:29:47.378484 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375765 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 00:29:47.378484 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375768 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 00:29:47.379010 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375770 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 00:29:47.379010 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375773 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 00:29:47.379010 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375775 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 00:29:47.379010 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375777 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 00:29:47.379010 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375780 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 00:29:47.379010 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375783 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 00:29:47.379010 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375786 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 00:29:47.379010 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375788 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 00:29:47.379010 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375791 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 00:29:47.379010 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375793 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 00:29:47.379010 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375796 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 00:29:47.379010 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375798 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 00:29:47.379010 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375801 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 00:29:47.379010 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375803 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 00:29:47.379010 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375806 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 00:29:47.379010 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375808 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 00:29:47.379010 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375811 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 00:29:47.379010 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375814 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 00:29:47.379010 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375817 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 00:29:47.379010 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375820 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 00:29:47.379512 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375823 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 00:29:47.379512 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375825 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 00:29:47.379512 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375832 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 00:29:47.379512 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375835 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 00:29:47.379512 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.375838 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 00:29:47.379512 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376259 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 00:29:47.379512 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376264 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 00:29:47.379512 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376267 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 00:29:47.379512 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376275 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 00:29:47.379512 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376284 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 00:29:47.379512 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376287 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 00:29:47.379512 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376291 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 00:29:47.379512 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376295 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 00:29:47.379512 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376297 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 00:29:47.379512 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376300 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 00:29:47.379512 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376302 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 00:29:47.379512 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376318 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 00:29:47.379512 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376326 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 00:29:47.379512 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376329 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 00:29:47.379983 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376332 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 00:29:47.379983 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376334 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 00:29:47.379983 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376337 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 00:29:47.379983 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376339 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 00:29:47.379983 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376342 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 00:29:47.379983 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376344 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 00:29:47.379983 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376347 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 00:29:47.379983 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376349 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 00:29:47.379983 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376351 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 00:29:47.379983 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376354 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 00:29:47.379983 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376356 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 00:29:47.379983 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376359 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 00:29:47.379983 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376361 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 00:29:47.379983 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376364 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 00:29:47.379983 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376366 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 00:29:47.379983 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376370 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 00:29:47.379983 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376374 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 00:29:47.379983 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376377 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 00:29:47.379983 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376380 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 00:29:47.380477 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376383 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 00:29:47.380477 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376385 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 00:29:47.380477 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376388 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 00:29:47.380477 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376390 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 00:29:47.380477 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376398 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 00:29:47.380477 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376400 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 00:29:47.380477 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376403 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 00:29:47.380477 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376405 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 00:29:47.380477 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376407 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 00:29:47.380477 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376410 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 00:29:47.380477 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376412 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 00:29:47.380477 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376416 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 00:29:47.380477 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376419 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 00:29:47.380477 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376422 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 00:29:47.380477 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376424 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 00:29:47.380477 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376426 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 00:29:47.380477 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376429 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 00:29:47.380477 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376433 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 00:29:47.380477 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376436 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 00:29:47.380970 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376438 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 00:29:47.380970 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376441 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 00:29:47.380970 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376443 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 00:29:47.380970 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376447 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 00:29:47.380970 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376449 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 00:29:47.380970 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376451 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 00:29:47.380970 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376454 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 00:29:47.380970 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376457 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 00:29:47.380970 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376459 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 00:29:47.380970 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376463 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 00:29:47.380970 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376466 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 00:29:47.380970 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376468 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 00:29:47.380970 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376470 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 00:29:47.380970 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376473 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 00:29:47.380970 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376475 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 00:29:47.380970 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376478 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 00:29:47.380970 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376480 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 00:29:47.380970 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376483 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 00:29:47.380970 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376492 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 00:29:47.380970 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376512 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 00:29:47.381507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376518 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 00:29:47.381507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376522 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 00:29:47.381507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376526 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 00:29:47.381507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376529 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 00:29:47.381507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376544 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 00:29:47.381507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376547 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 00:29:47.381507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376550 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 00:29:47.381507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376552 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 00:29:47.381507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376555 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 00:29:47.381507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376557 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 00:29:47.381507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376560 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 00:29:47.381507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376562 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 00:29:47.381507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376565 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 00:29:47.381507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376568 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 00:29:47.381507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376570 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 00:29:47.381507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376572 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 00:29:47.381507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376575 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 00:29:47.381507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376577 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 00:29:47.381507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376580 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 00:29:47.381507 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376582 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376584 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376588 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376591 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376593 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376596 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376598 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.376600 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376665 2582 flags.go:64] FLAG: --address="0.0.0.0" Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376699 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376708 2582 flags.go:64] FLAG: --anonymous-auth="true" Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376712 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376721 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376724 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376728 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376732 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376735 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376738 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376741 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376744 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376747 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376750 2582 flags.go:64] FLAG: --cgroup-root="" Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376752 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 17 00:29:47.381994 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376755 2582 flags.go:64] FLAG: --client-ca-file="" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376758 2582 flags.go:64] FLAG: --cloud-provider="external" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376761 2582 flags.go:64] FLAG: --cluster-dns="[]" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376768 2582 flags.go:64] FLAG: --cluster-domain="" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376771 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376774 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376777 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376780 2582 flags.go:64] FLAG: --container-log-max-files="5" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376784 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376786 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376789 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376794 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376797 2582 flags.go:64] FLAG: --contention-profiling="false" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376800 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376803 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376806 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376809 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376813 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376815 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376818 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376821 2582 flags.go:64] FLAG: --enable-load-reader="false" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376824 2582 flags.go:64] FLAG: --enable-server="true" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376832 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376838 2582 flags.go:64] FLAG: --event-burst="100" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376841 2582 flags.go:64] FLAG: --event-qps="50" Sep 17 00:29:47.382553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376844 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376847 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376850 2582 flags.go:64] FLAG: --eviction-hard="" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376854 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376857 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376860 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376862 2582 flags.go:64] FLAG: --eviction-soft="" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376865 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376868 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376871 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376874 2582 flags.go:64] FLAG: --experimental-mounter-path="" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376876 2582 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376879 2582 flags.go:64] FLAG: --fail-swap-on="true" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376882 2582 flags.go:64] FLAG: --feature-gates="" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376885 2582 flags.go:64] FLAG: --file-check-frequency="20s" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376889 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376892 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376895 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376899 2582 flags.go:64] FLAG: --healthz-port="10248" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376902 2582 flags.go:64] FLAG: --help="false" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376904 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-134-177.ec2.internal" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376908 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376910 2582 flags.go:64] FLAG: --http-check-frequency="20s" Sep 17 00:29:47.383188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376913 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376917 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376920 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376922 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376925 2582 flags.go:64] FLAG: --image-service-endpoint="" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376928 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376931 2582 flags.go:64] FLAG: --kube-api-burst="100" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376939 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376942 2582 flags.go:64] FLAG: --kube-api-qps="50" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376945 2582 flags.go:64] FLAG: --kube-reserved="" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376948 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376950 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376953 2582 flags.go:64] FLAG: --kubelet-cgroups="" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376956 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376958 2582 flags.go:64] FLAG: --lock-file="" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376961 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376964 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376966 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376971 2582 flags.go:64] FLAG: --log-json-split-stream="false" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376974 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376976 2582 flags.go:64] FLAG: --log-text-split-stream="false" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376979 2582 flags.go:64] FLAG: --logging-format="text" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376982 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376985 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 17 00:29:47.383798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376987 2582 flags.go:64] FLAG: --manifest-url="" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376990 2582 flags.go:64] FLAG: --manifest-url-header="" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376995 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.376999 2582 flags.go:64] FLAG: --max-open-files="1000000" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377003 2582 flags.go:64] FLAG: --max-pods="110" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377006 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377009 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377012 2582 flags.go:64] FLAG: --memory-manager-policy="None" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377014 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377017 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377020 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377023 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377030 2582 flags.go:64] FLAG: --node-status-max-images="50" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377032 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377035 2582 flags.go:64] FLAG: --oom-score-adj="-999" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377038 2582 flags.go:64] FLAG: --pod-cidr="" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377042 2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377047 2582 flags.go:64] FLAG: --pod-manifest-path="" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377050 2582 flags.go:64] FLAG: --pod-max-pids="-1" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377054 2582 flags.go:64] FLAG: --pods-per-core="0" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377057 2582 flags.go:64] FLAG: --port="10250" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377060 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377062 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b9e30dc0f5f7386e" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377065 2582 flags.go:64] FLAG: --qos-reserved="" Sep 17 00:29:47.384387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377068 2582 flags.go:64] FLAG: --read-only-port="10255" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377071 2582 flags.go:64] FLAG: --register-node="true" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377074 2582 flags.go:64] FLAG: --register-with-taints="" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377077 2582 flags.go:64] FLAG: --registry-burst="10" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377080 2582 flags.go:64] FLAG: --registry-qps="5" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377082 2582 flags.go:64] FLAG: --reserved-cpus="" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377085 2582 flags.go:64] FLAG: --reserved-memory="" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377089 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377091 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377094 2582 flags.go:64] FLAG: --rotate-certificates="false" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377097 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377100 2582 flags.go:64] FLAG: --runonce="false" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377103 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377106 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377109 2582 flags.go:64] FLAG: --seccomp-default="false" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377112 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377114 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377117 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377120 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377122 2582 flags.go:64] FLAG: --storage-driver-password="root" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377125 2582 flags.go:64] FLAG: --storage-driver-secure="false" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377128 2582 flags.go:64] FLAG: --storage-driver-table="stats" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377130 2582 flags.go:64] FLAG: --storage-driver-user="root" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377133 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377136 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 17 00:29:47.385064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377139 2582 flags.go:64] FLAG: --system-cgroups="" Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377142 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377148 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377151 2582 flags.go:64] FLAG: --tls-cert-file="" Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377154 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377160 2582 flags.go:64] FLAG: --tls-min-version="" Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377163 2582 flags.go:64] FLAG: --tls-private-key-file="" Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377165 2582 flags.go:64] FLAG: --topology-manager-policy="none" Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377168 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377171 2582 flags.go:64] FLAG: --topology-manager-scope="container" Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377173 2582 flags.go:64] FLAG: --v="2" Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377177 2582 flags.go:64] FLAG: --version="false" Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377181 2582 flags.go:64] FLAG: --vmodule="" Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377184 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.377187 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377349 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377354 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377357 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377360 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377362 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377365 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377367 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377370 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 00:29:47.385700 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377372 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 00:29:47.386284 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377375 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 00:29:47.386284 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377377 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 00:29:47.386284 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377380 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 00:29:47.386284 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377382 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 00:29:47.386284 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377385 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 00:29:47.386284 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377387 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 00:29:47.386284 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377390 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 00:29:47.386284 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377392 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 00:29:47.386284 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377395 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 00:29:47.386284 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377403 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 00:29:47.386284 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377407 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 00:29:47.386284 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377410 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 00:29:47.386284 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377412 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 00:29:47.386284 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377414 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 00:29:47.386284 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377417 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 00:29:47.386284 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377420 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 00:29:47.386284 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377422 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 00:29:47.386284 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377425 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 00:29:47.386284 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377427 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 00:29:47.386284 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377430 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 00:29:47.387144 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377432 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 00:29:47.387144 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377435 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 00:29:47.387144 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377437 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 00:29:47.387144 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377440 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 00:29:47.387144 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377442 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 00:29:47.387144 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377445 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 00:29:47.387144 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377447 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 00:29:47.387144 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377450 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 00:29:47.387144 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377452 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 00:29:47.387144 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377455 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 00:29:47.387144 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377457 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 00:29:47.387144 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377460 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 00:29:47.387144 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377462 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 00:29:47.387144 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377465 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 00:29:47.387144 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377467 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 00:29:47.387144 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377470 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 00:29:47.387144 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377472 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 00:29:47.387144 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377475 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 00:29:47.387144 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377477 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 00:29:47.387144 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377480 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 00:29:47.387891 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377482 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 00:29:47.387891 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377485 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 00:29:47.387891 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377508 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 00:29:47.387891 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377514 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 00:29:47.387891 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377517 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 00:29:47.387891 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377520 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 00:29:47.387891 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377523 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 00:29:47.387891 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377526 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 00:29:47.387891 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377528 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 00:29:47.387891 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377531 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 00:29:47.387891 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377534 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 00:29:47.387891 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377536 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 00:29:47.387891 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377539 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 00:29:47.387891 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377541 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 00:29:47.387891 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377545 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 00:29:47.387891 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377549 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 00:29:47.387891 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377552 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 00:29:47.387891 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377554 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 00:29:47.387891 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377557 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 00:29:47.388385 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377560 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 00:29:47.388385 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377562 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 00:29:47.388385 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377565 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 00:29:47.388385 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377567 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 00:29:47.388385 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377570 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 00:29:47.388385 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377574 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 00:29:47.388385 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377578 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 00:29:47.388385 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377581 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 00:29:47.388385 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377584 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 00:29:47.388385 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377587 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 00:29:47.388385 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377589 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 00:29:47.388385 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377592 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 00:29:47.388385 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377594 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 00:29:47.388385 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377597 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 00:29:47.388385 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377600 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 00:29:47.388385 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377604 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 00:29:47.388385 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377608 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 00:29:47.388385 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377611 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 00:29:47.388385 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377614 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 00:29:47.388922 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377617 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 00:29:47.388922 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377620 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 00:29:47.388922 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377623 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 00:29:47.388922 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377625 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 00:29:47.388922 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377628 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 00:29:47.388922 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377630 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 00:29:47.388922 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377633 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 00:29:47.388922 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377635 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 00:29:47.388922 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377638 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 00:29:47.388922 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377640 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 00:29:47.388922 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377643 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 00:29:47.388922 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.377645 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 00:29:47.388922 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.378367 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 00:29:47.388922 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.386128 2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 17 00:29:47.388922 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.386141 2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 17 00:29:47.389338 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.386159 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 17 00:29:47.389338 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.386165 2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 17 00:29:47.389338 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.386367 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 00:29:47.389338 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387528 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 00:29:47.389338 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387537 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 00:29:47.389338 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387542 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 00:29:47.389338 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387545 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 00:29:47.389338 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387548 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 00:29:47.389338 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387553 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 00:29:47.389338 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387604 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 00:29:47.389338 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387745 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 00:29:47.389338 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387749 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 00:29:47.389338 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387753 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 00:29:47.389338 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387756 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 00:29:47.389338 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387759 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 00:29:47.389338 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387762 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 00:29:47.389338 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387765 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 00:29:47.389338 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387769 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 00:29:47.389338 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387772 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 00:29:47.389823 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387775 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 00:29:47.389823 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387778 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 00:29:47.389823 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387781 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 00:29:47.389823 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387783 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 00:29:47.389823 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387786 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 00:29:47.389823 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387788 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 00:29:47.389823 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387791 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 00:29:47.389823 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387793 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 00:29:47.389823 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387796 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 00:29:47.389823 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387798 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 00:29:47.389823 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387801 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 00:29:47.389823 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387804 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 00:29:47.389823 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387806 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 00:29:47.389823 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387809 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 00:29:47.389823 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387811 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 00:29:47.389823 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387814 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 00:29:47.389823 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387816 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 00:29:47.389823 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387819 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 00:29:47.389823 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387822 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 00:29:47.389823 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387825 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 00:29:47.390302 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387828 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 00:29:47.390302 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387830 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 00:29:47.390302 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387833 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 00:29:47.390302 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387837 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 00:29:47.390302 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387840 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 00:29:47.390302 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387842 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 00:29:47.390302 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387845 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 00:29:47.390302 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387848 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 00:29:47.390302 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387853 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 00:29:47.390302 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387857 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 00:29:47.390302 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387860 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 00:29:47.390302 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387862 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 00:29:47.390302 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387865 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 00:29:47.390302 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387868 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 00:29:47.390302 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387870 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 00:29:47.390302 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387873 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 00:29:47.390302 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387876 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 00:29:47.390302 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387879 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 00:29:47.390302 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387882 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 00:29:47.390302 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387885 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 00:29:47.390798 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387887 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 00:29:47.390798 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387890 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 00:29:47.390798 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387893 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 00:29:47.390798 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387896 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 00:29:47.390798 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387898 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 00:29:47.390798 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387901 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 00:29:47.390798 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387904 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 00:29:47.390798 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387906 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 00:29:47.390798 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387910 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 00:29:47.390798 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387912 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 00:29:47.390798 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387915 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 00:29:47.390798 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387918 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 00:29:47.390798 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387920 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 00:29:47.390798 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387923 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 00:29:47.390798 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387926 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 00:29:47.390798 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387946 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 00:29:47.390798 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387950 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 00:29:47.390798 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387954 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 00:29:47.390798 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387958 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 00:29:47.391314 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387961 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 00:29:47.391314 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387964 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 00:29:47.391314 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387968 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 00:29:47.391314 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387973 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 00:29:47.391314 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387977 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 00:29:47.391314 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387980 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 00:29:47.391314 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387983 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 00:29:47.391314 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387986 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 00:29:47.391314 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387988 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 00:29:47.391314 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387991 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 00:29:47.391314 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387993 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 00:29:47.391314 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387996 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 00:29:47.391314 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.387998 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 00:29:47.391314 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388001 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 00:29:47.391314 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388004 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 00:29:47.391314 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388007 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 00:29:47.391314 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388009 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 00:29:47.391314 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388011 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 00:29:47.391314 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388014 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 00:29:47.391314 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388016 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 00:29:47.391896 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388019 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 00:29:47.391896 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388021 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 00:29:47.391896 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388024 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 00:29:47.391896 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.388030 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 00:29:47.391896 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388144 2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 17 00:29:47.391896 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388147 2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 17 00:29:47.391896 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388150 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 17 00:29:47.391896 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388154 2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 17 00:29:47.391896 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388158 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 17 00:29:47.391896 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388162 2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 17 00:29:47.391896 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388165 2582 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 17 00:29:47.391896 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388168 2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 17 00:29:47.391896 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388171 2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 17 00:29:47.391896 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388174 2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 17 00:29:47.392264 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388177 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 17 00:29:47.392264 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388179 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 17 00:29:47.392264 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388182 2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 17 00:29:47.392264 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388184 2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 17 00:29:47.392264 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388186 2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 17 00:29:47.392264 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388189 2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 17 00:29:47.392264 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388191 2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 17 00:29:47.392264 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388194 2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 17 00:29:47.392264 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388196 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 17 00:29:47.392264 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388198 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 17 00:29:47.392264 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388201 2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 17 00:29:47.392264 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388203 2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 17 00:29:47.392264 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388206 2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 17 00:29:47.392264 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388208 2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 17 00:29:47.392264 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388210 2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 17 00:29:47.392264 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388213 2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 17 00:29:47.392264 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388216 2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 17 00:29:47.392264 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388218 2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 17 00:29:47.392264 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388221 2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 17 00:29:47.392264 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388223 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 17 00:29:47.392771 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388227 2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 17 00:29:47.392771 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388231 2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 17 00:29:47.392771 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388234 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 17 00:29:47.392771 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388236 2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 17 00:29:47.392771 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388239 2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 17 00:29:47.392771 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388242 2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 17 00:29:47.392771 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388244 2582 feature_gate.go:328] unrecognized feature gate: Example Sep 17 00:29:47.392771 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388247 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 17 00:29:47.392771 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388249 2582 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 17 00:29:47.392771 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388252 2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 17 00:29:47.392771 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388254 2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 17 00:29:47.392771 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388257 2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 17 00:29:47.392771 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388259 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 17 00:29:47.392771 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388262 2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 17 00:29:47.392771 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388264 2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 17 00:29:47.392771 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388267 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 17 00:29:47.392771 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388269 2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 17 00:29:47.392771 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388272 2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 17 00:29:47.392771 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388274 2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 17 00:29:47.392771 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388278 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 17 00:29:47.393248 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388280 2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 17 00:29:47.393248 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388283 2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 17 00:29:47.393248 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388285 2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 17 00:29:47.393248 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388289 2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 17 00:29:47.393248 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388292 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 17 00:29:47.393248 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388295 2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 17 00:29:47.393248 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388297 2582 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 17 00:29:47.393248 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388300 2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 17 00:29:47.393248 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388303 2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 17 00:29:47.393248 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388305 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 17 00:29:47.393248 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388308 2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 17 00:29:47.393248 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388311 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 17 00:29:47.393248 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388313 2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 17 00:29:47.393248 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388316 2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 17 00:29:47.393248 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388319 2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 17 00:29:47.393248 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388321 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 17 00:29:47.393248 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388323 2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 17 00:29:47.393248 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388326 2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 17 00:29:47.393248 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388328 2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 17 00:29:47.393735 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388331 2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 17 00:29:47.393735 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388333 2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 17 00:29:47.393735 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388336 2582 feature_gate.go:328] unrecognized feature gate: Example2 Sep 17 00:29:47.393735 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388338 2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 17 00:29:47.393735 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388341 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 17 00:29:47.393735 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388343 2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 17 00:29:47.393735 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388345 2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 17 00:29:47.393735 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388348 2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 17 00:29:47.393735 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388350 2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 17 00:29:47.393735 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388352 2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 17 00:29:47.393735 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388355 2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 17 00:29:47.393735 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388357 2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 17 00:29:47.393735 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388360 2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 17 00:29:47.393735 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388362 2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 17 00:29:47.393735 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388365 2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 17 00:29:47.393735 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388367 2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 17 00:29:47.393735 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388370 2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 17 00:29:47.393735 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388372 2582 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 17 00:29:47.393735 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388375 2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 17 00:29:47.393735 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388377 2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 17 00:29:47.394243 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388379 2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 17 00:29:47.394243 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388382 2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 17 00:29:47.394243 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388385 2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 17 00:29:47.394243 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388387 2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 17 00:29:47.394243 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388390 2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 17 00:29:47.394243 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388392 2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 17 00:29:47.394243 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388395 2582 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 17 00:29:47.394243 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388398 2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 17 00:29:47.394243 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388400 2582 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 17 00:29:47.394243 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:47.388403 2582 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 17 00:29:47.394243 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.388408 2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 17 00:29:47.394243 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.388531 2582 server.go:962] "Client rotation is on, will bootstrap in background" Sep 17 00:29:47.394243 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.391040 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 17 00:29:47.394243 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.392059 2582 server.go:1019] "Starting client certificate rotation" Sep 17 00:29:47.394243 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.392156 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 00:29:47.394635 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.392190 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 17 00:29:47.395747 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.395733 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 00:29:47.398755 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.398737 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 17 00:29:47.410975 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.410957 2582 log.go:25] "Validated CRI v1 runtime API" Sep 17 00:29:47.416347 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.416330 2582 log.go:25] "Validated CRI v1 image API" Sep 17 00:29:47.417245 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.417232 2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 17 00:29:47.419764 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.419749 2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bac6a049-c924-4730-9ccf-ee154034bb99:/dev/nvme0n1p3 d52c2469-8cd4-4c72-b575-b7b74082dd97:/dev/nvme0n1p4] Sep 17 00:29:47.419818 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.419766 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 17 00:29:47.424680 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.424567 2582 manager.go:216] Machine: {Timestamp:2026-09-17 00:29:47.424102079 +0000 UTC m=+0.354735558 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100261 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27a548186362d437fbfa3b03ebe647 SystemUUID:ec27a548-1863-62d4-37fb-fa3b03ebe647 BootID:31a9f604-c8d2-45ec-9167-8ba3a64ad235 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d8:8d:46:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d8:8d:46:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:9d:00:1e:ef:1d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 17 00:29:47.424680 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.424675 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 17 00:29:47.424784 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.424744 2582 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 17 00:29:47.425456 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.425438 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 00:29:47.428692 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.428664 2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 17 00:29:47.428835 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.428697 2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-177.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 17 00:29:47.429589 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.429579 2582 topology_manager.go:138] "Creating topology manager with none policy" Sep 17 00:29:47.429619 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.429591 2582 container_manager_linux.go:309] "Creating device plugin manager" Sep 17 00:29:47.429619 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.429599 2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 00:29:47.429619 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.429611 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 17 00:29:47.430918 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.430909 2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 17 00:29:47.436175 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.436162 2582 state_mem.go:36] "Initialized new in-memory state store" Sep 17 00:29:47.436298 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.436289 2582 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 17 00:29:47.436356 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.436347 2582 kubelet.go:486] "Attempting to sync node with API server" Sep 17 00:29:47.436385 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.436370 2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 17 00:29:47.436411 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.436392 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 17 00:29:47.436411 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.436405 2582 kubelet.go:398] "Adding apiserver pod source" Sep 17 00:29:47.436471 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.436436 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 17 00:29:47.436893 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.436878 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 00:29:47.436982 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.436901 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 17 00:29:47.440840 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.440817 2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 17 00:29:47.443253 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.443236 2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 17 00:29:47.443344 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.443279 2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 17 00:29:47.445180 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.445168 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 17 00:29:47.445238 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.445188 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 17 00:29:47.445238 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.445198 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 17 00:29:47.445238 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.445205 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 17 00:29:47.445238 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.445214 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 17 00:29:47.445238 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.445219 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 17 00:29:47.445238 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.445232 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 17 00:29:47.445238 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.445238 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 17 00:29:47.445426 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.445245 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 17 00:29:47.445426 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.445251 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 17 00:29:47.445426 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.445259 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 17 00:29:47.445426 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.445264 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 17 00:29:47.447023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.447009 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 17 00:29:47.447023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.447021 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 17 00:29:47.448469 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:47.448446 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 17 00:29:47.448548 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:47.448450 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-177.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 17 00:29:47.448666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.448646 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-177.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 17 00:29:47.450812 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.450794 2582 server.go:1268] "Started kubelet" Sep 17 00:29:47.450888 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.450858 2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 17 00:29:47.450953 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.450879 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 17 00:29:47.450953 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.450925 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 17 00:29:47.451664 ip-10-0-134-177 systemd[1]: Started Kubernetes Kubelet. Sep 17 00:29:47.452965 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.452952 2582 server.go:310] "Adding debug handlers to kubelet server" Sep 17 00:29:47.453406 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.453385 2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 17 00:29:47.455441 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:47.455423 2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 17 00:29:47.459266 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.459245 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 17 00:29:47.459972 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.459954 2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 17 00:29:47.460115 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:47.460086 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-177.ec2.internal\" not found" Sep 17 00:29:47.460765 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.460735 2582 factory.go:55] Registering systemd factory Sep 17 00:29:47.460765 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.460745 2582 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 17 00:29:47.460765 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.460756 2582 factory.go:223] Registration of the systemd container factory successfully Sep 17 00:29:47.460765 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.460763 2582 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 17 00:29:47.460955 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.460742 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 17 00:29:47.460955 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.460863 2582 reconstruct.go:97] "Volume reconstruction finished" Sep 17 00:29:47.460955 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.460873 2582 reconciler.go:29] "Reconciler: start to sync state" Sep 17 00:29:47.461042 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.460981 2582 factory.go:153] Registering CRI-O factory Sep 17 00:29:47.461042 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.461012 2582 factory.go:223] Registration of the crio container factory successfully Sep 17 00:29:47.461109 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.461076 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 17 00:29:47.461109 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.461098 2582 factory.go:103] Registering Raw factory Sep 17 00:29:47.461181 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.461112 2582 manager.go:1196] Started watching for new ooms in manager Sep 17 00:29:47.461575 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.461553 2582 manager.go:349] Starting recovery of all containers Sep 17 00:29:47.465146 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:47.465018 2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 17 00:29:47.465146 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:47.465026 2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-177.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 17 00:29:47.466330 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:47.465113 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-177.ec2.internal.18d5f4411d453427 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-177.ec2.internal,UID:ip-10-0-134-177.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-177.ec2.internal,},FirstTimestamp:2026-09-17 00:29:47.450766375 +0000 UTC m=+0.381399852,LastTimestamp:2026-09-17 00:29:47.450766375 +0000 UTC m=+0.381399852,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-177.ec2.internal,}" Sep 17 00:29:47.474642 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.474625 2582 manager.go:354] Recovery completed Sep 17 00:29:47.474860 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.474780 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bgm4q" Sep 17 00:29:47.479117 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.479106 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 00:29:47.479533 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.479515 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bgm4q" Sep 17 00:29:47.481646 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.481633 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasSufficientMemory" Sep 17 00:29:47.481721 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.481659 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 00:29:47.481721 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.481672 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasSufficientPID" Sep 17 00:29:47.482205 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.482191 2582 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 17 00:29:47.482262 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.482204 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 00:29:47.482298 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.482206 2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 17 00:29:47.482298 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.482280 2582 state_mem.go:36] "Initialized new in-memory state store" Sep 17 00:29:47.483339 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:47.483276 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-177.ec2.internal.18d5f4411f1c63da default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-177.ec2.internal,UID:ip-10-0-134-177.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-177.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-177.ec2.internal,},FirstTimestamp:2026-09-17 00:29:47.481646042 +0000 UTC m=+0.412279520,LastTimestamp:2026-09-17 00:29:47.481646042 +0000 UTC m=+0.412279520,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-177.ec2.internal,}" Sep 17 00:29:47.484289 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.484273 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasSufficientMemory" Sep 17 00:29:47.484337 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.484304 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 00:29:47.484337 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.484318 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasSufficientPID" Sep 17 00:29:47.484398 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.484390 2582 policy_none.go:49] "None policy: Start" Sep 17 00:29:47.484426 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.484404 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 17 00:29:47.484426 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.484414 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 17 00:29:47.486437 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.486425 2582 policy_none.go:47] "Start" Sep 17 00:29:47.528119 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.528104 2582 manager.go:337] "Starting Device Plugin manager" Sep 17 00:29:47.551008 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:47.528144 2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 17 00:29:47.551008 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.528155 2582 server.go:87] "Starting device plugin registration server" Sep 17 00:29:47.551008 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.528461 2582 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 17 00:29:47.551008 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.528472 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 17 00:29:47.551008 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.528562 2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 17 00:29:47.551008 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.528661 2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 17 00:29:47.551008 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.528668 2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 17 00:29:47.551008 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:47.529146 2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 17 00:29:47.551008 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:47.529191 2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-177.ec2.internal\" not found" Sep 17 00:29:47.628935 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.628837 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 00:29:47.629805 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.629777 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasSufficientMemory" Sep 17 00:29:47.629902 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.629814 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 00:29:47.629902 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.629825 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasSufficientPID" Sep 17 00:29:47.629902 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.629851 2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-177.ec2.internal" Sep 17 00:29:47.637885 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.637868 2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-177.ec2.internal" Sep 17 00:29:47.638023 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:47.637894 2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-177.ec2.internal\": node \"ip-10-0-134-177.ec2.internal\" not found" Sep 17 00:29:47.642292 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.642272 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 17 00:29:47.643628 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.643612 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 17 00:29:47.643697 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.643659 2582 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 17 00:29:47.643697 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.643680 2582 kubelet.go:2448] "Starting kubelet main sync loop" Sep 17 00:29:47.643768 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:47.643721 2582 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 17 00:29:47.645554 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.645538 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 00:29:47.659426 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:47.659411 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-177.ec2.internal\" not found" Sep 17 00:29:47.744561 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.744537 2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-177.ec2.internal"] Sep 17 00:29:47.744691 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.744614 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 00:29:47.745515 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.745479 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasSufficientMemory" Sep 17 00:29:47.745574 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.745527 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 00:29:47.745574 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.745548 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasSufficientPID" Sep 17 00:29:47.747955 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.747943 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 00:29:47.748070 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.748056 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal" Sep 17 00:29:47.748116 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.748095 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 00:29:47.748681 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.748666 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasSufficientMemory" Sep 17 00:29:47.748681 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.748674 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasSufficientMemory" Sep 17 00:29:47.748799 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.748696 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 00:29:47.748799 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.748706 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasSufficientPID" Sep 17 00:29:47.748799 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.748696 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 00:29:47.748799 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.748776 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasSufficientPID" Sep 17 00:29:47.750933 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.750916 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-177.ec2.internal" Sep 17 00:29:47.751021 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.750942 2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 17 00:29:47.751565 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.751551 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasSufficientMemory" Sep 17 00:29:47.751652 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.751569 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasNoDiskPressure" Sep 17 00:29:47.751652 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.751578 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeHasSufficientPID" Sep 17 00:29:47.760205 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:47.760190 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-177.ec2.internal\" not found" Sep 17 00:29:47.763455 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.763442 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73d3f85df08cf592386ff2d120f060eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal\" (UID: \"73d3f85df08cf592386ff2d120f060eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal" Sep 17 00:29:47.763538 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.763465 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73d3f85df08cf592386ff2d120f060eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal\" (UID: \"73d3f85df08cf592386ff2d120f060eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal" Sep 17 00:29:47.763538 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.763479 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/81937eb52bebe19931264bfcfb8d032c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-177.ec2.internal\" (UID: \"81937eb52bebe19931264bfcfb8d032c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-177.ec2.internal" Sep 17 00:29:47.764098 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:47.764080 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-177.ec2.internal\" not found" node="ip-10-0-134-177.ec2.internal" Sep 17 00:29:47.767537 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:47.767521 2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-177.ec2.internal\" not found" node="ip-10-0-134-177.ec2.internal" Sep 17 00:29:47.860471 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:47.860429 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-177.ec2.internal\" not found" Sep 17 00:29:47.863716 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.863701 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73d3f85df08cf592386ff2d120f060eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal\" (UID: \"73d3f85df08cf592386ff2d120f060eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal" Sep 17 00:29:47.863765 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.863724 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73d3f85df08cf592386ff2d120f060eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal\" (UID: \"73d3f85df08cf592386ff2d120f060eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal" Sep 17 00:29:47.863765 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.863745 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/81937eb52bebe19931264bfcfb8d032c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-177.ec2.internal\" (UID: \"81937eb52bebe19931264bfcfb8d032c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-177.ec2.internal" Sep 17 00:29:47.863841 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.863797 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/73d3f85df08cf592386ff2d120f060eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal\" (UID: \"73d3f85df08cf592386ff2d120f060eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal" Sep 17 00:29:47.863841 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.863813 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73d3f85df08cf592386ff2d120f060eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal\" (UID: \"73d3f85df08cf592386ff2d120f060eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal" Sep 17 00:29:47.863841 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:47.863802 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/81937eb52bebe19931264bfcfb8d032c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-177.ec2.internal\" (UID: \"81937eb52bebe19931264bfcfb8d032c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-177.ec2.internal" Sep 17 00:29:47.961232 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:47.961133 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-177.ec2.internal\" not found" Sep 17 00:29:48.061640 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:48.061606 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-177.ec2.internal\" not found" Sep 17 00:29:48.066926 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.066909 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal" Sep 17 00:29:48.070019 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.070005 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-177.ec2.internal" Sep 17 00:29:48.161997 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:48.161967 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-177.ec2.internal\" not found" Sep 17 00:29:48.262373 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:48.262315 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-177.ec2.internal\" not found" Sep 17 00:29:48.362839 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:48.362815 2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-177.ec2.internal\" not found" Sep 17 00:29:48.392054 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.392037 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 17 00:29:48.392391 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.392163 2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 17 00:29:48.421392 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.421365 2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 00:29:48.436850 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.436825 2582 apiserver.go:52] "Watching apiserver" Sep 17 00:29:48.445387 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.445359 2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 17 00:29:48.446035 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.446010 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-bs8f6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm","openshift-image-registry/node-ca-d7jth","openshift-multus/multus-additional-cni-plugins-6pld9","openshift-multus/network-metrics-daemon-rh4cz","openshift-network-diagnostics/network-check-target-xv4br","openshift-ovn-kubernetes/ovnkube-node-wzmbr","kube-system/konnectivity-agent-55r97","openshift-cluster-node-tuning-operator/tuned-vnqww","openshift-dns/node-resolver-kzvtd","openshift-multus/multus-fmx98","openshift-network-operator/iptables-alerter-hpwmd"] Sep 17 00:29:48.448603 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.448587 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:29:48.448676 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:48.448658 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bs8f6" podUID="dda66642-c725-4b58-82ac-283b33d09dab" Sep 17 00:29:48.450768 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.450749 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.452584 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.452567 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 17 00:29:48.452847 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.452833 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 17 00:29:48.452943 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.452923 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d7jth" Sep 17 00:29:48.453031 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.453017 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m68w8\"" Sep 17 00:29:48.453275 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.453260 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 17 00:29:48.454690 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.454674 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 17 00:29:48.454794 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.454777 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 17 00:29:48.454846 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.454807 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tqndc\"" Sep 17 00:29:48.455102 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.455086 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 17 00:29:48.455231 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.455216 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.456974 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.456958 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 17 00:29:48.457050 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.456959 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lvf5v\"" Sep 17 00:29:48.457222 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.457208 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 17 00:29:48.457332 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.457308 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 17 00:29:48.457417 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.457403 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 17 00:29:48.457535 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.457521 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 17 00:29:48.457592 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.457541 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:29:48.457637 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:48.457590 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4cz" podUID="6155b312-55da-43b1-9b1f-4d51dfbb0d42" Sep 17 00:29:48.459651 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.459638 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 17 00:29:48.459736 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.459723 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:29:48.459789 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:48.459774 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xv4br" podUID="c59a8ddf-3715-4032-a1d4-2f01bd95aed1" Sep 17 00:29:48.460189 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.460173 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal" Sep 17 00:29:48.462062 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.462044 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.464261 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.464238 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 17 00:29:48.464365 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.464283 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 17 00:29:48.464433 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.464389 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-55r97" Sep 17 00:29:48.464609 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.464591 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 17 00:29:48.465079 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.465061 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 17 00:29:48.465397 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.465379 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-47w5m\"" Sep 17 00:29:48.465542 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.465526 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 17 00:29:48.465710 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.465693 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 17 00:29:48.466089 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466070 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-device-dir\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.466143 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466123 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvr5s\" (UniqueName: \"kubernetes.io/projected/8bdcdeea-d660-4ba8-8186-a2c495239b90-kube-api-access-zvr5s\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.466195 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466154 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 17 00:29:48.466195 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466172 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq56c\" (UniqueName: \"kubernetes.io/projected/6155b312-55da-43b1-9b1f-4d51dfbb0d42-kube-api-access-gq56c\") pod \"network-metrics-daemon-rh4cz\" (UID: \"6155b312-55da-43b1-9b1f-4d51dfbb0d42\") " pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:29:48.466288 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466209 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8bdcdeea-d660-4ba8-8186-a2c495239b90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.466288 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466236 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8bdcdeea-d660-4ba8-8186-a2c495239b90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.466288 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466259 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kt69v\"" Sep 17 00:29:48.466433 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466312 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8ksg\" (UniqueName: \"kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg\") pod \"network-check-target-xv4br\" (UID: \"c59a8ddf-3715-4032-a1d4-2f01bd95aed1\") " pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:29:48.466433 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466356 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-run-systemd\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.466433 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466384 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-node-log\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.466433 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466409 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-run-ovn-kubernetes\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.466433 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466424 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs\") pod \"network-metrics-daemon-rh4cz\" (UID: \"6155b312-55da-43b1-9b1f-4d51dfbb0d42\") " pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:29:48.466649 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466440 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bdcdeea-d660-4ba8-8186-a2c495239b90-cnibin\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.466649 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466480 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-etc-openvswitch\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.466649 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466545 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-cni-bin\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.466649 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466605 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-cni-netd\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.466649 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466632 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-kubelet\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.466854 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466656 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-var-lib-openvswitch\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.466854 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466658 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 17 00:29:48.466854 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466690 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-run-ovn\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.466854 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466722 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.466854 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466748 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2861d237-812c-450e-aec8-954e0324b4b5-ovn-node-metrics-cert\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.466854 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466787 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2861d237-812c-450e-aec8-954e0324b4b5-ovnkube-script-lib\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.466854 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466808 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-sys-fs\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.467178 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466859 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.467178 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466854 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kw4f\" (UniqueName: \"kubernetes.io/projected/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-kube-api-access-9kw4f\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.467178 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466893 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4m4m\" (UniqueName: \"kubernetes.io/projected/ddb33aca-e881-421e-b64f-ca6aaa46298d-kube-api-access-c4m4m\") pod \"node-ca-d7jth\" (UID: \"ddb33aca-e881-421e-b64f-ca6aaa46298d\") " pod="openshift-image-registry/node-ca-d7jth" Sep 17 00:29:48.467178 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466920 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bdcdeea-d660-4ba8-8186-a2c495239b90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.467178 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466946 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-slash\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.467178 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466968 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-run-openvswitch\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.467178 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.466991 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2861d237-812c-450e-aec8-954e0324b4b5-env-overrides\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.467178 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.467019 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dda66642-c725-4b58-82ac-283b33d09dab-dbus\") pod \"global-pull-secret-syncer-bs8f6\" (UID: \"dda66642-c725-4b58-82ac-283b33d09dab\") " pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:29:48.467178 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.467063 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bdcdeea-d660-4ba8-8186-a2c495239b90-cni-binary-copy\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.467178 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.467102 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-run-netns\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.467178 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.467124 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncg95\" (UniqueName: \"kubernetes.io/projected/2861d237-812c-450e-aec8-954e0324b4b5-kube-api-access-ncg95\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.467579 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.467189 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret\") pod \"global-pull-secret-syncer-bs8f6\" (UID: \"dda66642-c725-4b58-82ac-283b33d09dab\") " pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:29:48.467579 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.467214 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bdcdeea-d660-4ba8-8186-a2c495239b90-system-cni-dir\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.467579 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.467249 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-systemd-units\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.467579 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.467275 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-log-socket\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.467579 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.467299 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dda66642-c725-4b58-82ac-283b33d09dab-kubelet-config\") pod \"global-pull-secret-syncer-bs8f6\" (UID: \"dda66642-c725-4b58-82ac-283b33d09dab\") " pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:29:48.467579 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.467322 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.467579 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.467350 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.467579 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.467390 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ddb33aca-e881-421e-b64f-ca6aaa46298d-host\") pod \"node-ca-d7jth\" (UID: \"ddb33aca-e881-421e-b64f-ca6aaa46298d\") " pod="openshift-image-registry/node-ca-d7jth" Sep 17 00:29:48.467579 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.467412 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ddb33aca-e881-421e-b64f-ca6aaa46298d-serviceca\") pod \"node-ca-d7jth\" (UID: \"ddb33aca-e881-421e-b64f-ca6aaa46298d\") " pod="openshift-image-registry/node-ca-d7jth" Sep 17 00:29:48.467579 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.467439 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2861d237-812c-450e-aec8-954e0324b4b5-ovnkube-config\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.467579 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.467470 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-socket-dir\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.467579 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.467532 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-registration-dir\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.468619 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.468596 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wh4r5\"" Sep 17 00:29:48.468710 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.468617 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 17 00:29:48.468766 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.468716 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 17 00:29:48.469339 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.469309 2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 17 00:29:48.469457 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.469346 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kzvtd" Sep 17 00:29:48.470933 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.470914 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 17 00:29:48.471075 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.471056 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 17 00:29:48.471208 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.471195 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-n4jsj\"" Sep 17 00:29:48.471770 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.471735 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.473429 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.473411 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 00:29:48.473563 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.473492 2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-177.ec2.internal" Sep 17 00:29:48.473626 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.473602 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bdws5\"" Sep 17 00:29:48.473691 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.473658 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 17 00:29:48.474088 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.473968 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal"] Sep 17 00:29:48.474088 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.474038 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hpwmd" Sep 17 00:29:48.478001 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.476246 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 17 00:29:48.478001 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.476327 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tmsdl\"" Sep 17 00:29:48.478001 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.476398 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 17 00:29:48.478001 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.476344 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 17 00:29:48.484380 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.484361 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-177.ec2.internal"] Sep 17 00:29:48.484468 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.484361 2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 17 00:29:48.485639 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.485596 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 00:24:48 +0000 UTC" deadline="2028-02-24 04:33:30.31257566 +0000 UTC" Sep 17 00:29:48.485706 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.485638 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12604h3m41.826940541s" Sep 17 00:29:48.488246 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.488231 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vsh2s" Sep 17 00:29:48.495526 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.495487 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vsh2s" Sep 17 00:29:48.561843 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.561825 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 17 00:29:48.567995 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.567976 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bdcdeea-d660-4ba8-8186-a2c495239b90-cni-binary-copy\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.568074 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568010 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-run-netns\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.568074 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568037 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret\") pod \"global-pull-secret-syncer-bs8f6\" (UID: \"dda66642-c725-4b58-82ac-283b33d09dab\") " pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:29:48.568074 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568063 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-systemd-units\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.568199 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568084 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-run-netns\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.568199 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568089 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-run\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.568199 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568151 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-systemd-units\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.568311 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:48.568195 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 00:29:48.568311 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568202 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-cni-binary-copy\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.568311 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568238 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zh5n\" (UniqueName: \"kubernetes.io/projected/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-kube-api-access-8zh5n\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.568311 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:48.568274 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret podName:dda66642-c725-4b58-82ac-283b33d09dab nodeName:}" failed. No retries permitted until 2026-09-17 00:29:49.068242704 +0000 UTC m=+1.998876183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret") pod "global-pull-secret-syncer-bs8f6" (UID: "dda66642-c725-4b58-82ac-283b33d09dab") : object "kube-system"/"original-pull-secret" not registered Sep 17 00:29:48.568311 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568292 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dda66642-c725-4b58-82ac-283b33d09dab-kubelet-config\") pod \"global-pull-secret-syncer-bs8f6\" (UID: \"dda66642-c725-4b58-82ac-283b33d09dab\") " pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:29:48.568311 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568308 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.568606 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568333 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ddb33aca-e881-421e-b64f-ca6aaa46298d-serviceca\") pod \"node-ca-d7jth\" (UID: \"ddb33aca-e881-421e-b64f-ca6aaa46298d\") " pod="openshift-image-registry/node-ca-d7jth" Sep 17 00:29:48.568606 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568355 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2861d237-812c-450e-aec8-954e0324b4b5-ovnkube-config\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.568606 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568358 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dda66642-c725-4b58-82ac-283b33d09dab-kubelet-config\") pod \"global-pull-secret-syncer-bs8f6\" (UID: \"dda66642-c725-4b58-82ac-283b33d09dab\") " pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:29:48.568606 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568379 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-etc-modprobe-d\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.568606 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568403 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-etc-kubernetes\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.568606 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568428 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-host-run-k8s-cni-cncf-io\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.568606 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568462 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d95051d-2d7f-4b93-bf06-1114d4794251-iptables-alerter-script\") pod \"iptables-alerter-hpwmd\" (UID: \"2d95051d-2d7f-4b93-bf06-1114d4794251\") " pod="openshift-network-operator/iptables-alerter-hpwmd" Sep 17 00:29:48.568606 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568493 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-registration-dir\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.568606 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568537 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-device-dir\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.568606 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568554 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bdcdeea-d660-4ba8-8186-a2c495239b90-cni-binary-copy\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.569037 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568616 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-registration-dir\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.569037 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568668 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-device-dir\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.569037 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568699 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3073465-da94-43fa-bd66-bc3158a4b719-tmp\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.569037 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568687 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.569037 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568731 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-host-var-lib-cni-multus\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.569037 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568761 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-run-systemd\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.569037 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568788 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-cni-bin\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.569037 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568802 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-run-systemd\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.569037 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568813 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-cni-netd\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.569037 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568839 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs\") pod \"network-metrics-daemon-rh4cz\" (UID: \"6155b312-55da-43b1-9b1f-4d51dfbb0d42\") " pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:29:48.569037 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568870 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-cni-netd\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.569037 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568880 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bdcdeea-d660-4ba8-8186-a2c495239b90-cnibin\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.569037 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568895 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ddb33aca-e881-421e-b64f-ca6aaa46298d-serviceca\") pod \"node-ca-d7jth\" (UID: \"ddb33aca-e881-421e-b64f-ca6aaa46298d\") " pod="openshift-image-registry/node-ca-d7jth" Sep 17 00:29:48.569037 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568855 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-cni-bin\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.569037 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568918 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-etc-sysctl-d\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.569037 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:48.568927 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 00:29:48.569037 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568931 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bdcdeea-d660-4ba8-8186-a2c495239b90-cnibin\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.569037 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568947 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-sys\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.569794 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568963 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2861d237-812c-450e-aec8-954e0324b4b5-ovnkube-config\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.569794 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.568973 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b3073465-da94-43fa-bd66-bc3158a4b719-etc-tuned\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.569794 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:48.568997 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs podName:6155b312-55da-43b1-9b1f-4d51dfbb0d42 nodeName:}" failed. No retries permitted until 2026-09-17 00:29:49.068982339 +0000 UTC m=+1.999615804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs") pod "network-metrics-daemon-rh4cz" (UID: "6155b312-55da-43b1-9b1f-4d51dfbb0d42") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 00:29:48.569794 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569011 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-hostroot\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.569794 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569038 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-etc-kubernetes\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.569794 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569053 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d95051d-2d7f-4b93-bf06-1114d4794251-host-slash\") pod \"iptables-alerter-hpwmd\" (UID: \"2d95051d-2d7f-4b93-bf06-1114d4794251\") " pod="openshift-network-operator/iptables-alerter-hpwmd" Sep 17 00:29:48.569794 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569069 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-var-lib-openvswitch\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.569794 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-run-ovn\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.569794 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569110 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2861d237-812c-450e-aec8-954e0324b4b5-ovn-node-metrics-cert\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.569794 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569132 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2861d237-812c-450e-aec8-954e0324b4b5-ovnkube-script-lib\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.569794 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569168 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-var-lib-openvswitch\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.569794 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569172 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-multus-cni-dir\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.569794 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569212 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-host-var-lib-kubelet\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.569794 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569286 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d6d5c16-9a78-4f6e-b33b-bd62d16302b0-hosts-file\") pod \"node-resolver-kzvtd\" (UID: \"6d6d5c16-9a78-4f6e-b33b-bd62d16302b0\") " pod="openshift-dns/node-resolver-kzvtd" Sep 17 00:29:48.569794 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569304 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bdcdeea-d660-4ba8-8186-a2c495239b90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.569794 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569334 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-run-ovn\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.570454 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569352 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-slash\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.570454 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569380 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-slash\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.570454 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569420 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-run-openvswitch\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.570454 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569451 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-var-lib-kubelet\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.570454 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569453 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bdcdeea-d660-4ba8-8186-a2c495239b90-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.570454 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569475 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-multus-socket-dir-parent\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.570454 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569516 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-run-openvswitch\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.570454 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569541 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dda66642-c725-4b58-82ac-283b33d09dab-dbus\") pod \"global-pull-secret-syncer-bs8f6\" (UID: \"dda66642-c725-4b58-82ac-283b33d09dab\") " pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:29:48.570454 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569564 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncg95\" (UniqueName: \"kubernetes.io/projected/2861d237-812c-450e-aec8-954e0324b4b5-kube-api-access-ncg95\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.570454 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569585 2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 17 00:29:48.570454 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569599 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-etc-sysconfig\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.570454 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569635 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dda66642-c725-4b58-82ac-283b33d09dab-dbus\") pod \"global-pull-secret-syncer-bs8f6\" (UID: \"dda66642-c725-4b58-82ac-283b33d09dab\") " pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:29:48.570454 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569646 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2861d237-812c-450e-aec8-954e0324b4b5-ovnkube-script-lib\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.570454 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569651 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-host\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.570454 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569678 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bdcdeea-d660-4ba8-8186-a2c495239b90-system-cni-dir\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.570454 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569703 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-log-socket\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.570454 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569710 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bdcdeea-d660-4ba8-8186-a2c495239b90-system-cni-dir\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.571129 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569725 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ec239ad8-714d-4805-b7c4-a0159e28b289-agent-certs\") pod \"konnectivity-agent-55r97\" (UID: \"ec239ad8-714d-4805-b7c4-a0159e28b289\") " pod="kube-system/konnectivity-agent-55r97" Sep 17 00:29:48.571129 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569734 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-log-socket\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.571129 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569777 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-multus-conf-dir\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.571129 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569803 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.571129 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569825 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ddb33aca-e881-421e-b64f-ca6aaa46298d-host\") pod \"node-ca-d7jth\" (UID: \"ddb33aca-e881-421e-b64f-ca6aaa46298d\") " pod="openshift-image-registry/node-ca-d7jth" Sep 17 00:29:48.571129 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569842 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-etc-systemd\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.571129 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569857 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-socket-dir\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.571129 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569875 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.571129 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569881 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ddb33aca-e881-421e-b64f-ca6aaa46298d-host\") pod \"node-ca-d7jth\" (UID: \"ddb33aca-e881-421e-b64f-ca6aaa46298d\") " pod="openshift-image-registry/node-ca-d7jth" Sep 17 00:29:48.571129 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569884 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvr5s\" (UniqueName: \"kubernetes.io/projected/8bdcdeea-d660-4ba8-8186-a2c495239b90-kube-api-access-zvr5s\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.571129 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569914 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-etc-sysctl-conf\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.571129 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.569973 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-socket-dir\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.571129 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570003 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-cnibin\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.571129 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570022 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-host-var-lib-cni-bin\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.571129 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570039 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq56c\" (UniqueName: \"kubernetes.io/projected/6155b312-55da-43b1-9b1f-4d51dfbb0d42-kube-api-access-gq56c\") pod \"network-metrics-daemon-rh4cz\" (UID: \"6155b312-55da-43b1-9b1f-4d51dfbb0d42\") " pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:29:48.571129 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570055 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8bdcdeea-d660-4ba8-8186-a2c495239b90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.571129 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570070 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8bdcdeea-d660-4ba8-8186-a2c495239b90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.571666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570086 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ksg\" (UniqueName: \"kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg\") pod \"network-check-target-xv4br\" (UID: \"c59a8ddf-3715-4032-a1d4-2f01bd95aed1\") " pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:29:48.571666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570102 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-node-log\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.571666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570117 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-run-ovn-kubernetes\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.571666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570137 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-multus-daemon-config\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.571666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570152 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-etc-openvswitch\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.571666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570166 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-lib-modules\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.571666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570183 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-system-cni-dir\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.571666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570209 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-host-run-multus-certs\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.571666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570233 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d6d5c16-9a78-4f6e-b33b-bd62d16302b0-tmp-dir\") pod \"node-resolver-kzvtd\" (UID: \"6d6d5c16-9a78-4f6e-b33b-bd62d16302b0\") " pod="openshift-dns/node-resolver-kzvtd" Sep 17 00:29:48.571666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570253 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-run-ovn-kubernetes\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.571666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570259 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ql82\" (UniqueName: \"kubernetes.io/projected/2d95051d-2d7f-4b93-bf06-1114d4794251-kube-api-access-9ql82\") pod \"iptables-alerter-hpwmd\" (UID: \"2d95051d-2d7f-4b93-bf06-1114d4794251\") " pod="openshift-network-operator/iptables-alerter-hpwmd" Sep 17 00:29:48.571666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570191 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-node-log\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.571666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570298 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-kubelet\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.571666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570308 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-etc-openvswitch\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.571666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570326 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.571666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570357 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-kubelet\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.571666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570368 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ec239ad8-714d-4805-b7c4-a0159e28b289-konnectivity-ca\") pod \"konnectivity-agent-55r97\" (UID: \"ec239ad8-714d-4805-b7c4-a0159e28b289\") " pod="kube-system/konnectivity-agent-55r97" Sep 17 00:29:48.572159 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570390 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-host-run-netns\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.572159 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570402 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2861d237-812c-450e-aec8-954e0324b4b5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.572159 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570413 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-sys-fs\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.572159 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570439 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kw4f\" (UniqueName: \"kubernetes.io/projected/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-kube-api-access-9kw4f\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.572159 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570462 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4m4m\" (UniqueName: \"kubernetes.io/projected/ddb33aca-e881-421e-b64f-ca6aaa46298d-kube-api-access-c4m4m\") pod \"node-ca-d7jth\" (UID: \"ddb33aca-e881-421e-b64f-ca6aaa46298d\") " pod="openshift-image-registry/node-ca-d7jth" Sep 17 00:29:48.572159 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570472 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-sys-fs\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.572159 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570492 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2861d237-812c-450e-aec8-954e0324b4b5-env-overrides\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.572159 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570555 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46nxf\" (UniqueName: \"kubernetes.io/projected/b3073465-da94-43fa-bd66-bc3158a4b719-kube-api-access-46nxf\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.572159 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570581 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngx88\" (UniqueName: \"kubernetes.io/projected/6d6d5c16-9a78-4f6e-b33b-bd62d16302b0-kube-api-access-ngx88\") pod \"node-resolver-kzvtd\" (UID: \"6d6d5c16-9a78-4f6e-b33b-bd62d16302b0\") " pod="openshift-dns/node-resolver-kzvtd" Sep 17 00:29:48.572159 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570597 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8bdcdeea-d660-4ba8-8186-a2c495239b90-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.572159 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570676 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8bdcdeea-d660-4ba8-8186-a2c495239b90-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.572159 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.570866 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2861d237-812c-450e-aec8-954e0324b4b5-env-overrides\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.573057 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.573040 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2861d237-812c-450e-aec8-954e0324b4b5-ovn-node-metrics-cert\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.574839 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:48.574825 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 00:29:48.574894 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:48.574841 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 00:29:48.574894 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:48.574851 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m8ksg for pod openshift-network-diagnostics/network-check-target-xv4br: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 00:29:48.574960 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:48.574908 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg podName:c59a8ddf-3715-4032-a1d4-2f01bd95aed1 nodeName:}" failed. No retries permitted until 2026-09-17 00:29:49.074896589 +0000 UTC m=+2.005530067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m8ksg" (UniqueName: "kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg") pod "network-check-target-xv4br" (UID: "c59a8ddf-3715-4032-a1d4-2f01bd95aed1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 00:29:48.577492 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.577473 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq56c\" (UniqueName: \"kubernetes.io/projected/6155b312-55da-43b1-9b1f-4d51dfbb0d42-kube-api-access-gq56c\") pod \"network-metrics-daemon-rh4cz\" (UID: \"6155b312-55da-43b1-9b1f-4d51dfbb0d42\") " pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:29:48.577595 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.577530 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncg95\" (UniqueName: \"kubernetes.io/projected/2861d237-812c-450e-aec8-954e0324b4b5-kube-api-access-ncg95\") pod \"ovnkube-node-wzmbr\" (UID: \"2861d237-812c-450e-aec8-954e0324b4b5\") " pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.577659 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.577638 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvr5s\" (UniqueName: \"kubernetes.io/projected/8bdcdeea-d660-4ba8-8186-a2c495239b90-kube-api-access-zvr5s\") pod \"multus-additional-cni-plugins-6pld9\" (UID: \"8bdcdeea-d660-4ba8-8186-a2c495239b90\") " pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.578084 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.578064 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kw4f\" (UniqueName: \"kubernetes.io/projected/46d0a8fe-92bc-4057-a0ad-5f19fcfde71b-kube-api-access-9kw4f\") pod \"aws-ebs-csi-driver-node-pc6fm\" (UID: \"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.578459 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.578441 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4m4m\" (UniqueName: \"kubernetes.io/projected/ddb33aca-e881-421e-b64f-ca6aaa46298d-kube-api-access-c4m4m\") pod \"node-ca-d7jth\" (UID: \"ddb33aca-e881-421e-b64f-ca6aaa46298d\") " pod="openshift-image-registry/node-ca-d7jth" Sep 17 00:29:48.670825 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.670799 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ec239ad8-714d-4805-b7c4-a0159e28b289-agent-certs\") pod \"konnectivity-agent-55r97\" (UID: \"ec239ad8-714d-4805-b7c4-a0159e28b289\") " pod="kube-system/konnectivity-agent-55r97" Sep 17 00:29:48.670825 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.670828 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-multus-conf-dir\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.671028 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.670844 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-etc-systemd\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.671028 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.670862 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-etc-sysctl-conf\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.671028 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.670937 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-etc-systemd\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.671028 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.670959 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-etc-sysctl-conf\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.671028 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.670986 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-cnibin\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.671028 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.670988 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-multus-conf-dir\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.671028 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671001 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-host-var-lib-cni-bin\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.671311 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671033 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-host-var-lib-cni-bin\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.671311 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671048 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-multus-daemon-config\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.671311 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671067 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-lib-modules\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.671311 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671083 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-system-cni-dir\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.671311 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671080 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-cnibin\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.671311 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671103 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-host-run-multus-certs\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.671311 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671143 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-host-run-multus-certs\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.671311 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671169 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d6d5c16-9a78-4f6e-b33b-bd62d16302b0-tmp-dir\") pod \"node-resolver-kzvtd\" (UID: \"6d6d5c16-9a78-4f6e-b33b-bd62d16302b0\") " pod="openshift-dns/node-resolver-kzvtd" Sep 17 00:29:48.671311 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671191 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ql82\" (UniqueName: \"kubernetes.io/projected/2d95051d-2d7f-4b93-bf06-1114d4794251-kube-api-access-9ql82\") pod \"iptables-alerter-hpwmd\" (UID: \"2d95051d-2d7f-4b93-bf06-1114d4794251\") " pod="openshift-network-operator/iptables-alerter-hpwmd" Sep 17 00:29:48.671311 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671212 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-system-cni-dir\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.671311 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671222 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ec239ad8-714d-4805-b7c4-a0159e28b289-konnectivity-ca\") pod \"konnectivity-agent-55r97\" (UID: \"ec239ad8-714d-4805-b7c4-a0159e28b289\") " pod="kube-system/konnectivity-agent-55r97" Sep 17 00:29:48.671311 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671251 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-lib-modules\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.671311 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671272 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-host-run-netns\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.671311 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671305 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46nxf\" (UniqueName: \"kubernetes.io/projected/b3073465-da94-43fa-bd66-bc3158a4b719-kube-api-access-46nxf\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.672023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671330 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngx88\" (UniqueName: \"kubernetes.io/projected/6d6d5c16-9a78-4f6e-b33b-bd62d16302b0-kube-api-access-ngx88\") pod \"node-resolver-kzvtd\" (UID: \"6d6d5c16-9a78-4f6e-b33b-bd62d16302b0\") " pod="openshift-dns/node-resolver-kzvtd" Sep 17 00:29:48.672023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671372 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-run\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.672023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671395 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-cni-binary-copy\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.672023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671418 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zh5n\" (UniqueName: \"kubernetes.io/projected/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-kube-api-access-8zh5n\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.672023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671444 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-etc-modprobe-d\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.672023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671514 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-etc-kubernetes\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.672023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671608 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-etc-modprobe-d\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.672023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671613 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-run\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.672023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671629 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-multus-daemon-config\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.672023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671638 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-host-run-k8s-cni-cncf-io\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.672023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671665 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-host-run-netns\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.672023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671669 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-host-run-k8s-cni-cncf-io\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.672023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671690 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d95051d-2d7f-4b93-bf06-1114d4794251-iptables-alerter-script\") pod \"iptables-alerter-hpwmd\" (UID: \"2d95051d-2d7f-4b93-bf06-1114d4794251\") " pod="openshift-network-operator/iptables-alerter-hpwmd" Sep 17 00:29:48.672023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671695 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ec239ad8-714d-4805-b7c4-a0159e28b289-konnectivity-ca\") pod \"konnectivity-agent-55r97\" (UID: \"ec239ad8-714d-4805-b7c4-a0159e28b289\") " pod="kube-system/konnectivity-agent-55r97" Sep 17 00:29:48.672023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671699 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d6d5c16-9a78-4f6e-b33b-bd62d16302b0-tmp-dir\") pod \"node-resolver-kzvtd\" (UID: \"6d6d5c16-9a78-4f6e-b33b-bd62d16302b0\") " pod="openshift-dns/node-resolver-kzvtd" Sep 17 00:29:48.672023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671718 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3073465-da94-43fa-bd66-bc3158a4b719-tmp\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.672023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671776 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-etc-kubernetes\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.672023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671885 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-host-var-lib-cni-multus\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.672808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671936 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-etc-sysctl-d\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.672808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671964 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-sys\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.672808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671987 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b3073465-da94-43fa-bd66-bc3158a4b719-etc-tuned\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.672808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.671992 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-host-var-lib-cni-multus\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.672808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672011 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-hostroot\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.672808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672050 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-etc-kubernetes\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.672808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672072 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d95051d-2d7f-4b93-bf06-1114d4794251-host-slash\") pod \"iptables-alerter-hpwmd\" (UID: \"2d95051d-2d7f-4b93-bf06-1114d4794251\") " pod="openshift-network-operator/iptables-alerter-hpwmd" Sep 17 00:29:48.672808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672091 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-sys\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.672808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672100 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-multus-cni-dir\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.672808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672126 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-host-var-lib-kubelet\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.672808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672148 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d6d5c16-9a78-4f6e-b33b-bd62d16302b0-hosts-file\") pod \"node-resolver-kzvtd\" (UID: \"6d6d5c16-9a78-4f6e-b33b-bd62d16302b0\") " pod="openshift-dns/node-resolver-kzvtd" Sep 17 00:29:48.672808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672154 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-cni-binary-copy\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.672808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672173 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-var-lib-kubelet\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.672808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672179 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-etc-sysctl-d\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.672808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672197 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-multus-socket-dir-parent\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.672808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672225 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-etc-sysconfig\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.672808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672223 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-hostroot\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.672808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672224 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-etc-kubernetes\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.673310 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672276 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-var-lib-kubelet\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.673310 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672287 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-multus-cni-dir\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.673310 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672297 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-multus-socket-dir-parent\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.673310 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672323 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-host\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.673310 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672333 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2d95051d-2d7f-4b93-bf06-1114d4794251-iptables-alerter-script\") pod \"iptables-alerter-hpwmd\" (UID: \"2d95051d-2d7f-4b93-bf06-1114d4794251\") " pod="openshift-network-operator/iptables-alerter-hpwmd" Sep 17 00:29:48.673310 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672349 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d95051d-2d7f-4b93-bf06-1114d4794251-host-slash\") pod \"iptables-alerter-hpwmd\" (UID: \"2d95051d-2d7f-4b93-bf06-1114d4794251\") " pod="openshift-network-operator/iptables-alerter-hpwmd" Sep 17 00:29:48.673310 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672352 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-etc-sysconfig\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.673310 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672321 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-host-var-lib-kubelet\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.673310 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672399 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3073465-da94-43fa-bd66-bc3158a4b719-host\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.673310 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.672403 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6d6d5c16-9a78-4f6e-b33b-bd62d16302b0-hosts-file\") pod \"node-resolver-kzvtd\" (UID: \"6d6d5c16-9a78-4f6e-b33b-bd62d16302b0\") " pod="openshift-dns/node-resolver-kzvtd" Sep 17 00:29:48.673626 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.673401 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ec239ad8-714d-4805-b7c4-a0159e28b289-agent-certs\") pod \"konnectivity-agent-55r97\" (UID: \"ec239ad8-714d-4805-b7c4-a0159e28b289\") " pod="kube-system/konnectivity-agent-55r97" Sep 17 00:29:48.673894 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.673876 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3073465-da94-43fa-bd66-bc3158a4b719-tmp\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.674286 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.674271 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b3073465-da94-43fa-bd66-bc3158a4b719-etc-tuned\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.678807 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.678787 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngx88\" (UniqueName: \"kubernetes.io/projected/6d6d5c16-9a78-4f6e-b33b-bd62d16302b0-kube-api-access-ngx88\") pod \"node-resolver-kzvtd\" (UID: \"6d6d5c16-9a78-4f6e-b33b-bd62d16302b0\") " pod="openshift-dns/node-resolver-kzvtd" Sep 17 00:29:48.678987 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.678971 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46nxf\" (UniqueName: \"kubernetes.io/projected/b3073465-da94-43fa-bd66-bc3158a4b719-kube-api-access-46nxf\") pod \"tuned-vnqww\" (UID: \"b3073465-da94-43fa-bd66-bc3158a4b719\") " pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.679114 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.679090 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ql82\" (UniqueName: \"kubernetes.io/projected/2d95051d-2d7f-4b93-bf06-1114d4794251-kube-api-access-9ql82\") pod \"iptables-alerter-hpwmd\" (UID: \"2d95051d-2d7f-4b93-bf06-1114d4794251\") " pod="openshift-network-operator/iptables-alerter-hpwmd" Sep 17 00:29:48.679545 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.679518 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zh5n\" (UniqueName: \"kubernetes.io/projected/29c30e84-c3d7-4e23-a4dd-efc80b57ccb5-kube-api-access-8zh5n\") pod \"multus-fmx98\" (UID: \"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5\") " pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.721842 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:48.721812 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d3f85df08cf592386ff2d120f060eb.slice/crio-08b33978db7e77b2237eb729a8b9446359eed2929a4c948ee44cff481b39dd83 WatchSource:0}: Error finding container 08b33978db7e77b2237eb729a8b9446359eed2929a4c948ee44cff481b39dd83: Status 404 returned error can't find the container with id 08b33978db7e77b2237eb729a8b9446359eed2929a4c948ee44cff481b39dd83 Sep 17 00:29:48.722094 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:48.722076 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81937eb52bebe19931264bfcfb8d032c.slice/crio-f44867cda73e19684ad12f25b5eb520f5d9b5c634c560834b5ffbf07d87e55fc WatchSource:0}: Error finding container f44867cda73e19684ad12f25b5eb520f5d9b5c634c560834b5ffbf07d87e55fc: Status 404 returned error can't find the container with id f44867cda73e19684ad12f25b5eb520f5d9b5c634c560834b5ffbf07d87e55fc Sep 17 00:29:48.725894 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.725881 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 00:29:48.773973 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.773951 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" Sep 17 00:29:48.782853 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:48.782832 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d0a8fe_92bc_4057_a0ad_5f19fcfde71b.slice/crio-ae934a074e2af0760a2b3e48c3be0553f4a186117d0a8a02b56a090418c3f957 WatchSource:0}: Error finding container ae934a074e2af0760a2b3e48c3be0553f4a186117d0a8a02b56a090418c3f957: Status 404 returned error can't find the container with id ae934a074e2af0760a2b3e48c3be0553f4a186117d0a8a02b56a090418c3f957 Sep 17 00:29:48.788464 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.788449 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d7jth" Sep 17 00:29:48.794360 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.794229 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6pld9" Sep 17 00:29:48.796439 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:48.796419 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb33aca_e881_421e_b64f_ca6aaa46298d.slice/crio-1c7e3e27796431e26e2384c5ba665591ff185cc6cdd98f52c8bff6395bd7c4d3 WatchSource:0}: Error finding container 1c7e3e27796431e26e2384c5ba665591ff185cc6cdd98f52c8bff6395bd7c4d3: Status 404 returned error can't find the container with id 1c7e3e27796431e26e2384c5ba665591ff185cc6cdd98f52c8bff6395bd7c4d3 Sep 17 00:29:48.804557 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:48.804538 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bdcdeea_d660_4ba8_8186_a2c495239b90.slice/crio-fa102ecc54ce0cdea530acecc3191f11058b663a70a5362a0c7cc089673e1fac WatchSource:0}: Error finding container fa102ecc54ce0cdea530acecc3191f11058b663a70a5362a0c7cc089673e1fac: Status 404 returned error can't find the container with id fa102ecc54ce0cdea530acecc3191f11058b663a70a5362a0c7cc089673e1fac Sep 17 00:29:48.825881 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.825854 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:29:48.834168 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:48.834147 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2861d237_812c_450e_aec8_954e0324b4b5.slice/crio-3f66a59c066f87adf4ab2df2f95d6dc88a918ebcd63db3f4049bcca645545895 WatchSource:0}: Error finding container 3f66a59c066f87adf4ab2df2f95d6dc88a918ebcd63db3f4049bcca645545895: Status 404 returned error can't find the container with id 3f66a59c066f87adf4ab2df2f95d6dc88a918ebcd63db3f4049bcca645545895 Sep 17 00:29:48.840368 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.840347 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-55r97" Sep 17 00:29:48.847745 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:48.847726 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec239ad8_714d_4805_b7c4_a0159e28b289.slice/crio-3e2775b4dfd872f9d029d3618965ccb4fbd78539423623b4847280f1da5e86f7 WatchSource:0}: Error finding container 3e2775b4dfd872f9d029d3618965ccb4fbd78539423623b4847280f1da5e86f7: Status 404 returned error can't find the container with id 3e2775b4dfd872f9d029d3618965ccb4fbd78539423623b4847280f1da5e86f7 Sep 17 00:29:48.857007 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.856992 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vnqww" Sep 17 00:29:48.863708 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.863690 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kzvtd" Sep 17 00:29:48.864303 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:48.864285 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3073465_da94_43fa_bd66_bc3158a4b719.slice/crio-09143b25294a25655534b82a61eac6de86185fdfd91991c1a9ad91a7734fd518 WatchSource:0}: Error finding container 09143b25294a25655534b82a61eac6de86185fdfd91991c1a9ad91a7734fd518: Status 404 returned error can't find the container with id 09143b25294a25655534b82a61eac6de86185fdfd91991c1a9ad91a7734fd518 Sep 17 00:29:48.868396 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.868377 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fmx98" Sep 17 00:29:48.872112 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:48.872096 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6d5c16_9a78_4f6e_b33b_bd62d16302b0.slice/crio-77a6db9c130d26a04ca827c1be26c087090ba4626cb7f70de5e6a52d05556906 WatchSource:0}: Error finding container 77a6db9c130d26a04ca827c1be26c087090ba4626cb7f70de5e6a52d05556906: Status 404 returned error can't find the container with id 77a6db9c130d26a04ca827c1be26c087090ba4626cb7f70de5e6a52d05556906 Sep 17 00:29:48.876612 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:48.876593 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c30e84_c3d7_4e23_a4dd_efc80b57ccb5.slice/crio-443664d1685db2e80f01ca8840b2eda2e174404f8d74a908f3b8742585b3313c WatchSource:0}: Error finding container 443664d1685db2e80f01ca8840b2eda2e174404f8d74a908f3b8742585b3313c: Status 404 returned error can't find the container with id 443664d1685db2e80f01ca8840b2eda2e174404f8d74a908f3b8742585b3313c Sep 17 00:29:48.880938 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.880921 2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 00:29:48.889371 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.889347 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hpwmd" Sep 17 00:29:48.892634 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:48.892616 2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 00:29:48.897698 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:29:48.897682 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d95051d_2d7f_4b93_bf06_1114d4794251.slice/crio-dc132ceeb39a9ec5de72e9edcb1dbfebe686bb89c6009d36f894d1b44e8f423e WatchSource:0}: Error finding container dc132ceeb39a9ec5de72e9edcb1dbfebe686bb89c6009d36f894d1b44e8f423e: Status 404 returned error can't find the container with id dc132ceeb39a9ec5de72e9edcb1dbfebe686bb89c6009d36f894d1b44e8f423e Sep 17 00:29:48.899305 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:48.899286 2582 kuberuntime_manager.go:1459] "Unhandled Error" err="container iptables-alerter start failed in pod iptables-alerter-hpwmd_openshift-network-operator(2d95051d-2d7f-4b93-bf06-1114d4794251): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Sep 17 00:29:48.899380 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:48.899314 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-hpwmd" podUID="2d95051d-2d7f-4b93-bf06-1114d4794251" Sep 17 00:29:49.074075 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:49.073872 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret\") pod \"global-pull-secret-syncer-bs8f6\" (UID: \"dda66642-c725-4b58-82ac-283b33d09dab\") " pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:29:49.074075 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:49.074056 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs\") pod \"network-metrics-daemon-rh4cz\" (UID: \"6155b312-55da-43b1-9b1f-4d51dfbb0d42\") " pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:29:49.074222 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:49.074023 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 00:29:49.074222 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:49.074163 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 00:29:49.074222 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:49.074172 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret podName:dda66642-c725-4b58-82ac-283b33d09dab nodeName:}" failed. No retries permitted until 2026-09-17 00:29:50.07415571 +0000 UTC m=+3.004789177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret") pod "global-pull-secret-syncer-bs8f6" (UID: "dda66642-c725-4b58-82ac-283b33d09dab") : object "kube-system"/"original-pull-secret" not registered Sep 17 00:29:49.074222 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:49.074200 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs podName:6155b312-55da-43b1-9b1f-4d51dfbb0d42 nodeName:}" failed. No retries permitted until 2026-09-17 00:29:50.074188986 +0000 UTC m=+3.004822475 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs") pod "network-metrics-daemon-rh4cz" (UID: "6155b312-55da-43b1-9b1f-4d51dfbb0d42") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 00:29:49.175267 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:49.175242 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ksg\" (UniqueName: \"kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg\") pod \"network-check-target-xv4br\" (UID: \"c59a8ddf-3715-4032-a1d4-2f01bd95aed1\") " pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:29:49.175409 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:49.175372 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 00:29:49.175409 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:49.175385 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 00:29:49.175409 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:49.175394 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m8ksg for pod openshift-network-diagnostics/network-check-target-xv4br: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 00:29:49.175539 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:49.175441 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg podName:c59a8ddf-3715-4032-a1d4-2f01bd95aed1 nodeName:}" failed. No retries permitted until 2026-09-17 00:29:50.175428305 +0000 UTC m=+3.106061785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m8ksg" (UniqueName: "kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg") pod "network-check-target-xv4br" (UID: "c59a8ddf-3715-4032-a1d4-2f01bd95aed1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 00:29:49.498927 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:49.498823 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 00:24:49 +0000 UTC" deadline="2028-05-25 16:52:57.902615602 +0000 UTC" Sep 17 00:29:49.498927 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:49.498880 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14800h23m8.403740948s" Sep 17 00:29:49.644744 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:49.644711 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:29:49.644916 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:49.644845 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xv4br" podUID="c59a8ddf-3715-4032-a1d4-2f01bd95aed1" Sep 17 00:29:49.645649 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:49.645624 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:29:49.645777 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:49.645748 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4cz" podUID="6155b312-55da-43b1-9b1f-4d51dfbb0d42" Sep 17 00:29:49.673598 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:49.673532 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-55r97" event={"ID":"ec239ad8-714d-4805-b7c4-a0159e28b289","Type":"ContainerStarted","Data":"3e2775b4dfd872f9d029d3618965ccb4fbd78539423623b4847280f1da5e86f7"} Sep 17 00:29:49.687679 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:49.687645 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pld9" event={"ID":"8bdcdeea-d660-4ba8-8186-a2c495239b90","Type":"ContainerStarted","Data":"fa102ecc54ce0cdea530acecc3191f11058b663a70a5362a0c7cc089673e1fac"} Sep 17 00:29:49.703808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:49.703766 2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 17 00:29:49.710808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:49.710716 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d7jth" event={"ID":"ddb33aca-e881-421e-b64f-ca6aaa46298d","Type":"ContainerStarted","Data":"1c7e3e27796431e26e2384c5ba665591ff185cc6cdd98f52c8bff6395bd7c4d3"} Sep 17 00:29:49.729176 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:49.729147 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hpwmd" event={"ID":"2d95051d-2d7f-4b93-bf06-1114d4794251","Type":"ContainerStarted","Data":"dc132ceeb39a9ec5de72e9edcb1dbfebe686bb89c6009d36f894d1b44e8f423e"} Sep 17 00:29:49.732295 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:49.732263 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"iptables-alerter\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0444a9580f430c7c6b52f78667a80a82198c825805f1f2004fdfdf9aed2b95f5\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-network-operator/iptables-alerter-hpwmd" podUID="2d95051d-2d7f-4b93-bf06-1114d4794251" Sep 17 00:29:49.745274 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:49.745244 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fmx98" event={"ID":"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5","Type":"ContainerStarted","Data":"443664d1685db2e80f01ca8840b2eda2e174404f8d74a908f3b8742585b3313c"} Sep 17 00:29:49.766236 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:49.766162 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" event={"ID":"2861d237-812c-450e-aec8-954e0324b4b5","Type":"ContainerStarted","Data":"3f66a59c066f87adf4ab2df2f95d6dc88a918ebcd63db3f4049bcca645545895"} Sep 17 00:29:49.780147 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:49.780118 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" event={"ID":"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b","Type":"ContainerStarted","Data":"ae934a074e2af0760a2b3e48c3be0553f4a186117d0a8a02b56a090418c3f957"} Sep 17 00:29:49.794693 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:49.794657 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-177.ec2.internal" event={"ID":"81937eb52bebe19931264bfcfb8d032c","Type":"ContainerStarted","Data":"f44867cda73e19684ad12f25b5eb520f5d9b5c634c560834b5ffbf07d87e55fc"} Sep 17 00:29:49.821262 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:49.821228 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal" event={"ID":"73d3f85df08cf592386ff2d120f060eb","Type":"ContainerStarted","Data":"08b33978db7e77b2237eb729a8b9446359eed2929a4c948ee44cff481b39dd83"} Sep 17 00:29:49.841912 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:49.841438 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kzvtd" event={"ID":"6d6d5c16-9a78-4f6e-b33b-bd62d16302b0","Type":"ContainerStarted","Data":"77a6db9c130d26a04ca827c1be26c087090ba4626cb7f70de5e6a52d05556906"} Sep 17 00:29:49.845652 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:49.845045 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vnqww" event={"ID":"b3073465-da94-43fa-bd66-bc3158a4b719","Type":"ContainerStarted","Data":"09143b25294a25655534b82a61eac6de86185fdfd91991c1a9ad91a7734fd518"} Sep 17 00:29:50.083642 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:50.082713 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret\") pod \"global-pull-secret-syncer-bs8f6\" (UID: \"dda66642-c725-4b58-82ac-283b33d09dab\") " pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:29:50.083642 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:50.082768 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs\") pod \"network-metrics-daemon-rh4cz\" (UID: \"6155b312-55da-43b1-9b1f-4d51dfbb0d42\") " pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:29:50.083642 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:50.082891 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 00:29:50.083642 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:50.082955 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs podName:6155b312-55da-43b1-9b1f-4d51dfbb0d42 nodeName:}" failed. No retries permitted until 2026-09-17 00:29:52.082935713 +0000 UTC m=+5.013569194 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs") pod "network-metrics-daemon-rh4cz" (UID: "6155b312-55da-43b1-9b1f-4d51dfbb0d42") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 00:29:50.083642 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:50.083023 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 00:29:50.083642 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:50.083059 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret podName:dda66642-c725-4b58-82ac-283b33d09dab nodeName:}" failed. No retries permitted until 2026-09-17 00:29:52.083046882 +0000 UTC m=+5.013680364 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret") pod "global-pull-secret-syncer-bs8f6" (UID: "dda66642-c725-4b58-82ac-283b33d09dab") : object "kube-system"/"original-pull-secret" not registered Sep 17 00:29:50.183941 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:50.183874 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ksg\" (UniqueName: \"kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg\") pod \"network-check-target-xv4br\" (UID: \"c59a8ddf-3715-4032-a1d4-2f01bd95aed1\") " pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:29:50.184122 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:50.184091 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 00:29:50.184122 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:50.184121 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 00:29:50.184250 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:50.184136 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m8ksg for pod openshift-network-diagnostics/network-check-target-xv4br: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 00:29:50.184250 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:50.184205 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg podName:c59a8ddf-3715-4032-a1d4-2f01bd95aed1 nodeName:}" failed. No retries permitted until 2026-09-17 00:29:52.184184873 +0000 UTC m=+5.114818351 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m8ksg" (UniqueName: "kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg") pod "network-check-target-xv4br" (UID: "c59a8ddf-3715-4032-a1d4-2f01bd95aed1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 00:29:50.500149 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:50.500007 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 00:24:49 +0000 UTC" deadline="2028-05-28 08:49:05.914864408 +0000 UTC" Sep 17 00:29:50.500149 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:50.500069 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14864h19m15.414799761s" Sep 17 00:29:50.644901 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:50.644862 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:29:50.645103 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:50.645011 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bs8f6" podUID="dda66642-c725-4b58-82ac-283b33d09dab" Sep 17 00:29:51.647165 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:51.647078 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:29:51.647683 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:51.647218 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xv4br" podUID="c59a8ddf-3715-4032-a1d4-2f01bd95aed1" Sep 17 00:29:51.647901 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:51.647880 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:29:51.648028 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:51.648006 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4cz" podUID="6155b312-55da-43b1-9b1f-4d51dfbb0d42" Sep 17 00:29:52.102267 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:52.102161 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret\") pod \"global-pull-secret-syncer-bs8f6\" (UID: \"dda66642-c725-4b58-82ac-283b33d09dab\") " pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:29:52.102267 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:52.102219 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs\") pod \"network-metrics-daemon-rh4cz\" (UID: \"6155b312-55da-43b1-9b1f-4d51dfbb0d42\") " pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:29:52.102551 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:52.102434 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 00:29:52.102551 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:52.102508 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs podName:6155b312-55da-43b1-9b1f-4d51dfbb0d42 nodeName:}" failed. No retries permitted until 2026-09-17 00:29:56.102486166 +0000 UTC m=+9.033119648 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs") pod "network-metrics-daemon-rh4cz" (UID: "6155b312-55da-43b1-9b1f-4d51dfbb0d42") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 00:29:52.102712 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:52.102674 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 00:29:52.102769 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:52.102716 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret podName:dda66642-c725-4b58-82ac-283b33d09dab nodeName:}" failed. No retries permitted until 2026-09-17 00:29:56.102702926 +0000 UTC m=+9.033336411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret") pod "global-pull-secret-syncer-bs8f6" (UID: "dda66642-c725-4b58-82ac-283b33d09dab") : object "kube-system"/"original-pull-secret" not registered Sep 17 00:29:52.203265 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:52.203225 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ksg\" (UniqueName: \"kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg\") pod \"network-check-target-xv4br\" (UID: \"c59a8ddf-3715-4032-a1d4-2f01bd95aed1\") " pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:29:52.203457 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:52.203422 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 00:29:52.203457 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:52.203441 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 00:29:52.203457 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:52.203455 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m8ksg for pod openshift-network-diagnostics/network-check-target-xv4br: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 00:29:52.203655 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:52.203515 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg podName:c59a8ddf-3715-4032-a1d4-2f01bd95aed1 nodeName:}" failed. No retries permitted until 2026-09-17 00:29:56.20349556 +0000 UTC m=+9.134129034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m8ksg" (UniqueName: "kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg") pod "network-check-target-xv4br" (UID: "c59a8ddf-3715-4032-a1d4-2f01bd95aed1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 00:29:52.644936 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:52.644389 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:29:52.644936 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:52.644563 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bs8f6" podUID="dda66642-c725-4b58-82ac-283b33d09dab" Sep 17 00:29:53.644772 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:53.644122 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:29:53.644772 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:53.644137 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:29:53.644772 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:53.644250 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xv4br" podUID="c59a8ddf-3715-4032-a1d4-2f01bd95aed1" Sep 17 00:29:53.644772 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:53.644722 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4cz" podUID="6155b312-55da-43b1-9b1f-4d51dfbb0d42" Sep 17 00:29:54.644219 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:54.644180 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:29:54.644404 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:54.644346 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bs8f6" podUID="dda66642-c725-4b58-82ac-283b33d09dab" Sep 17 00:29:55.644030 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:55.643992 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:29:55.644030 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:55.644032 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:29:55.644601 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:55.644142 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4cz" podUID="6155b312-55da-43b1-9b1f-4d51dfbb0d42" Sep 17 00:29:55.644601 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:55.644256 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xv4br" podUID="c59a8ddf-3715-4032-a1d4-2f01bd95aed1" Sep 17 00:29:56.140409 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:56.140367 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret\") pod \"global-pull-secret-syncer-bs8f6\" (UID: \"dda66642-c725-4b58-82ac-283b33d09dab\") " pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:29:56.140583 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:56.140436 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs\") pod \"network-metrics-daemon-rh4cz\" (UID: \"6155b312-55da-43b1-9b1f-4d51dfbb0d42\") " pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:29:56.140650 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:56.140613 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 00:29:56.140711 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:56.140690 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs podName:6155b312-55da-43b1-9b1f-4d51dfbb0d42 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:04.140667526 +0000 UTC m=+17.071301011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs") pod "network-metrics-daemon-rh4cz" (UID: "6155b312-55da-43b1-9b1f-4d51dfbb0d42") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 00:29:56.141090 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:56.141070 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 00:29:56.141189 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:56.141129 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret podName:dda66642-c725-4b58-82ac-283b33d09dab nodeName:}" failed. No retries permitted until 2026-09-17 00:30:04.141113731 +0000 UTC m=+17.071747216 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret") pod "global-pull-secret-syncer-bs8f6" (UID: "dda66642-c725-4b58-82ac-283b33d09dab") : object "kube-system"/"original-pull-secret" not registered Sep 17 00:29:56.241088 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:56.241055 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ksg\" (UniqueName: \"kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg\") pod \"network-check-target-xv4br\" (UID: \"c59a8ddf-3715-4032-a1d4-2f01bd95aed1\") " pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:29:56.241292 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:56.241264 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 00:29:56.241292 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:56.241292 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 00:29:56.241397 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:56.241307 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m8ksg for pod openshift-network-diagnostics/network-check-target-xv4br: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 00:29:56.241397 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:56.241381 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg podName:c59a8ddf-3715-4032-a1d4-2f01bd95aed1 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:04.24136053 +0000 UTC m=+17.171994019 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m8ksg" (UniqueName: "kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg") pod "network-check-target-xv4br" (UID: "c59a8ddf-3715-4032-a1d4-2f01bd95aed1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 00:29:56.644305 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:56.644264 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:29:56.645006 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:56.644436 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bs8f6" podUID="dda66642-c725-4b58-82ac-283b33d09dab" Sep 17 00:29:57.644634 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:57.644596 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:29:57.645113 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:57.644716 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4cz" podUID="6155b312-55da-43b1-9b1f-4d51dfbb0d42" Sep 17 00:29:57.645113 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:57.644753 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:29:57.645113 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:57.644852 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xv4br" podUID="c59a8ddf-3715-4032-a1d4-2f01bd95aed1" Sep 17 00:29:58.644948 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:58.644407 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:29:58.644948 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:58.644554 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bs8f6" podUID="dda66642-c725-4b58-82ac-283b33d09dab" Sep 17 00:29:59.644058 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:59.644016 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:29:59.644240 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:59.644146 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xv4br" podUID="c59a8ddf-3715-4032-a1d4-2f01bd95aed1" Sep 17 00:29:59.644585 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:29:59.644562 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:29:59.644724 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:29:59.644702 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4cz" podUID="6155b312-55da-43b1-9b1f-4d51dfbb0d42" Sep 17 00:30:00.644175 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:00.644147 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:30:00.644542 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:00.644236 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bs8f6" podUID="dda66642-c725-4b58-82ac-283b33d09dab" Sep 17 00:30:01.644997 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:01.644475 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:30:01.644997 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:01.644541 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:30:01.644997 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:01.644641 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4cz" podUID="6155b312-55da-43b1-9b1f-4d51dfbb0d42" Sep 17 00:30:01.644997 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:01.644795 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xv4br" podUID="c59a8ddf-3715-4032-a1d4-2f01bd95aed1" Sep 17 00:30:02.644009 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:02.643969 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:30:02.644216 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:02.644185 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bs8f6" podUID="dda66642-c725-4b58-82ac-283b33d09dab" Sep 17 00:30:03.644922 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:03.644884 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:30:03.645370 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:03.645027 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4cz" podUID="6155b312-55da-43b1-9b1f-4d51dfbb0d42" Sep 17 00:30:03.645370 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:03.645080 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:30:03.645370 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:03.645191 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xv4br" podUID="c59a8ddf-3715-4032-a1d4-2f01bd95aed1" Sep 17 00:30:04.207062 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:04.207033 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret\") pod \"global-pull-secret-syncer-bs8f6\" (UID: \"dda66642-c725-4b58-82ac-283b33d09dab\") " pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:30:04.207224 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:04.207074 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs\") pod \"network-metrics-daemon-rh4cz\" (UID: \"6155b312-55da-43b1-9b1f-4d51dfbb0d42\") " pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:30:04.207224 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:04.207186 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 00:30:04.207224 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:04.207188 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 00:30:04.207355 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:04.207250 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs podName:6155b312-55da-43b1-9b1f-4d51dfbb0d42 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:20.207234267 +0000 UTC m=+33.137867734 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs") pod "network-metrics-daemon-rh4cz" (UID: "6155b312-55da-43b1-9b1f-4d51dfbb0d42") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 00:30:04.207355 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:04.207264 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret podName:dda66642-c725-4b58-82ac-283b33d09dab nodeName:}" failed. No retries permitted until 2026-09-17 00:30:20.207258057 +0000 UTC m=+33.137891522 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret") pod "global-pull-secret-syncer-bs8f6" (UID: "dda66642-c725-4b58-82ac-283b33d09dab") : object "kube-system"/"original-pull-secret" not registered Sep 17 00:30:04.308197 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:04.308162 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ksg\" (UniqueName: \"kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg\") pod \"network-check-target-xv4br\" (UID: \"c59a8ddf-3715-4032-a1d4-2f01bd95aed1\") " pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:30:04.308380 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:04.308326 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 00:30:04.308380 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:04.308348 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 00:30:04.308380 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:04.308358 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m8ksg for pod openshift-network-diagnostics/network-check-target-xv4br: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 00:30:04.308552 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:04.308419 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg podName:c59a8ddf-3715-4032-a1d4-2f01bd95aed1 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:20.308397634 +0000 UTC m=+33.239031120 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m8ksg" (UniqueName: "kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg") pod "network-check-target-xv4br" (UID: "c59a8ddf-3715-4032-a1d4-2f01bd95aed1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 00:30:04.644667 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:04.644638 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:30:04.644839 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:04.644762 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bs8f6" podUID="dda66642-c725-4b58-82ac-283b33d09dab" Sep 17 00:30:05.644723 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:05.644696 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:30:05.645274 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:05.644686 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:30:05.645274 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:05.644826 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xv4br" podUID="c59a8ddf-3715-4032-a1d4-2f01bd95aed1" Sep 17 00:30:05.645274 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:05.644906 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4cz" podUID="6155b312-55da-43b1-9b1f-4d51dfbb0d42" Sep 17 00:30:06.644631 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:06.644602 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:30:06.644811 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:06.644703 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bs8f6" podUID="dda66642-c725-4b58-82ac-283b33d09dab" Sep 17 00:30:07.644714 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:07.644655 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:30:07.644907 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:07.644817 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4cz" podUID="6155b312-55da-43b1-9b1f-4d51dfbb0d42" Sep 17 00:30:07.645265 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:07.645076 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:30:07.645265 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:07.645172 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xv4br" podUID="c59a8ddf-3715-4032-a1d4-2f01bd95aed1" Sep 17 00:30:08.644441 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:08.644412 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:30:08.644558 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:08.644530 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bs8f6" podUID="dda66642-c725-4b58-82ac-283b33d09dab" Sep 17 00:30:08.904398 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:08.904225 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" event={"ID":"2861d237-812c-450e-aec8-954e0324b4b5","Type":"ContainerStarted","Data":"4082b05d8512d5bc33eacbec099a941a20e154e8ac48be73031dfb4d319b4fed"} Sep 17 00:30:08.940461 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:08.940400 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vnqww" podStartSLOduration=1.295399127 podStartE2EDuration="20.940383913s" totalImagesPullingTime="19.644984786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:08.928993328 +0000 UTC m=+21.859626815" watchObservedRunningTime="2026-09-17 00:30:08.940383913 +0000 UTC m=+21.871017408" Sep 17 00:30:08.940732 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:08.940706 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-177.ec2.internal" podStartSLOduration=19.940697328 podStartE2EDuration="19.940697328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:08.940570198 +0000 UTC m=+21.871203685" watchObservedRunningTime="2026-09-17 00:30:08.940697328 +0000 UTC m=+21.871330814" Sep 17 00:30:09.644908 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.644690 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:30:09.645041 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.644763 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:30:09.645041 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:09.644953 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4cz" podUID="6155b312-55da-43b1-9b1f-4d51dfbb0d42" Sep 17 00:30:09.645041 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:09.645009 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xv4br" podUID="c59a8ddf-3715-4032-a1d4-2f01bd95aed1" Sep 17 00:30:09.914123 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.914095 2582 generic.go:356] "Generic (PLEG): container finished" podID="8bdcdeea-d660-4ba8-8186-a2c495239b90" containerID="9e8e9afbe26e6fc11a60f5b48295d73f2b299bfdeed7594b887763600983a7b9" exitCode=0 Sep 17 00:30:09.914872 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.914178 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pld9" event={"ID":"8bdcdeea-d660-4ba8-8186-a2c495239b90","Type":"ContainerDied","Data":"9e8e9afbe26e6fc11a60f5b48295d73f2b299bfdeed7594b887763600983a7b9"} Sep 17 00:30:09.915822 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.915801 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d7jth" event={"ID":"ddb33aca-e881-421e-b64f-ca6aaa46298d","Type":"ContainerStarted","Data":"d3d50ac08b31b84da597931659c677afed937a0c46d9bd8cf9b6911ad577b5df"} Sep 17 00:30:09.917360 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.917336 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fmx98" event={"ID":"29c30e84-c3d7-4e23-a4dd-efc80b57ccb5","Type":"ContainerStarted","Data":"d865cef056ca0c046746173196024a553f5e43f73d6824de2213bf737b4b1f56"} Sep 17 00:30:09.920595 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.920577 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" event={"ID":"2861d237-812c-450e-aec8-954e0324b4b5","Type":"ContainerStarted","Data":"cf540a73f1fa9026962d3f28e4795bb7eb8714a3063d8e263e7ce82a1de6b484"} Sep 17 00:30:09.920693 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.920646 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" event={"ID":"2861d237-812c-450e-aec8-954e0324b4b5","Type":"ContainerStarted","Data":"0a3944de3d09c5836e9bc310636453d87d42ed74efeccad5bb38d73d62e8e9b1"} Sep 17 00:30:09.920751 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.920700 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" event={"ID":"2861d237-812c-450e-aec8-954e0324b4b5","Type":"ContainerStarted","Data":"27db6dabaed92f08b472e5ce1e17dff98292cb92eb2848fdd7422e7ab7e582fd"} Sep 17 00:30:09.920807 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.920750 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" event={"ID":"2861d237-812c-450e-aec8-954e0324b4b5","Type":"ContainerStarted","Data":"7d2981c34ab2442393391c6af6a26347dd23bf129c6174048d98a4eca6599ea2"} Sep 17 00:30:09.920807 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.920797 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" event={"ID":"2861d237-812c-450e-aec8-954e0324b4b5","Type":"ContainerStarted","Data":"2a5274c2af11300df294e7d4acf32fb2d46d32c2787d72a95c394eb00dd267a3"} Sep 17 00:30:09.922030 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.922009 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" event={"ID":"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b","Type":"ContainerStarted","Data":"d7f59a148341048d45477ff8bcb8c7fe3d16fa6831e24706fe3a55219b4c0fb9"} Sep 17 00:30:09.923287 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.923267 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-177.ec2.internal" event={"ID":"81937eb52bebe19931264bfcfb8d032c","Type":"ContainerStarted","Data":"6a1965455032832c68ec7f2bed6054b7b39fecac0812c5ff69311d18a177c43a"} Sep 17 00:30:09.924686 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.924665 2582 generic.go:356] "Generic (PLEG): container finished" podID="73d3f85df08cf592386ff2d120f060eb" containerID="b40fc35a0261772c75a6a2cfa13cf0198fb6e8dd0cf2208d04b97af4db0284fa" exitCode=0 Sep 17 00:30:09.924774 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.924739 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal" event={"ID":"73d3f85df08cf592386ff2d120f060eb","Type":"ContainerDied","Data":"b40fc35a0261772c75a6a2cfa13cf0198fb6e8dd0cf2208d04b97af4db0284fa"} Sep 17 00:30:09.926068 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.926047 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kzvtd" event={"ID":"6d6d5c16-9a78-4f6e-b33b-bd62d16302b0","Type":"ContainerStarted","Data":"29f893994b18e6cc37a31ae50653ac871063bf5428fff36858bc94d283d4cdd9"} Sep 17 00:30:09.927435 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.927413 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vnqww" event={"ID":"b3073465-da94-43fa-bd66-bc3158a4b719","Type":"ContainerStarted","Data":"a920606b6bdf18703217702fd591b08eb0bf4ec3058fdd375ddc91380194e596"} Sep 17 00:30:09.928939 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.928918 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-55r97" event={"ID":"ec239ad8-714d-4805-b7c4-a0159e28b289","Type":"ContainerStarted","Data":"8ed379216793305bc3a3a62227d7f9c11b5c465d81b1f7e3cf5595015f18acd2"} Sep 17 00:30:09.945039 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.945002 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-55r97" podStartSLOduration=6.373669053 podStartE2EDuration="21.944988998s" totalImagesPullingTime="15.571319945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:09.944653118 +0000 UTC m=+22.875286605" watchObservedRunningTime="2026-09-17 00:30:09.944988998 +0000 UTC m=+22.875622484" Sep 17 00:30:09.960004 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.959957 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kzvtd" podStartSLOduration=2.322766781 podStartE2EDuration="21.95994381s" totalImagesPullingTime="19.637177029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:09.959826196 +0000 UTC m=+22.890459685" watchObservedRunningTime="2026-09-17 00:30:09.95994381 +0000 UTC m=+22.890577297" Sep 17 00:30:09.975210 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:09.975173 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fmx98" podStartSLOduration=1.9571232809999999 podStartE2EDuration="21.975163723s" totalImagesPullingTime="20.018040442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:09.974936767 +0000 UTC m=+22.905570256" watchObservedRunningTime="2026-09-17 00:30:09.975163723 +0000 UTC m=+22.905797211" Sep 17 00:30:10.003876 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:10.003833 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d7jth" podStartSLOduration=2.291360082 podStartE2EDuration="22.003816525s" totalImagesPullingTime="19.712456443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:10.003623437 +0000 UTC m=+22.934256906" watchObservedRunningTime="2026-09-17 00:30:10.003816525 +0000 UTC m=+22.934450012" Sep 17 00:30:10.509138 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:10.509115 2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 17 00:30:10.543133 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:10.543047 2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T00:30:10.509132039Z","UUID":"03885a07-be69-44e5-948b-2a92d4510b55","Handler":null,"Name":"","Endpoint":""} Sep 17 00:30:10.544332 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:10.544315 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 17 00:30:10.544418 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:10.544352 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 17 00:30:10.644675 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:10.644645 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:30:10.644806 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:10.644739 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bs8f6" podUID="dda66642-c725-4b58-82ac-283b33d09dab" Sep 17 00:30:10.933871 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:10.933628 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hpwmd" event={"ID":"2d95051d-2d7f-4b93-bf06-1114d4794251","Type":"ContainerStarted","Data":"481441042b78475dc04ffc088a5025685131f184d9593c797db85361537d4a59"} Sep 17 00:30:10.935524 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:10.935480 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" event={"ID":"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b","Type":"ContainerStarted","Data":"3f8788dcde4518c232e2bcc0686d895c3ecb358c9f978d94192e38bdfc2fd5bd"} Sep 17 00:30:10.937213 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:10.937188 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal" event={"ID":"73d3f85df08cf592386ff2d120f060eb","Type":"ContainerStarted","Data":"aadcc0c3f104b4bed084f410fb8b0dfe0b3ec00f2013399943a858a181a14116"} Sep 17 00:30:10.946951 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:10.946913 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hpwmd" podStartSLOduration=3.182241178 podStartE2EDuration="22.946900915s" totalImagesPullingTime="19.764659737s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-17 00:30:10.946547511 +0000 UTC m=+23.877180995" watchObservedRunningTime="2026-09-17 00:30:10.946900915 +0000 UTC m=+23.877534452" Sep 17 00:30:10.960542 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:10.960481 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-177.ec2.internal" podStartSLOduration=21.960469756 podStartE2EDuration="21.960469756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:10.960193494 +0000 UTC m=+23.890826980" watchObservedRunningTime="2026-09-17 00:30:10.960469756 +0000 UTC m=+23.891103242" Sep 17 00:30:11.413532 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:11.413416 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-55r97" Sep 17 00:30:11.414112 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:11.414089 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-55r97" Sep 17 00:30:11.644852 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:11.644819 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:30:11.644852 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:11.644851 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:30:11.645228 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:11.644950 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xv4br" podUID="c59a8ddf-3715-4032-a1d4-2f01bd95aed1" Sep 17 00:30:11.645228 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:11.645092 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4cz" podUID="6155b312-55da-43b1-9b1f-4d51dfbb0d42" Sep 17 00:30:11.945422 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:11.945337 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" event={"ID":"2861d237-812c-450e-aec8-954e0324b4b5","Type":"ContainerStarted","Data":"372f5ab1570ee5fd9dc439a2d58b31fd40e1f2627a3ca0d5674c1320c29efd90"} Sep 17 00:30:11.947779 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:11.947747 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" event={"ID":"46d0a8fe-92bc-4057-a0ad-5f19fcfde71b","Type":"ContainerStarted","Data":"3f68fa403092aef8902edba55de8ae718a7da0d5b35072faaa7b2b8209e56581"} Sep 17 00:30:11.948539 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:11.948315 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-55r97" Sep 17 00:30:11.948539 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:11.948480 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-55r97" Sep 17 00:30:11.977102 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:11.977045 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pc6fm" podStartSLOduration=1.998236925 podStartE2EDuration="23.977029461s" totalImagesPullingTime="21.978792536s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:29:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:11.964137505 +0000 UTC m=+24.894770993" watchObservedRunningTime="2026-09-17 00:30:11.977029461 +0000 UTC m=+24.907662949" Sep 17 00:30:12.644240 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:12.644212 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:30:12.644416 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:12.644306 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bs8f6" podUID="dda66642-c725-4b58-82ac-283b33d09dab" Sep 17 00:30:13.643942 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:13.643910 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:30:13.644328 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:13.644057 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4cz" podUID="6155b312-55da-43b1-9b1f-4d51dfbb0d42" Sep 17 00:30:13.644404 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:13.644388 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:30:13.644525 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:13.644485 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xv4br" podUID="c59a8ddf-3715-4032-a1d4-2f01bd95aed1" Sep 17 00:30:13.955247 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:13.955170 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" event={"ID":"2861d237-812c-450e-aec8-954e0324b4b5","Type":"ContainerStarted","Data":"7110ecca03e3ac0f6f04340cc189b0c103e48bf4806c82122b9e5257a67d7878"} Sep 17 00:30:13.955529 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:13.955483 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:30:13.955668 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:13.955577 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:30:13.971775 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:13.971753 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:30:13.971903 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:13.971847 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:30:13.978662 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:13.978623 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" podStartSLOduration=6.153454501 podStartE2EDuration="25.978610534s" totalImagesPullingTime="19.825156033s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:13.978467509 +0000 UTC m=+26.909100995" watchObservedRunningTime="2026-09-17 00:30:13.978610534 +0000 UTC m=+26.909244024" Sep 17 00:30:14.644181 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:14.644003 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:30:14.644620 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:14.644262 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bs8f6" podUID="dda66642-c725-4b58-82ac-283b33d09dab" Sep 17 00:30:14.957514 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:14.957420 2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 17 00:30:15.602355 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:15.602323 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bs8f6"] Sep 17 00:30:15.602527 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:15.602468 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:30:15.602617 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:15.602592 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bs8f6" podUID="dda66642-c725-4b58-82ac-283b33d09dab" Sep 17 00:30:15.605656 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:15.605629 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xv4br"] Sep 17 00:30:15.605803 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:15.605734 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:30:15.605870 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:15.605828 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xv4br" podUID="c59a8ddf-3715-4032-a1d4-2f01bd95aed1" Sep 17 00:30:15.606251 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:15.606232 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rh4cz"] Sep 17 00:30:15.606351 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:15.606315 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:30:15.606426 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:15.606392 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4cz" podUID="6155b312-55da-43b1-9b1f-4d51dfbb0d42" Sep 17 00:30:15.960438 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:15.960363 2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 17 00:30:17.645436 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:17.645406 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:30:17.645928 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:17.645494 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bs8f6" podUID="dda66642-c725-4b58-82ac-283b33d09dab" Sep 17 00:30:17.645928 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:17.645558 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:30:17.645928 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:17.645609 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4cz" podUID="6155b312-55da-43b1-9b1f-4d51dfbb0d42" Sep 17 00:30:17.645928 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:17.645630 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:30:17.645928 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:17.645671 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xv4br" podUID="c59a8ddf-3715-4032-a1d4-2f01bd95aed1" Sep 17 00:30:18.639327 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:18.639292 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:30:18.639548 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:18.639536 2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 17 00:30:18.649666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:18.649629 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" podUID="2861d237-812c-450e-aec8-954e0324b4b5" containerName="ovnkube-controller" probeResult="failure" output="" Sep 17 00:30:18.658058 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:18.658030 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" podUID="2861d237-812c-450e-aec8-954e0324b4b5" containerName="ovnkube-controller" probeResult="failure" output="" Sep 17 00:30:18.968707 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:18.968622 2582 generic.go:356] "Generic (PLEG): container finished" podID="8bdcdeea-d660-4ba8-8186-a2c495239b90" containerID="81f5a421fad4d66270418e4c488171290115d563c8fa5dcf6e5b1018b50c50fd" exitCode=0 Sep 17 00:30:18.968707 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:18.968686 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pld9" event={"ID":"8bdcdeea-d660-4ba8-8186-a2c495239b90","Type":"ContainerDied","Data":"81f5a421fad4d66270418e4c488171290115d563c8fa5dcf6e5b1018b50c50fd"} Sep 17 00:30:19.644855 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:19.644668 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:30:19.645130 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:19.644694 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:30:19.645130 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:19.644945 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xv4br" podUID="c59a8ddf-3715-4032-a1d4-2f01bd95aed1" Sep 17 00:30:19.645130 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:19.644699 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:30:19.645130 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:19.644994 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bs8f6" podUID="dda66642-c725-4b58-82ac-283b33d09dab" Sep 17 00:30:19.645130 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:19.645102 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4cz" podUID="6155b312-55da-43b1-9b1f-4d51dfbb0d42" Sep 17 00:30:19.972854 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:19.972761 2582 generic.go:356] "Generic (PLEG): container finished" podID="8bdcdeea-d660-4ba8-8186-a2c495239b90" containerID="dc983927b61f799fd6d63fb49dbe8515723d43dcef66e1a54adf19ed2b08f623" exitCode=0 Sep 17 00:30:19.972854 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:19.972819 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pld9" event={"ID":"8bdcdeea-d660-4ba8-8186-a2c495239b90","Type":"ContainerDied","Data":"dc983927b61f799fd6d63fb49dbe8515723d43dcef66e1a54adf19ed2b08f623"} Sep 17 00:30:20.227411 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:20.227331 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret\") pod \"global-pull-secret-syncer-bs8f6\" (UID: \"dda66642-c725-4b58-82ac-283b33d09dab\") " pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:30:20.227411 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:20.227372 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs\") pod \"network-metrics-daemon-rh4cz\" (UID: \"6155b312-55da-43b1-9b1f-4d51dfbb0d42\") " pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:30:20.227612 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:20.227483 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 00:30:20.227612 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:20.227483 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 17 00:30:20.227612 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:20.227567 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs podName:6155b312-55da-43b1-9b1f-4d51dfbb0d42 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:52.227548571 +0000 UTC m=+65.158182050 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs") pod "network-metrics-daemon-rh4cz" (UID: "6155b312-55da-43b1-9b1f-4d51dfbb0d42") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 17 00:30:20.227612 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:20.227589 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret podName:dda66642-c725-4b58-82ac-283b33d09dab nodeName:}" failed. No retries permitted until 2026-09-17 00:30:52.22758038 +0000 UTC m=+65.158213850 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret") pod "global-pull-secret-syncer-bs8f6" (UID: "dda66642-c725-4b58-82ac-283b33d09dab") : object "kube-system"/"original-pull-secret" not registered Sep 17 00:30:20.328178 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:20.328145 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ksg\" (UniqueName: \"kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg\") pod \"network-check-target-xv4br\" (UID: \"c59a8ddf-3715-4032-a1d4-2f01bd95aed1\") " pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:30:20.328329 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:20.328315 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 17 00:30:20.328369 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:20.328336 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 17 00:30:20.328369 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:20.328347 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m8ksg for pod openshift-network-diagnostics/network-check-target-xv4br: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 00:30:20.328435 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:20.328399 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg podName:c59a8ddf-3715-4032-a1d4-2f01bd95aed1 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:52.328385156 +0000 UTC m=+65.259018637 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m8ksg" (UniqueName: "kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg") pod "network-check-target-xv4br" (UID: "c59a8ddf-3715-4032-a1d4-2f01bd95aed1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 17 00:30:21.644678 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.644594 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:30:21.645014 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.644604 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:30:21.645014 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:21.644718 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rh4cz" podUID="6155b312-55da-43b1-9b1f-4d51dfbb0d42" Sep 17 00:30:21.645014 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.644601 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:30:21.645014 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:21.644805 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bs8f6" podUID="dda66642-c725-4b58-82ac-283b33d09dab" Sep 17 00:30:21.645014 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:21.644881 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xv4br" podUID="c59a8ddf-3715-4032-a1d4-2f01bd95aed1" Sep 17 00:30:21.878282 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.878255 2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-177.ec2.internal" event="NodeReady" Sep 17 00:30:21.878450 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.878359 2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 17 00:30:21.909397 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.909330 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-758869bcb7-jtbqg"] Sep 17 00:30:21.911821 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.911805 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:21.913934 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.913914 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 17 00:30:21.914063 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.913983 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cjf6n\"" Sep 17 00:30:21.914456 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.914435 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 17 00:30:21.914573 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.914433 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 17 00:30:21.919149 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.919124 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 17 00:30:21.922469 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.922407 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k"] Sep 17 00:30:21.924926 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.924907 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" Sep 17 00:30:21.926185 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.926156 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-758869bcb7-jtbqg"] Sep 17 00:30:21.926928 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.926903 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8vx7h"] Sep 17 00:30:21.927284 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.927052 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 17 00:30:21.927284 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.927079 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 17 00:30:21.927386 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.927373 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-m9f5h\"" Sep 17 00:30:21.929649 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.929634 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6qb8z"] Sep 17 00:30:21.929774 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.929761 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8vx7h" Sep 17 00:30:21.931866 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.931848 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 17 00:30:21.931979 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.931887 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 17 00:30:21.932139 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.932112 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n9svt\"" Sep 17 00:30:21.932201 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.932166 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6qb8z" Sep 17 00:30:21.933987 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.933970 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6wf2d\"" Sep 17 00:30:21.934089 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.933993 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 17 00:30:21.934089 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.934015 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 17 00:30:21.934275 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.934255 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 17 00:30:21.935181 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.935163 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k"] Sep 17 00:30:21.939036 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.939018 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8vx7h"] Sep 17 00:30:21.939903 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.939828 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6qb8z"] Sep 17 00:30:21.980097 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.980063 2582 generic.go:356] "Generic (PLEG): container finished" podID="8bdcdeea-d660-4ba8-8186-a2c495239b90" containerID="c380a34d973bd2757b6f46052f5bfc77e7ae15b2fc77c2c202b5813688ba63f2" exitCode=0 Sep 17 00:30:21.980097 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:21.980101 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pld9" event={"ID":"8bdcdeea-d660-4ba8-8186-a2c495239b90","Type":"ContainerDied","Data":"c380a34d973bd2757b6f46052f5bfc77e7ae15b2fc77c2c202b5813688ba63f2"} Sep 17 00:30:22.040860 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.040821 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-bound-sa-token\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.040860 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.040858 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:30:22.041057 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.040899 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-config-volume\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:30:22.041057 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.040913 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/446d7623-90a7-453f-b8cc-dc54feffd5b2-cert\") pod \"ingress-canary-6qb8z\" (UID: \"446d7623-90a7-453f-b8cc-dc54feffd5b2\") " pod="openshift-ingress-canary/ingress-canary-6qb8z" Sep 17 00:30:22.041057 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.040940 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.041057 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.040956 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2qdf\" (UniqueName: \"kubernetes.io/projected/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-kube-api-access-k2qdf\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:30:22.041057 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.041003 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxkdr\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-kube-api-access-qxkdr\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.041057 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.041017 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-tmp-dir\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:30:22.041057 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.041033 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mp6z\" (UniqueName: \"kubernetes.io/projected/446d7623-90a7-453f-b8cc-dc54feffd5b2-kube-api-access-5mp6z\") pod \"ingress-canary-6qb8z\" (UID: \"446d7623-90a7-453f-b8cc-dc54feffd5b2\") " pod="openshift-ingress-canary/ingress-canary-6qb8z" Sep 17 00:30:22.041276 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.041088 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-ca-trust-extracted\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.041276 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.041132 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-trusted-ca\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.041276 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.041154 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-installation-pull-secrets\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.041276 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.041199 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-b6l4k\" (UID: \"5cc2bc3b-4179-4230-9a3b-de67ca98ad50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" Sep 17 00:30:22.041276 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.041221 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-b6l4k\" (UID: \"5cc2bc3b-4179-4230-9a3b-de67ca98ad50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" Sep 17 00:30:22.041276 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.041241 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-image-registry-private-configuration\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.041276 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.041257 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-certificates\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.063961 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.063924 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55r97_ec239ad8-714d-4805-b7c4-a0159e28b289/konnectivity-agent/0.log" limit=1024 Sep 17 00:30:22.142036 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.142001 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-installation-pull-secrets\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.142189 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.142071 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-b6l4k\" (UID: \"5cc2bc3b-4179-4230-9a3b-de67ca98ad50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" Sep 17 00:30:22.142189 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.142092 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-b6l4k\" (UID: \"5cc2bc3b-4179-4230-9a3b-de67ca98ad50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" Sep 17 00:30:22.142189 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.142119 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-image-registry-private-configuration\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.142189 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.142143 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-certificates\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.142189 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.142172 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-bound-sa-token\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.142443 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.142193 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:30:22.142443 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:22.142243 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 00:30:22.142443 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.142259 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-config-volume\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:30:22.142443 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.142282 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/446d7623-90a7-453f-b8cc-dc54feffd5b2-cert\") pod \"ingress-canary-6qb8z\" (UID: \"446d7623-90a7-453f-b8cc-dc54feffd5b2\") " pod="openshift-ingress-canary/ingress-canary-6qb8z" Sep 17 00:30:22.142443 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:22.142334 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert podName:5cc2bc3b-4179-4230-9a3b-de67ca98ad50 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:22.642311208 +0000 UTC m=+35.572944698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-b6l4k" (UID: "5cc2bc3b-4179-4230-9a3b-de67ca98ad50") : secret "networking-console-plugin-cert" not found Sep 17 00:30:22.142443 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.142378 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.142443 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.142416 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2qdf\" (UniqueName: \"kubernetes.io/projected/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-kube-api-access-k2qdf\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:30:22.142818 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.142465 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxkdr\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-kube-api-access-qxkdr\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.142818 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.142524 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-tmp-dir\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:30:22.142818 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.142559 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mp6z\" (UniqueName: \"kubernetes.io/projected/446d7623-90a7-453f-b8cc-dc54feffd5b2-kube-api-access-5mp6z\") pod \"ingress-canary-6qb8z\" (UID: \"446d7623-90a7-453f-b8cc-dc54feffd5b2\") " pod="openshift-ingress-canary/ingress-canary-6qb8z" Sep 17 00:30:22.142818 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.142584 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-ca-trust-extracted\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.142818 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.142608 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-trusted-ca\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.143073 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:22.142857 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 00:30:22.143073 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:22.142873 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758869bcb7-jtbqg: secret "image-registry-tls" not found Sep 17 00:30:22.143073 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.142882 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-b6l4k\" (UID: \"5cc2bc3b-4179-4230-9a3b-de67ca98ad50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" Sep 17 00:30:22.143073 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:22.142923 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls podName:b1b2f065-efd1-4d6c-8ec8-5c116b880c66 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:22.642906628 +0000 UTC m=+35.573540110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls") pod "image-registry-758869bcb7-jtbqg" (UID: "b1b2f065-efd1-4d6c-8ec8-5c116b880c66") : secret "image-registry-tls" not found Sep 17 00:30:22.143073 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:22.142990 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 00:30:22.143073 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:22.143022 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls podName:3f05fc1f-6243-45c9-8f0b-d8d6a40b6225 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:22.643011032 +0000 UTC m=+35.573644531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls") pod "dns-default-8vx7h" (UID: "3f05fc1f-6243-45c9-8f0b-d8d6a40b6225") : secret "dns-default-metrics-tls" not found Sep 17 00:30:22.143410 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.143359 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-config-volume\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:30:22.143518 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.143479 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-trusted-ca\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.143614 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.143541 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-ca-trust-extracted\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.143614 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.143558 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-tmp-dir\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:30:22.143840 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.143815 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-certificates\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.145166 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.145141 2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55r97_ec239ad8-714d-4805-b7c4-a0159e28b289/konnectivity-agent/0.log" limit=1024 Sep 17 00:30:22.147155 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.147134 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/446d7623-90a7-453f-b8cc-dc54feffd5b2-cert\") pod \"ingress-canary-6qb8z\" (UID: \"446d7623-90a7-453f-b8cc-dc54feffd5b2\") " pod="openshift-ingress-canary/ingress-canary-6qb8z" Sep 17 00:30:22.147155 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.147145 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-installation-pull-secrets\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.147274 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.147145 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-image-registry-private-configuration\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.151026 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.150984 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-bound-sa-token\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.151480 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.151458 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2qdf\" (UniqueName: \"kubernetes.io/projected/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-kube-api-access-k2qdf\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:30:22.152781 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.152762 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxkdr\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-kube-api-access-qxkdr\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.153059 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.153037 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mp6z\" (UniqueName: \"kubernetes.io/projected/446d7623-90a7-453f-b8cc-dc54feffd5b2-kube-api-access-5mp6z\") pod \"ingress-canary-6qb8z\" (UID: \"446d7623-90a7-453f-b8cc-dc54feffd5b2\") " pod="openshift-ingress-canary/ingress-canary-6qb8z" Sep 17 00:30:22.248278 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.248178 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6qb8z" Sep 17 00:30:22.525790 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.525689 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6qb8z"] Sep 17 00:30:22.530665 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:30:22.530567 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446d7623_90a7_453f_b8cc_dc54feffd5b2.slice/crio-b733becbf58e51afabfcef453a77db8284a9247c57d43393542ff80eab3e56f1 WatchSource:0}: Error finding container b733becbf58e51afabfcef453a77db8284a9247c57d43393542ff80eab3e56f1: Status 404 returned error can't find the container with id b733becbf58e51afabfcef453a77db8284a9247c57d43393542ff80eab3e56f1 Sep 17 00:30:22.645969 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.645909 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:30:22.646612 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.646020 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:22.646612 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:22.646070 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 00:30:22.646612 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.646094 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-b6l4k\" (UID: \"5cc2bc3b-4179-4230-9a3b-de67ca98ad50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" Sep 17 00:30:22.646612 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:22.646159 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls podName:3f05fc1f-6243-45c9-8f0b-d8d6a40b6225 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:23.646138538 +0000 UTC m=+36.576772027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls") pod "dns-default-8vx7h" (UID: "3f05fc1f-6243-45c9-8f0b-d8d6a40b6225") : secret "dns-default-metrics-tls" not found Sep 17 00:30:22.646612 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:22.646180 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 00:30:22.646612 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:22.646199 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758869bcb7-jtbqg: secret "image-registry-tls" not found Sep 17 00:30:22.646612 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:22.646207 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 00:30:22.646612 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:22.646252 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls podName:b1b2f065-efd1-4d6c-8ec8-5c116b880c66 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:23.646233097 +0000 UTC m=+36.576866580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls") pod "image-registry-758869bcb7-jtbqg" (UID: "b1b2f065-efd1-4d6c-8ec8-5c116b880c66") : secret "image-registry-tls" not found Sep 17 00:30:22.646612 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:22.646270 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert podName:5cc2bc3b-4179-4230-9a3b-de67ca98ad50 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:23.646260305 +0000 UTC m=+36.576893771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-b6l4k" (UID: "5cc2bc3b-4179-4230-9a3b-de67ca98ad50") : secret "networking-console-plugin-cert" not found Sep 17 00:30:22.983778 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:22.983694 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6qb8z" event={"ID":"446d7623-90a7-453f-b8cc-dc54feffd5b2","Type":"ContainerStarted","Data":"b733becbf58e51afabfcef453a77db8284a9247c57d43393542ff80eab3e56f1"} Sep 17 00:30:23.646870 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:23.646839 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:30:23.647323 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:23.646840 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:30:23.647323 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:23.646846 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:30:23.650320 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:23.650298 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 00:30:23.650572 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:23.650556 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 00:30:23.650782 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:23.650768 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 00:30:23.650997 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:23.650981 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lqlnk\"" Sep 17 00:30:23.651236 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:23.651215 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 00:30:23.651445 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:23.651429 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-plv5t\"" Sep 17 00:30:23.656650 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:23.655914 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-b6l4k\" (UID: \"5cc2bc3b-4179-4230-9a3b-de67ca98ad50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" Sep 17 00:30:23.656650 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:23.655976 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:30:23.656650 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:23.656045 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:23.656650 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:23.656154 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 00:30:23.656650 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:23.656166 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758869bcb7-jtbqg: secret "image-registry-tls" not found Sep 17 00:30:23.656650 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:23.656213 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls podName:b1b2f065-efd1-4d6c-8ec8-5c116b880c66 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:25.656199742 +0000 UTC m=+38.586833231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls") pod "image-registry-758869bcb7-jtbqg" (UID: "b1b2f065-efd1-4d6c-8ec8-5c116b880c66") : secret "image-registry-tls" not found Sep 17 00:30:23.656650 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:23.656477 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 00:30:23.656650 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:23.656542 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert podName:5cc2bc3b-4179-4230-9a3b-de67ca98ad50 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:25.656530414 +0000 UTC m=+38.587163892 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-b6l4k" (UID: "5cc2bc3b-4179-4230-9a3b-de67ca98ad50") : secret "networking-console-plugin-cert" not found Sep 17 00:30:23.656650 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:23.656607 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 00:30:23.656650 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:23.656631 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls podName:3f05fc1f-6243-45c9-8f0b-d8d6a40b6225 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:25.65662441 +0000 UTC m=+38.587257888 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls") pod "dns-default-8vx7h" (UID: "3f05fc1f-6243-45c9-8f0b-d8d6a40b6225") : secret "dns-default-metrics-tls" not found Sep 17 00:30:25.674131 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:25.673916 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:30:25.674598 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:25.674075 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 00:30:25.674598 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:25.674279 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 00:30:25.674598 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:25.674197 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:25.674598 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:25.674293 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758869bcb7-jtbqg: secret "image-registry-tls" not found Sep 17 00:30:25.674598 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:25.674294 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls podName:3f05fc1f-6243-45c9-8f0b-d8d6a40b6225 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:29.674268157 +0000 UTC m=+42.604901650 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls") pod "dns-default-8vx7h" (UID: "3f05fc1f-6243-45c9-8f0b-d8d6a40b6225") : secret "dns-default-metrics-tls" not found Sep 17 00:30:25.674598 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:25.674386 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls podName:b1b2f065-efd1-4d6c-8ec8-5c116b880c66 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:29.674365286 +0000 UTC m=+42.604998755 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls") pod "image-registry-758869bcb7-jtbqg" (UID: "b1b2f065-efd1-4d6c-8ec8-5c116b880c66") : secret "image-registry-tls" not found Sep 17 00:30:25.674598 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:25.674468 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-b6l4k\" (UID: \"5cc2bc3b-4179-4230-9a3b-de67ca98ad50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" Sep 17 00:30:25.674598 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:25.674603 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 00:30:25.674983 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:25.674641 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert podName:5cc2bc3b-4179-4230-9a3b-de67ca98ad50 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:29.674629698 +0000 UTC m=+42.605263191 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-b6l4k" (UID: "5cc2bc3b-4179-4230-9a3b-de67ca98ad50") : secret "networking-console-plugin-cert" not found Sep 17 00:30:25.992531 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:25.992447 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6qb8z" event={"ID":"446d7623-90a7-453f-b8cc-dc54feffd5b2","Type":"ContainerStarted","Data":"8f34280fb6d88c2570e448eb46847fe45e6adb80bff89ff029c205e06627db83"} Sep 17 00:30:26.006183 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:26.006127 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6qb8z" podStartSLOduration=1.840375319 podStartE2EDuration="5.006106016s" totalImagesPullingTime="3.165730697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:26.005104266 +0000 UTC m=+38.935737752" watchObservedRunningTime="2026-09-17 00:30:26.006106016 +0000 UTC m=+38.936739504" Sep 17 00:30:29.003224 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:29.003189 2582 generic.go:356] "Generic (PLEG): container finished" podID="8bdcdeea-d660-4ba8-8186-a2c495239b90" containerID="f7e9d1263d8800e0d19928dc619a5b63d964ed4fa754d6f8bca59790403a9e53" exitCode=0 Sep 17 00:30:29.003224 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:29.003228 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pld9" event={"ID":"8bdcdeea-d660-4ba8-8186-a2c495239b90","Type":"ContainerDied","Data":"f7e9d1263d8800e0d19928dc619a5b63d964ed4fa754d6f8bca59790403a9e53"} Sep 17 00:30:29.709258 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:29.709227 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:30:29.709415 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:29.709285 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:29.709415 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:29.709322 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-b6l4k\" (UID: \"5cc2bc3b-4179-4230-9a3b-de67ca98ad50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" Sep 17 00:30:29.709415 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:29.709368 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 00:30:29.709415 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:29.709396 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 00:30:29.709565 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:29.709436 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 00:30:29.709565 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:29.709451 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758869bcb7-jtbqg: secret "image-registry-tls" not found Sep 17 00:30:29.709565 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:29.709439 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls podName:3f05fc1f-6243-45c9-8f0b-d8d6a40b6225 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:37.709422022 +0000 UTC m=+50.640055507 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls") pod "dns-default-8vx7h" (UID: "3f05fc1f-6243-45c9-8f0b-d8d6a40b6225") : secret "dns-default-metrics-tls" not found Sep 17 00:30:29.709565 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:29.709511 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert podName:5cc2bc3b-4179-4230-9a3b-de67ca98ad50 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:37.709483867 +0000 UTC m=+50.640117331 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-b6l4k" (UID: "5cc2bc3b-4179-4230-9a3b-de67ca98ad50") : secret "networking-console-plugin-cert" not found Sep 17 00:30:29.709565 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:29.709530 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls podName:b1b2f065-efd1-4d6c-8ec8-5c116b880c66 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:37.709522532 +0000 UTC m=+50.640155997 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls") pod "image-registry-758869bcb7-jtbqg" (UID: "b1b2f065-efd1-4d6c-8ec8-5c116b880c66") : secret "image-registry-tls" not found Sep 17 00:30:30.008146 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:30.008068 2582 generic.go:356] "Generic (PLEG): container finished" podID="8bdcdeea-d660-4ba8-8186-a2c495239b90" containerID="4daea1756e7df88936a5a5fbf8dac07b1934fcb1004b42234c18bd38309cb935" exitCode=0 Sep 17 00:30:30.008146 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:30.008109 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pld9" event={"ID":"8bdcdeea-d660-4ba8-8186-a2c495239b90","Type":"ContainerDied","Data":"4daea1756e7df88936a5a5fbf8dac07b1934fcb1004b42234c18bd38309cb935"} Sep 17 00:30:31.013647 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:31.013610 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6pld9" event={"ID":"8bdcdeea-d660-4ba8-8186-a2c495239b90","Type":"ContainerStarted","Data":"987c42aaddeee1a7c339ee881fa468763154acfc273d559c69b6b12d975c7983"} Sep 17 00:30:31.032081 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:31.032033 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6pld9" podStartSLOduration=7.574479299 podStartE2EDuration="43.032018288s" totalImagesPullingTime="35.457538989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:29:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:31.031170334 +0000 UTC m=+43.961803835" watchObservedRunningTime="2026-09-17 00:30:31.032018288 +0000 UTC m=+43.962651775" Sep 17 00:30:37.766367 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:37.766324 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-b6l4k\" (UID: \"5cc2bc3b-4179-4230-9a3b-de67ca98ad50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" Sep 17 00:30:37.766367 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:37.766372 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:30:37.766800 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:37.766414 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:37.766800 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:37.766482 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 00:30:37.766800 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:37.766533 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 00:30:37.766800 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:37.766544 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758869bcb7-jtbqg: secret "image-registry-tls" not found Sep 17 00:30:37.766800 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:37.766549 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 00:30:37.766800 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:37.766577 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert podName:5cc2bc3b-4179-4230-9a3b-de67ca98ad50 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:53.766561636 +0000 UTC m=+66.697195128 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-b6l4k" (UID: "5cc2bc3b-4179-4230-9a3b-de67ca98ad50") : secret "networking-console-plugin-cert" not found Sep 17 00:30:37.766800 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:37.766592 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls podName:b1b2f065-efd1-4d6c-8ec8-5c116b880c66 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:53.766586028 +0000 UTC m=+66.697219493 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls") pod "image-registry-758869bcb7-jtbqg" (UID: "b1b2f065-efd1-4d6c-8ec8-5c116b880c66") : secret "image-registry-tls" not found Sep 17 00:30:37.766800 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:37.766610 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls podName:3f05fc1f-6243-45c9-8f0b-d8d6a40b6225 nodeName:}" failed. No retries permitted until 2026-09-17 00:30:53.766597397 +0000 UTC m=+66.697230862 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls") pod "dns-default-8vx7h" (UID: "3f05fc1f-6243-45c9-8f0b-d8d6a40b6225") : secret "dns-default-metrics-tls" not found Sep 17 00:30:48.659183 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:48.659154 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wzmbr" Sep 17 00:30:52.272929 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:52.272896 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret\") pod \"global-pull-secret-syncer-bs8f6\" (UID: \"dda66642-c725-4b58-82ac-283b33d09dab\") " pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:30:52.272929 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:52.272932 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs\") pod \"network-metrics-daemon-rh4cz\" (UID: \"6155b312-55da-43b1-9b1f-4d51dfbb0d42\") " pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:30:52.275146 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:52.275125 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 17 00:30:52.275229 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:52.275147 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 17 00:30:52.283746 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:52.283731 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 00:30:52.283794 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:52.283784 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs podName:6155b312-55da-43b1-9b1f-4d51dfbb0d42 nodeName:}" failed. No retries permitted until 2026-09-17 00:31:56.283769302 +0000 UTC m=+129.214402780 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs") pod "network-metrics-daemon-rh4cz" (UID: "6155b312-55da-43b1-9b1f-4d51dfbb0d42") : secret "metrics-daemon-secret" not found Sep 17 00:30:52.286693 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:52.286662 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dda66642-c725-4b58-82ac-283b33d09dab-original-pull-secret\") pod \"global-pull-secret-syncer-bs8f6\" (UID: \"dda66642-c725-4b58-82ac-283b33d09dab\") " pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:30:52.374063 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:52.374035 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ksg\" (UniqueName: \"kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg\") pod \"network-check-target-xv4br\" (UID: \"c59a8ddf-3715-4032-a1d4-2f01bd95aed1\") " pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:30:52.376208 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:52.376192 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 17 00:30:52.386077 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:52.386058 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 17 00:30:52.397857 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:52.397835 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8ksg\" (UniqueName: \"kubernetes.io/projected/c59a8ddf-3715-4032-a1d4-2f01bd95aed1-kube-api-access-m8ksg\") pod \"network-check-target-xv4br\" (UID: \"c59a8ddf-3715-4032-a1d4-2f01bd95aed1\") " pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:30:52.463614 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:52.463581 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lqlnk\"" Sep 17 00:30:52.472228 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:52.472208 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:30:52.474456 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:52.474440 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bs8f6" Sep 17 00:30:52.766958 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:52.766928 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xv4br"] Sep 17 00:30:52.770511 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:30:52.770473 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59a8ddf_3715_4032_a1d4_2f01bd95aed1.slice/crio-5703d1b4efeb85aa845aa8a97b724be4e36d09aaae148ef89fdea72470d9f44b WatchSource:0}: Error finding container 5703d1b4efeb85aa845aa8a97b724be4e36d09aaae148ef89fdea72470d9f44b: Status 404 returned error can't find the container with id 5703d1b4efeb85aa845aa8a97b724be4e36d09aaae148ef89fdea72470d9f44b Sep 17 00:30:52.792709 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:52.792685 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bs8f6"] Sep 17 00:30:52.795837 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:30:52.795815 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda66642_c725_4b58_82ac_283b33d09dab.slice/crio-41528088208a0848ccfb574fc644520a232a86480b12ac9d2ab6003504b62881 WatchSource:0}: Error finding container 41528088208a0848ccfb574fc644520a232a86480b12ac9d2ab6003504b62881: Status 404 returned error can't find the container with id 41528088208a0848ccfb574fc644520a232a86480b12ac9d2ab6003504b62881 Sep 17 00:30:53.065548 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:53.065515 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bs8f6" event={"ID":"dda66642-c725-4b58-82ac-283b33d09dab","Type":"ContainerStarted","Data":"41528088208a0848ccfb574fc644520a232a86480b12ac9d2ab6003504b62881"} Sep 17 00:30:53.066698 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:53.066673 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xv4br" event={"ID":"c59a8ddf-3715-4032-a1d4-2f01bd95aed1","Type":"ContainerStarted","Data":"5703d1b4efeb85aa845aa8a97b724be4e36d09aaae148ef89fdea72470d9f44b"} Sep 17 00:30:53.785776 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:53.785742 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:30:53.786271 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:53.785799 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-b6l4k\" (UID: \"5cc2bc3b-4179-4230-9a3b-de67ca98ad50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" Sep 17 00:30:53.786271 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:53.785828 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:30:53.786271 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:53.785899 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 00:30:53.786271 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:53.785920 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758869bcb7-jtbqg: secret "image-registry-tls" not found Sep 17 00:30:53.786271 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:53.785931 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 00:30:53.786271 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:53.785931 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 00:30:53.786271 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:53.785995 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls podName:b1b2f065-efd1-4d6c-8ec8-5c116b880c66 nodeName:}" failed. No retries permitted until 2026-09-17 00:31:25.785962314 +0000 UTC m=+98.716595802 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls") pod "image-registry-758869bcb7-jtbqg" (UID: "b1b2f065-efd1-4d6c-8ec8-5c116b880c66") : secret "image-registry-tls" not found Sep 17 00:30:53.786271 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:53.786014 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert podName:5cc2bc3b-4179-4230-9a3b-de67ca98ad50 nodeName:}" failed. No retries permitted until 2026-09-17 00:31:25.786004896 +0000 UTC m=+98.716638367 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-b6l4k" (UID: "5cc2bc3b-4179-4230-9a3b-de67ca98ad50") : secret "networking-console-plugin-cert" not found Sep 17 00:30:53.786271 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:30:53.786030 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls podName:3f05fc1f-6243-45c9-8f0b-d8d6a40b6225 nodeName:}" failed. No retries permitted until 2026-09-17 00:31:25.786021955 +0000 UTC m=+98.716655428 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls") pod "dns-default-8vx7h" (UID: "3f05fc1f-6243-45c9-8f0b-d8d6a40b6225") : secret "dns-default-metrics-tls" not found Sep 17 00:30:58.083402 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:58.083354 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bs8f6" event={"ID":"dda66642-c725-4b58-82ac-283b33d09dab","Type":"ContainerStarted","Data":"0038f3e97b61042c6eaf498e1d66ee6f68a57aae6f5f9621220dd26f6054ed0d"} Sep 17 00:30:58.084991 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:58.084953 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xv4br" event={"ID":"c59a8ddf-3715-4032-a1d4-2f01bd95aed1","Type":"ContainerStarted","Data":"870b20eec2afe7e9307b652bfb5931813d72c9bf04cfca4513fff8b6563a01d9"} Sep 17 00:30:58.085098 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:58.085081 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:30:58.098165 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:58.098118 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bs8f6" podStartSLOduration=65.027152188 podStartE2EDuration="1m10.098103479s" totalImagesPullingTime="5.070951291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:58.097452208 +0000 UTC m=+71.028085695" watchObservedRunningTime="2026-09-17 00:30:58.098103479 +0000 UTC m=+71.028736965" Sep 17 00:30:58.111814 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:30:58.111774 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xv4br" podStartSLOduration=65.01580513 podStartE2EDuration="1m10.111757338s" totalImagesPullingTime="5.095952208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:30:58.11129831 +0000 UTC m=+71.041931791" watchObservedRunningTime="2026-09-17 00:30:58.111757338 +0000 UTC m=+71.042390825" Sep 17 00:31:25.814017 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:25.813974 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:31:25.814378 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:25.814033 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-b6l4k\" (UID: \"5cc2bc3b-4179-4230-9a3b-de67ca98ad50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" Sep 17 00:31:25.814378 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:25.814056 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:31:25.814378 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:31:25.814115 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 00:31:25.814378 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:31:25.814130 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758869bcb7-jtbqg: secret "image-registry-tls" not found Sep 17 00:31:25.814378 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:31:25.814149 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 00:31:25.814378 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:31:25.814185 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls podName:b1b2f065-efd1-4d6c-8ec8-5c116b880c66 nodeName:}" failed. No retries permitted until 2026-09-17 00:32:29.814170021 +0000 UTC m=+162.744803499 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls") pod "image-registry-758869bcb7-jtbqg" (UID: "b1b2f065-efd1-4d6c-8ec8-5c116b880c66") : secret "image-registry-tls" not found Sep 17 00:31:25.814378 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:31:25.814183 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 00:31:25.814378 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:31:25.814198 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls podName:3f05fc1f-6243-45c9-8f0b-d8d6a40b6225 nodeName:}" failed. No retries permitted until 2026-09-17 00:32:29.814191829 +0000 UTC m=+162.744825294 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls") pod "dns-default-8vx7h" (UID: "3f05fc1f-6243-45c9-8f0b-d8d6a40b6225") : secret "dns-default-metrics-tls" not found Sep 17 00:31:25.814378 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:31:25.814244 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert podName:5cc2bc3b-4179-4230-9a3b-de67ca98ad50 nodeName:}" failed. No retries permitted until 2026-09-17 00:32:29.814225945 +0000 UTC m=+162.744859617 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-b6l4k" (UID: "5cc2bc3b-4179-4230-9a3b-de67ca98ad50") : secret "networking-console-plugin-cert" not found Sep 17 00:31:29.089534 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:29.089488 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xv4br" Sep 17 00:31:34.550326 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:34.550292 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b"] Sep 17 00:31:34.553740 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:34.553724 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" Sep 17 00:31:34.555596 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:34.555571 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 17 00:31:34.555886 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:34.555581 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 17 00:31:34.556246 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:34.556227 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 17 00:31:34.556327 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:34.556261 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 17 00:31:34.562374 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:34.562349 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b"] Sep 17 00:31:34.676583 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:34.676549 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ae4814f-7090-47bc-8c55-c96c923687bc-tmp\") pod \"klusterlet-addon-workmgr-77455fb9b5-njk2b\" (UID: \"3ae4814f-7090-47bc-8c55-c96c923687bc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" Sep 17 00:31:34.676760 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:34.676630 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3ae4814f-7090-47bc-8c55-c96c923687bc-klusterlet-config\") pod \"klusterlet-addon-workmgr-77455fb9b5-njk2b\" (UID: \"3ae4814f-7090-47bc-8c55-c96c923687bc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" Sep 17 00:31:34.676760 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:34.676654 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l77f\" (UniqueName: \"kubernetes.io/projected/3ae4814f-7090-47bc-8c55-c96c923687bc-kube-api-access-6l77f\") pod \"klusterlet-addon-workmgr-77455fb9b5-njk2b\" (UID: \"3ae4814f-7090-47bc-8c55-c96c923687bc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" Sep 17 00:31:34.776945 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:34.776916 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ae4814f-7090-47bc-8c55-c96c923687bc-tmp\") pod \"klusterlet-addon-workmgr-77455fb9b5-njk2b\" (UID: \"3ae4814f-7090-47bc-8c55-c96c923687bc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" Sep 17 00:31:34.776999 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:34.776970 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3ae4814f-7090-47bc-8c55-c96c923687bc-klusterlet-config\") pod \"klusterlet-addon-workmgr-77455fb9b5-njk2b\" (UID: \"3ae4814f-7090-47bc-8c55-c96c923687bc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" Sep 17 00:31:34.777094 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:34.777075 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l77f\" (UniqueName: \"kubernetes.io/projected/3ae4814f-7090-47bc-8c55-c96c923687bc-kube-api-access-6l77f\") pod \"klusterlet-addon-workmgr-77455fb9b5-njk2b\" (UID: \"3ae4814f-7090-47bc-8c55-c96c923687bc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" Sep 17 00:31:34.777386 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:34.777364 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ae4814f-7090-47bc-8c55-c96c923687bc-tmp\") pod \"klusterlet-addon-workmgr-77455fb9b5-njk2b\" (UID: \"3ae4814f-7090-47bc-8c55-c96c923687bc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" Sep 17 00:31:34.779474 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:34.779452 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3ae4814f-7090-47bc-8c55-c96c923687bc-klusterlet-config\") pod \"klusterlet-addon-workmgr-77455fb9b5-njk2b\" (UID: \"3ae4814f-7090-47bc-8c55-c96c923687bc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" Sep 17 00:31:34.784052 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:34.784025 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l77f\" (UniqueName: \"kubernetes.io/projected/3ae4814f-7090-47bc-8c55-c96c923687bc-kube-api-access-6l77f\") pod \"klusterlet-addon-workmgr-77455fb9b5-njk2b\" (UID: \"3ae4814f-7090-47bc-8c55-c96c923687bc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" Sep 17 00:31:34.863220 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:34.863147 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" Sep 17 00:31:35.130386 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:35.130290 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b"] Sep 17 00:31:35.135798 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:31:35.135764 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae4814f_7090_47bc_8c55_c96c923687bc.slice/crio-dcb53ea7a777185d3e911d6fa4bcdad01e268d2a483c3c41532905cff749213b WatchSource:0}: Error finding container dcb53ea7a777185d3e911d6fa4bcdad01e268d2a483c3c41532905cff749213b: Status 404 returned error can't find the container with id dcb53ea7a777185d3e911d6fa4bcdad01e268d2a483c3c41532905cff749213b Sep 17 00:31:35.174244 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:35.174216 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" event={"ID":"3ae4814f-7090-47bc-8c55-c96c923687bc","Type":"ContainerStarted","Data":"dcb53ea7a777185d3e911d6fa4bcdad01e268d2a483c3c41532905cff749213b"} Sep 17 00:31:40.189831 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:40.189744 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" event={"ID":"3ae4814f-7090-47bc-8c55-c96c923687bc","Type":"ContainerStarted","Data":"6bc0855c858effc762c013ec24a7fca6c6a7ba8245c05411471614cb40a5da8b"} Sep 17 00:31:40.190245 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:40.189953 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" Sep 17 00:31:40.191483 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:40.191463 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" Sep 17 00:31:40.203538 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:40.203484 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" podStartSLOduration=1.488607549 podStartE2EDuration="6.203470828s" totalImagesPullingTime="4.714863279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:31:40.202538737 +0000 UTC m=+113.133172221" watchObservedRunningTime="2026-09-17 00:31:40.203470828 +0000 UTC m=+113.134104327" Sep 17 00:31:56.330189 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:31:56.330143 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs\") pod \"network-metrics-daemon-rh4cz\" (UID: \"6155b312-55da-43b1-9b1f-4d51dfbb0d42\") " pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:31:56.330623 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:31:56.330288 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 00:31:56.330623 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:31:56.330354 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs podName:6155b312-55da-43b1-9b1f-4d51dfbb0d42 nodeName:}" failed. No retries permitted until 2026-09-17 00:33:58.330337828 +0000 UTC m=+251.260971307 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs") pod "network-metrics-daemon-rh4cz" (UID: "6155b312-55da-43b1-9b1f-4d51dfbb0d42") : secret "metrics-daemon-secret" not found Sep 17 00:32:24.921008 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:32:24.920965 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" podUID="b1b2f065-efd1-4d6c-8ec8-5c116b880c66" Sep 17 00:32:24.936232 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:32:24.936208 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" podUID="5cc2bc3b-4179-4230-9a3b-de67ca98ad50" Sep 17 00:32:24.943488 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:32:24.943457 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8vx7h" podUID="3f05fc1f-6243-45c9-8f0b-d8d6a40b6225" Sep 17 00:32:25.296871 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:32:25.296843 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8vx7h" Sep 17 00:32:25.297025 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:32:25.296844 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" Sep 17 00:32:25.297025 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:32:25.296844 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:32:26.668994 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:32:26.668954 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-rh4cz" podUID="6155b312-55da-43b1-9b1f-4d51dfbb0d42" Sep 17 00:32:29.858390 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:32:29.858340 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls\") pod \"image-registry-758869bcb7-jtbqg\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:32:29.858816 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:32:29.858410 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-b6l4k\" (UID: \"5cc2bc3b-4179-4230-9a3b-de67ca98ad50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" Sep 17 00:32:29.858816 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:32:29.858436 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:32:29.858816 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:32:29.858492 2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 17 00:32:29.858816 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:32:29.858533 2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758869bcb7-jtbqg: secret "image-registry-tls" not found Sep 17 00:32:29.858816 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:32:29.858568 2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 17 00:32:29.858816 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:32:29.858570 2582 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 17 00:32:29.858816 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:32:29.858606 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls podName:b1b2f065-efd1-4d6c-8ec8-5c116b880c66 nodeName:}" failed. No retries permitted until 2026-09-17 00:34:31.858587942 +0000 UTC m=+284.789221428 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls") pod "image-registry-758869bcb7-jtbqg" (UID: "b1b2f065-efd1-4d6c-8ec8-5c116b880c66") : secret "image-registry-tls" not found Sep 17 00:32:29.858816 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:32:29.858625 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert podName:5cc2bc3b-4179-4230-9a3b-de67ca98ad50 nodeName:}" failed. No retries permitted until 2026-09-17 00:34:31.858611503 +0000 UTC m=+284.789244967 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-b6l4k" (UID: "5cc2bc3b-4179-4230-9a3b-de67ca98ad50") : secret "networking-console-plugin-cert" not found Sep 17 00:32:29.858816 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:32:29.858638 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls podName:3f05fc1f-6243-45c9-8f0b-d8d6a40b6225 nodeName:}" failed. No retries permitted until 2026-09-17 00:34:31.858631703 +0000 UTC m=+284.789265167 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls") pod "dns-default-8vx7h" (UID: "3f05fc1f-6243-45c9-8f0b-d8d6a40b6225") : secret "dns-default-metrics-tls" not found Sep 17 00:32:40.644675 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:32:40.644582 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:33:40.191146 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:40.191051 2582 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" podUID="3ae4814f-7090-47bc-8c55-c96c923687bc" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.10:8000/readyz\": dial tcp 10.132.0.10:8000: connect: connection refused" Sep 17 00:33:40.478739 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:40.478656 2582 generic.go:356] "Generic (PLEG): container finished" podID="3ae4814f-7090-47bc-8c55-c96c923687bc" containerID="6bc0855c858effc762c013ec24a7fca6c6a7ba8245c05411471614cb40a5da8b" exitCode=1 Sep 17 00:33:40.478739 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:40.478721 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" event={"ID":"3ae4814f-7090-47bc-8c55-c96c923687bc","Type":"ContainerDied","Data":"6bc0855c858effc762c013ec24a7fca6c6a7ba8245c05411471614cb40a5da8b"} Sep 17 00:33:40.479049 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:40.479032 2582 scope.go:117] "RemoveContainer" containerID="6bc0855c858effc762c013ec24a7fca6c6a7ba8245c05411471614cb40a5da8b" Sep 17 00:33:41.482975 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:41.482937 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" event={"ID":"3ae4814f-7090-47bc-8c55-c96c923687bc","Type":"ContainerStarted","Data":"9fa38eb9ca379b748099d4e4cfdb173f6dec1105636d4bd87b8cb8a691eef6ce"} Sep 17 00:33:41.483340 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:41.483227 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" Sep 17 00:33:41.483809 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:41.483788 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-77455fb9b5-njk2b" Sep 17 00:33:53.937588 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:53.937552 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-nbp24"] Sep 17 00:33:53.940707 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:53.940688 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-nbp24" Sep 17 00:33:53.942583 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:53.942558 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 17 00:33:53.943231 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:53.943213 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 17 00:33:53.943292 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:53.943248 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-x8clq\"" Sep 17 00:33:53.943766 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:53.943751 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 17 00:33:53.943928 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:53.943907 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 17 00:33:53.947134 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:53.947111 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 17 00:33:53.953655 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:53.953634 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-nbp24"] Sep 17 00:33:54.026512 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.026478 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqxk7"] Sep 17 00:33:54.029326 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.029305 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69rz8"] Sep 17 00:33:54.029449 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.029422 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqxk7" Sep 17 00:33:54.031374 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.031348 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-dqrmf\"" Sep 17 00:33:54.031522 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.031388 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 17 00:33:54.031522 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.031389 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 17 00:33:54.031750 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.031733 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 17 00:33:54.032879 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.032861 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-gcss2"] Sep 17 00:33:54.032996 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.032982 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69rz8" Sep 17 00:33:54.034726 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.034706 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 17 00:33:54.034825 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.034769 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 17 00:33:54.035048 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.035029 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 17 00:33:54.035148 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.035033 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-4qnfj\"" Sep 17 00:33:54.035148 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.035134 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 17 00:33:54.035664 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.035646 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-gcss2" Sep 17 00:33:54.037278 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.037262 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 17 00:33:54.037372 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.037310 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-7pqjj\"" Sep 17 00:33:54.037372 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.037320 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 17 00:33:54.037476 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.037451 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 17 00:33:54.037545 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.037527 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 17 00:33:54.038013 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.037996 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8da41563-ba0d-45a3-a634-6d95efdf8857-serving-cert\") pod \"console-operator-696884fb9c-nbp24\" (UID: \"8da41563-ba0d-45a3-a634-6d95efdf8857\") " pod="openshift-console-operator/console-operator-696884fb9c-nbp24" Sep 17 00:33:54.038086 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.038023 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8da41563-ba0d-45a3-a634-6d95efdf8857-config\") pod \"console-operator-696884fb9c-nbp24\" (UID: \"8da41563-ba0d-45a3-a634-6d95efdf8857\") " pod="openshift-console-operator/console-operator-696884fb9c-nbp24" Sep 17 00:33:54.038157 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.038139 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qz7f\" (UniqueName: \"kubernetes.io/projected/8da41563-ba0d-45a3-a634-6d95efdf8857-kube-api-access-4qz7f\") pod \"console-operator-696884fb9c-nbp24\" (UID: \"8da41563-ba0d-45a3-a634-6d95efdf8857\") " pod="openshift-console-operator/console-operator-696884fb9c-nbp24" Sep 17 00:33:54.038205 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.038187 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8da41563-ba0d-45a3-a634-6d95efdf8857-trusted-ca\") pod \"console-operator-696884fb9c-nbp24\" (UID: \"8da41563-ba0d-45a3-a634-6d95efdf8857\") " pod="openshift-console-operator/console-operator-696884fb9c-nbp24" Sep 17 00:33:54.040712 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.040693 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqxk7"] Sep 17 00:33:54.041581 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.041562 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69rz8"] Sep 17 00:33:54.042462 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.042443 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 17 00:33:54.045641 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.045620 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-gcss2"] Sep 17 00:33:54.138988 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.138960 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qz7f\" (UniqueName: \"kubernetes.io/projected/8da41563-ba0d-45a3-a634-6d95efdf8857-kube-api-access-4qz7f\") pod \"console-operator-696884fb9c-nbp24\" (UID: \"8da41563-ba0d-45a3-a634-6d95efdf8857\") " pod="openshift-console-operator/console-operator-696884fb9c-nbp24" Sep 17 00:33:54.139130 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.138992 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b27a182-232b-40f3-a781-c51db96fd889-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kqxk7\" (UID: \"3b27a182-232b-40f3-a781-c51db96fd889\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqxk7" Sep 17 00:33:54.139130 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.139014 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7a57e9f-7ddf-46db-86f6-6772869c49d1-trusted-ca-bundle\") pod \"insights-operator-5844b856db-gcss2\" (UID: \"c7a57e9f-7ddf-46db-86f6-6772869c49d1\") " pod="openshift-insights/insights-operator-5844b856db-gcss2" Sep 17 00:33:54.139250 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.139127 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c7a57e9f-7ddf-46db-86f6-6772869c49d1-snapshots\") pod \"insights-operator-5844b856db-gcss2\" (UID: \"c7a57e9f-7ddf-46db-86f6-6772869c49d1\") " pod="openshift-insights/insights-operator-5844b856db-gcss2" Sep 17 00:33:54.139250 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.139166 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blct9\" (UniqueName: \"kubernetes.io/projected/2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c-kube-api-access-blct9\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69rz8\" (UID: \"2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69rz8" Sep 17 00:33:54.139250 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.139202 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7a57e9f-7ddf-46db-86f6-6772869c49d1-tmp\") pod \"insights-operator-5844b856db-gcss2\" (UID: \"c7a57e9f-7ddf-46db-86f6-6772869c49d1\") " pod="openshift-insights/insights-operator-5844b856db-gcss2" Sep 17 00:33:54.139250 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.139218 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7a57e9f-7ddf-46db-86f6-6772869c49d1-serving-cert\") pod \"insights-operator-5844b856db-gcss2\" (UID: \"c7a57e9f-7ddf-46db-86f6-6772869c49d1\") " pod="openshift-insights/insights-operator-5844b856db-gcss2" Sep 17 00:33:54.139250 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.139234 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7a57e9f-7ddf-46db-86f6-6772869c49d1-service-ca-bundle\") pod \"insights-operator-5844b856db-gcss2\" (UID: \"c7a57e9f-7ddf-46db-86f6-6772869c49d1\") " pod="openshift-insights/insights-operator-5844b856db-gcss2" Sep 17 00:33:54.139250 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.139249 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69rz8\" (UID: \"2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69rz8" Sep 17 00:33:54.139562 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.139301 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8da41563-ba0d-45a3-a634-6d95efdf8857-trusted-ca\") pod \"console-operator-696884fb9c-nbp24\" (UID: \"8da41563-ba0d-45a3-a634-6d95efdf8857\") " pod="openshift-console-operator/console-operator-696884fb9c-nbp24" Sep 17 00:33:54.139562 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.139339 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc5rv\" (UniqueName: \"kubernetes.io/projected/c7a57e9f-7ddf-46db-86f6-6772869c49d1-kube-api-access-wc5rv\") pod \"insights-operator-5844b856db-gcss2\" (UID: \"c7a57e9f-7ddf-46db-86f6-6772869c49d1\") " pod="openshift-insights/insights-operator-5844b856db-gcss2" Sep 17 00:33:54.139562 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.139361 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8da41563-ba0d-45a3-a634-6d95efdf8857-serving-cert\") pod \"console-operator-696884fb9c-nbp24\" (UID: \"8da41563-ba0d-45a3-a634-6d95efdf8857\") " pod="openshift-console-operator/console-operator-696884fb9c-nbp24" Sep 17 00:33:54.139562 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.139377 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69rz8\" (UID: \"2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69rz8" Sep 17 00:33:54.139562 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.139401 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8da41563-ba0d-45a3-a634-6d95efdf8857-config\") pod \"console-operator-696884fb9c-nbp24\" (UID: \"8da41563-ba0d-45a3-a634-6d95efdf8857\") " pod="openshift-console-operator/console-operator-696884fb9c-nbp24" Sep 17 00:33:54.139562 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.139433 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqwmt\" (UniqueName: \"kubernetes.io/projected/3b27a182-232b-40f3-a781-c51db96fd889-kube-api-access-xqwmt\") pod \"cluster-samples-operator-86499c89c4-kqxk7\" (UID: \"3b27a182-232b-40f3-a781-c51db96fd889\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqxk7" Sep 17 00:33:54.139957 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.139936 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8da41563-ba0d-45a3-a634-6d95efdf8857-config\") pod \"console-operator-696884fb9c-nbp24\" (UID: \"8da41563-ba0d-45a3-a634-6d95efdf8857\") " pod="openshift-console-operator/console-operator-696884fb9c-nbp24" Sep 17 00:33:54.140179 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.140159 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8da41563-ba0d-45a3-a634-6d95efdf8857-trusted-ca\") pod \"console-operator-696884fb9c-nbp24\" (UID: \"8da41563-ba0d-45a3-a634-6d95efdf8857\") " pod="openshift-console-operator/console-operator-696884fb9c-nbp24" Sep 17 00:33:54.141635 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.141620 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8da41563-ba0d-45a3-a634-6d95efdf8857-serving-cert\") pod \"console-operator-696884fb9c-nbp24\" (UID: \"8da41563-ba0d-45a3-a634-6d95efdf8857\") " pod="openshift-console-operator/console-operator-696884fb9c-nbp24" Sep 17 00:33:54.145708 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.145690 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qz7f\" (UniqueName: \"kubernetes.io/projected/8da41563-ba0d-45a3-a634-6d95efdf8857-kube-api-access-4qz7f\") pod \"console-operator-696884fb9c-nbp24\" (UID: \"8da41563-ba0d-45a3-a634-6d95efdf8857\") " pod="openshift-console-operator/console-operator-696884fb9c-nbp24" Sep 17 00:33:54.240641 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.240581 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7a57e9f-7ddf-46db-86f6-6772869c49d1-service-ca-bundle\") pod \"insights-operator-5844b856db-gcss2\" (UID: \"c7a57e9f-7ddf-46db-86f6-6772869c49d1\") " pod="openshift-insights/insights-operator-5844b856db-gcss2" Sep 17 00:33:54.240641 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.240609 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69rz8\" (UID: \"2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69rz8" Sep 17 00:33:54.240760 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.240644 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc5rv\" (UniqueName: \"kubernetes.io/projected/c7a57e9f-7ddf-46db-86f6-6772869c49d1-kube-api-access-wc5rv\") pod \"insights-operator-5844b856db-gcss2\" (UID: \"c7a57e9f-7ddf-46db-86f6-6772869c49d1\") " pod="openshift-insights/insights-operator-5844b856db-gcss2" Sep 17 00:33:54.240798 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.240761 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69rz8\" (UID: \"2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69rz8" Sep 17 00:33:54.240831 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.240796 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwmt\" (UniqueName: \"kubernetes.io/projected/3b27a182-232b-40f3-a781-c51db96fd889-kube-api-access-xqwmt\") pod \"cluster-samples-operator-86499c89c4-kqxk7\" (UID: \"3b27a182-232b-40f3-a781-c51db96fd889\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqxk7" Sep 17 00:33:54.240880 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.240852 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b27a182-232b-40f3-a781-c51db96fd889-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kqxk7\" (UID: \"3b27a182-232b-40f3-a781-c51db96fd889\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqxk7" Sep 17 00:33:54.240930 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.240883 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7a57e9f-7ddf-46db-86f6-6772869c49d1-trusted-ca-bundle\") pod \"insights-operator-5844b856db-gcss2\" (UID: \"c7a57e9f-7ddf-46db-86f6-6772869c49d1\") " pod="openshift-insights/insights-operator-5844b856db-gcss2" Sep 17 00:33:54.240930 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.240921 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c7a57e9f-7ddf-46db-86f6-6772869c49d1-snapshots\") pod \"insights-operator-5844b856db-gcss2\" (UID: \"c7a57e9f-7ddf-46db-86f6-6772869c49d1\") " pod="openshift-insights/insights-operator-5844b856db-gcss2" Sep 17 00:33:54.241029 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:33:54.240949 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 00:33:54.241029 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.240961 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blct9\" (UniqueName: \"kubernetes.io/projected/2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c-kube-api-access-blct9\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69rz8\" (UID: \"2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69rz8" Sep 17 00:33:54.241029 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:33:54.241004 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b27a182-232b-40f3-a781-c51db96fd889-samples-operator-tls podName:3b27a182-232b-40f3-a781-c51db96fd889 nodeName:}" failed. No retries permitted until 2026-09-17 00:33:54.740988245 +0000 UTC m=+247.671621730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3b27a182-232b-40f3-a781-c51db96fd889-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kqxk7" (UID: "3b27a182-232b-40f3-a781-c51db96fd889") : secret "samples-operator-tls" not found Sep 17 00:33:54.241029 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.241022 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7a57e9f-7ddf-46db-86f6-6772869c49d1-tmp\") pod \"insights-operator-5844b856db-gcss2\" (UID: \"c7a57e9f-7ddf-46db-86f6-6772869c49d1\") " pod="openshift-insights/insights-operator-5844b856db-gcss2" Sep 17 00:33:54.241229 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.241048 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7a57e9f-7ddf-46db-86f6-6772869c49d1-serving-cert\") pod \"insights-operator-5844b856db-gcss2\" (UID: \"c7a57e9f-7ddf-46db-86f6-6772869c49d1\") " pod="openshift-insights/insights-operator-5844b856db-gcss2" Sep 17 00:33:54.241294 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.241271 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7a57e9f-7ddf-46db-86f6-6772869c49d1-service-ca-bundle\") pod \"insights-operator-5844b856db-gcss2\" (UID: \"c7a57e9f-7ddf-46db-86f6-6772869c49d1\") " pod="openshift-insights/insights-operator-5844b856db-gcss2" Sep 17 00:33:54.241591 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.241546 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69rz8\" (UID: \"2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69rz8" Sep 17 00:33:54.241672 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.241645 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7a57e9f-7ddf-46db-86f6-6772869c49d1-tmp\") pod \"insights-operator-5844b856db-gcss2\" (UID: \"c7a57e9f-7ddf-46db-86f6-6772869c49d1\") " pod="openshift-insights/insights-operator-5844b856db-gcss2" Sep 17 00:33:54.241724 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.241698 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c7a57e9f-7ddf-46db-86f6-6772869c49d1-snapshots\") pod \"insights-operator-5844b856db-gcss2\" (UID: \"c7a57e9f-7ddf-46db-86f6-6772869c49d1\") " pod="openshift-insights/insights-operator-5844b856db-gcss2" Sep 17 00:33:54.241724 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.241718 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7a57e9f-7ddf-46db-86f6-6772869c49d1-trusted-ca-bundle\") pod \"insights-operator-5844b856db-gcss2\" (UID: \"c7a57e9f-7ddf-46db-86f6-6772869c49d1\") " pod="openshift-insights/insights-operator-5844b856db-gcss2" Sep 17 00:33:54.242854 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.242833 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69rz8\" (UID: \"2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69rz8" Sep 17 00:33:54.243357 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.243341 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c7a57e9f-7ddf-46db-86f6-6772869c49d1-serving-cert\") pod \"insights-operator-5844b856db-gcss2\" (UID: \"c7a57e9f-7ddf-46db-86f6-6772869c49d1\") " pod="openshift-insights/insights-operator-5844b856db-gcss2" Sep 17 00:33:54.248324 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.248305 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blct9\" (UniqueName: \"kubernetes.io/projected/2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c-kube-api-access-blct9\") pod \"kube-storage-version-migrator-operator-6c8f84f897-69rz8\" (UID: \"2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69rz8" Sep 17 00:33:54.248430 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.248397 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc5rv\" (UniqueName: \"kubernetes.io/projected/c7a57e9f-7ddf-46db-86f6-6772869c49d1-kube-api-access-wc5rv\") pod \"insights-operator-5844b856db-gcss2\" (UID: \"c7a57e9f-7ddf-46db-86f6-6772869c49d1\") " pod="openshift-insights/insights-operator-5844b856db-gcss2" Sep 17 00:33:54.248430 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.248425 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwmt\" (UniqueName: \"kubernetes.io/projected/3b27a182-232b-40f3-a781-c51db96fd889-kube-api-access-xqwmt\") pod \"cluster-samples-operator-86499c89c4-kqxk7\" (UID: \"3b27a182-232b-40f3-a781-c51db96fd889\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqxk7" Sep 17 00:33:54.250235 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.250213 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-nbp24" Sep 17 00:33:54.348803 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.348769 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69rz8" Sep 17 00:33:54.353812 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.353797 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-gcss2" Sep 17 00:33:54.518602 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.518533 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-nbp24"] Sep 17 00:33:54.521385 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:33:54.521362 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da41563_ba0d_45a3_a634_6d95efdf8857.slice/crio-6ec37fd8a7266c67c3228db23a6cbed67192dbe68f9c0a5003037f95c5f0a51e WatchSource:0}: Error finding container 6ec37fd8a7266c67c3228db23a6cbed67192dbe68f9c0a5003037f95c5f0a51e: Status 404 returned error can't find the container with id 6ec37fd8a7266c67c3228db23a6cbed67192dbe68f9c0a5003037f95c5f0a51e Sep 17 00:33:54.615266 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.615236 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69rz8"] Sep 17 00:33:54.618865 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:33:54.618825 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a87c150_6ca6_43b5_a7fc_e8a6b43feb3c.slice/crio-ca01468b6dcd2c1c68ce35d778a80b7186c63ff107034a2ea1f2447b30da460b WatchSource:0}: Error finding container ca01468b6dcd2c1c68ce35d778a80b7186c63ff107034a2ea1f2447b30da460b: Status 404 returned error can't find the container with id ca01468b6dcd2c1c68ce35d778a80b7186c63ff107034a2ea1f2447b30da460b Sep 17 00:33:54.634157 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.634115 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-gcss2"] Sep 17 00:33:54.637002 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:33:54.636976 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a57e9f_7ddf_46db_86f6_6772869c49d1.slice/crio-12f10e091e6b14df5571b43df8e6ed4f6b1d93f6073f2c7b5fb59bb2a77f5f1c WatchSource:0}: Error finding container 12f10e091e6b14df5571b43df8e6ed4f6b1d93f6073f2c7b5fb59bb2a77f5f1c: Status 404 returned error can't find the container with id 12f10e091e6b14df5571b43df8e6ed4f6b1d93f6073f2c7b5fb59bb2a77f5f1c Sep 17 00:33:54.745518 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:54.745464 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b27a182-232b-40f3-a781-c51db96fd889-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kqxk7\" (UID: \"3b27a182-232b-40f3-a781-c51db96fd889\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqxk7" Sep 17 00:33:54.745655 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:33:54.745602 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 00:33:54.745698 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:33:54.745663 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b27a182-232b-40f3-a781-c51db96fd889-samples-operator-tls podName:3b27a182-232b-40f3-a781-c51db96fd889 nodeName:}" failed. No retries permitted until 2026-09-17 00:33:55.745647413 +0000 UTC m=+248.676280878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3b27a182-232b-40f3-a781-c51db96fd889-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kqxk7" (UID: "3b27a182-232b-40f3-a781-c51db96fd889") : secret "samples-operator-tls" not found Sep 17 00:33:55.520156 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:55.520116 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-nbp24" event={"ID":"8da41563-ba0d-45a3-a634-6d95efdf8857","Type":"ContainerStarted","Data":"6ec37fd8a7266c67c3228db23a6cbed67192dbe68f9c0a5003037f95c5f0a51e"} Sep 17 00:33:55.521929 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:55.521840 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-gcss2" event={"ID":"c7a57e9f-7ddf-46db-86f6-6772869c49d1","Type":"ContainerStarted","Data":"12f10e091e6b14df5571b43df8e6ed4f6b1d93f6073f2c7b5fb59bb2a77f5f1c"} Sep 17 00:33:55.523312 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:55.523263 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69rz8" event={"ID":"2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c","Type":"ContainerStarted","Data":"ca01468b6dcd2c1c68ce35d778a80b7186c63ff107034a2ea1f2447b30da460b"} Sep 17 00:33:55.752801 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:55.752762 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b27a182-232b-40f3-a781-c51db96fd889-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kqxk7\" (UID: \"3b27a182-232b-40f3-a781-c51db96fd889\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqxk7" Sep 17 00:33:55.752994 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:33:55.752958 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 00:33:55.753065 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:33:55.753028 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b27a182-232b-40f3-a781-c51db96fd889-samples-operator-tls podName:3b27a182-232b-40f3-a781-c51db96fd889 nodeName:}" failed. No retries permitted until 2026-09-17 00:33:57.753009742 +0000 UTC m=+250.683643222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3b27a182-232b-40f3-a781-c51db96fd889-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kqxk7" (UID: "3b27a182-232b-40f3-a781-c51db96fd889") : secret "samples-operator-tls" not found Sep 17 00:33:57.770359 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:57.770325 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b27a182-232b-40f3-a781-c51db96fd889-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kqxk7\" (UID: \"3b27a182-232b-40f3-a781-c51db96fd889\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqxk7" Sep 17 00:33:57.770737 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:33:57.770492 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 00:33:57.770737 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:33:57.770582 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b27a182-232b-40f3-a781-c51db96fd889-samples-operator-tls podName:3b27a182-232b-40f3-a781-c51db96fd889 nodeName:}" failed. No retries permitted until 2026-09-17 00:34:01.770563226 +0000 UTC m=+254.701196712 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3b27a182-232b-40f3-a781-c51db96fd889-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kqxk7" (UID: "3b27a182-232b-40f3-a781-c51db96fd889") : secret "samples-operator-tls" not found Sep 17 00:33:58.376073 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.376034 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs\") pod \"network-metrics-daemon-rh4cz\" (UID: \"6155b312-55da-43b1-9b1f-4d51dfbb0d42\") " pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:33:58.376229 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:33:58.376195 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 17 00:33:58.376282 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:33:58.376271 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs podName:6155b312-55da-43b1-9b1f-4d51dfbb0d42 nodeName:}" failed. No retries permitted until 2026-09-17 00:36:00.376251765 +0000 UTC m=+373.306885234 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs") pod "network-metrics-daemon-rh4cz" (UID: "6155b312-55da-43b1-9b1f-4d51dfbb0d42") : secret "metrics-daemon-secret" not found Sep 17 00:33:58.539123 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.539097 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/0.log" Sep 17 00:33:58.539296 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.539135 2582 generic.go:356] "Generic (PLEG): container finished" podID="8da41563-ba0d-45a3-a634-6d95efdf8857" containerID="c528d6f605c8e9f349ca5a0f524ede34dd8d5beb3cb7ccf7b3d69c57d4191072" exitCode=255 Sep 17 00:33:58.539296 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.539168 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-nbp24" event={"ID":"8da41563-ba0d-45a3-a634-6d95efdf8857","Type":"ContainerDied","Data":"c528d6f605c8e9f349ca5a0f524ede34dd8d5beb3cb7ccf7b3d69c57d4191072"} Sep 17 00:33:58.539480 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.539464 2582 scope.go:117] "RemoveContainer" containerID="c528d6f605c8e9f349ca5a0f524ede34dd8d5beb3cb7ccf7b3d69c57d4191072" Sep 17 00:33:58.540880 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.540853 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-gcss2" event={"ID":"c7a57e9f-7ddf-46db-86f6-6772869c49d1","Type":"ContainerStarted","Data":"28bb97ca0a8c4f68b59a3d4d468f8e111c36db3823be355810d83835fa900416"} Sep 17 00:33:58.542711 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.542691 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69rz8" event={"ID":"2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c","Type":"ContainerStarted","Data":"ff5101a0ca80650d7e77df7fdcd5464acfc864549c8a72a50d2896f3abc1c1e6"} Sep 17 00:33:58.566378 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.566323 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69rz8" podStartSLOduration=1.614423227 podStartE2EDuration="4.566309105s" totalImagesPullingTime="2.951885878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:33:58.564882815 +0000 UTC m=+251.495516315" watchObservedRunningTime="2026-09-17 00:33:58.566309105 +0000 UTC m=+251.496942646" Sep 17 00:33:58.579244 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.579198 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-gcss2" podStartSLOduration=1.648790579 podStartE2EDuration="4.579182192s" totalImagesPullingTime="2.930391613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:33:58.578904941 +0000 UTC m=+251.509538429" watchObservedRunningTime="2026-09-17 00:33:58.579182192 +0000 UTC m=+251.509815680" Sep 17 00:33:58.820026 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.819989 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5gblm"] Sep 17 00:33:58.823317 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.823289 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:33:58.825261 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.825229 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 17 00:33:58.825261 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.825237 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4ctq5\"" Sep 17 00:33:58.825509 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.825485 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 17 00:33:58.832045 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.832025 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5gblm"] Sep 17 00:33:58.880413 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.880386 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:33:58.880544 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.880432 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/176754c0-81e9-405a-b0e4-6b9feb1f1320-crio-socket\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:33:58.880544 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.880451 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtk26\" (UniqueName: \"kubernetes.io/projected/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-api-access-wtk26\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:33:58.880670 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.880606 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/176754c0-81e9-405a-b0e4-6b9feb1f1320-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:33:58.880670 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.880641 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/176754c0-81e9-405a-b0e4-6b9feb1f1320-data-volume\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:33:58.981233 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.981172 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/176754c0-81e9-405a-b0e4-6b9feb1f1320-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:33:58.981233 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.981202 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/176754c0-81e9-405a-b0e4-6b9feb1f1320-data-volume\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:33:58.981400 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.981253 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:33:58.981400 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.981309 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/176754c0-81e9-405a-b0e4-6b9feb1f1320-crio-socket\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:33:58.981400 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:33:58.981325 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 00:33:58.981400 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.981330 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtk26\" (UniqueName: \"kubernetes.io/projected/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-api-access-wtk26\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:33:58.981400 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:33:58.981384 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/176754c0-81e9-405a-b0e4-6b9feb1f1320-insights-runtime-extractor-tls podName:176754c0-81e9-405a-b0e4-6b9feb1f1320 nodeName:}" failed. No retries permitted until 2026-09-17 00:33:59.481366453 +0000 UTC m=+252.411999941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/176754c0-81e9-405a-b0e4-6b9feb1f1320-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5gblm" (UID: "176754c0-81e9-405a-b0e4-6b9feb1f1320") : secret "insights-runtime-extractor-tls" not found Sep 17 00:33:58.981400 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:33:58.981387 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 00:33:58.981711 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.981425 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/176754c0-81e9-405a-b0e4-6b9feb1f1320-crio-socket\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:33:58.981711 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:33:58.981448 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-rbac-proxy-cm podName:176754c0-81e9-405a-b0e4-6b9feb1f1320 nodeName:}" failed. No retries permitted until 2026-09-17 00:33:59.481430604 +0000 UTC m=+252.412064086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5gblm" (UID: "176754c0-81e9-405a-b0e4-6b9feb1f1320") : configmap "kube-rbac-proxy" not found Sep 17 00:33:58.981711 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.981602 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/176754c0-81e9-405a-b0e4-6b9feb1f1320-data-volume\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:33:58.988929 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:58.988905 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtk26\" (UniqueName: \"kubernetes.io/projected/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-api-access-wtk26\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:33:59.486176 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:59.486143 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/176754c0-81e9-405a-b0e4-6b9feb1f1320-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:33:59.486347 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:59.486210 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:33:59.486347 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:33:59.486302 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 00:33:59.486459 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:33:59.486348 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 00:33:59.486459 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:33:59.486394 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/176754c0-81e9-405a-b0e4-6b9feb1f1320-insights-runtime-extractor-tls podName:176754c0-81e9-405a-b0e4-6b9feb1f1320 nodeName:}" failed. No retries permitted until 2026-09-17 00:34:00.486370173 +0000 UTC m=+253.417003748 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/176754c0-81e9-405a-b0e4-6b9feb1f1320-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5gblm" (UID: "176754c0-81e9-405a-b0e4-6b9feb1f1320") : secret "insights-runtime-extractor-tls" not found Sep 17 00:33:59.486459 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:33:59.486434 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-rbac-proxy-cm podName:176754c0-81e9-405a-b0e4-6b9feb1f1320 nodeName:}" failed. No retries permitted until 2026-09-17 00:34:00.48641069 +0000 UTC m=+253.417044155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5gblm" (UID: "176754c0-81e9-405a-b0e4-6b9feb1f1320") : configmap "kube-rbac-proxy" not found Sep 17 00:33:59.549984 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:59.549958 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/1.log" Sep 17 00:33:59.550406 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:59.550388 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/0.log" Sep 17 00:33:59.550528 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:59.550428 2582 generic.go:356] "Generic (PLEG): container finished" podID="8da41563-ba0d-45a3-a634-6d95efdf8857" containerID="b371d4900bc5fc1a406c9fd6d789a6256b2ad28108e062e7916df8a22d9f37a5" exitCode=255 Sep 17 00:33:59.550575 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:59.550531 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-nbp24" event={"ID":"8da41563-ba0d-45a3-a634-6d95efdf8857","Type":"ContainerDied","Data":"b371d4900bc5fc1a406c9fd6d789a6256b2ad28108e062e7916df8a22d9f37a5"} Sep 17 00:33:59.550615 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:59.550585 2582 scope.go:117] "RemoveContainer" containerID="c528d6f605c8e9f349ca5a0f524ede34dd8d5beb3cb7ccf7b3d69c57d4191072" Sep 17 00:33:59.550868 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:33:59.550851 2582 scope.go:117] "RemoveContainer" containerID="b371d4900bc5fc1a406c9fd6d789a6256b2ad28108e062e7916df8a22d9f37a5" Sep 17 00:33:59.551050 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:33:59.551033 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-nbp24_openshift-console-operator(8da41563-ba0d-45a3-a634-6d95efdf8857)\"" pod="openshift-console-operator/console-operator-696884fb9c-nbp24" podUID="8da41563-ba0d-45a3-a634-6d95efdf8857" Sep 17 00:34:00.494595 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:00.494543 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:34:00.495007 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:00.494650 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/176754c0-81e9-405a-b0e4-6b9feb1f1320-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:34:00.495007 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:00.494667 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 00:34:00.495007 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:00.494727 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-rbac-proxy-cm podName:176754c0-81e9-405a-b0e4-6b9feb1f1320 nodeName:}" failed. No retries permitted until 2026-09-17 00:34:02.49471276 +0000 UTC m=+255.425346238 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5gblm" (UID: "176754c0-81e9-405a-b0e4-6b9feb1f1320") : configmap "kube-rbac-proxy" not found Sep 17 00:34:00.495007 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:00.494757 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 00:34:00.495007 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:00.494803 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/176754c0-81e9-405a-b0e4-6b9feb1f1320-insights-runtime-extractor-tls podName:176754c0-81e9-405a-b0e4-6b9feb1f1320 nodeName:}" failed. No retries permitted until 2026-09-17 00:34:02.494787605 +0000 UTC m=+255.425421086 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/176754c0-81e9-405a-b0e4-6b9feb1f1320-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5gblm" (UID: "176754c0-81e9-405a-b0e4-6b9feb1f1320") : secret "insights-runtime-extractor-tls" not found Sep 17 00:34:00.554854 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:00.554827 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/1.log" Sep 17 00:34:00.555146 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:00.555126 2582 scope.go:117] "RemoveContainer" containerID="b371d4900bc5fc1a406c9fd6d789a6256b2ad28108e062e7916df8a22d9f37a5" Sep 17 00:34:00.555303 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:00.555284 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-nbp24_openshift-console-operator(8da41563-ba0d-45a3-a634-6d95efdf8857)\"" pod="openshift-console-operator/console-operator-696884fb9c-nbp24" podUID="8da41563-ba0d-45a3-a634-6d95efdf8857" Sep 17 00:34:00.767289 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:00.767216 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kzvtd_6d6d5c16-9a78-4f6e-b33b-bd62d16302b0/dns-node-resolver/0.log" Sep 17 00:34:01.767212 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:01.767183 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d7jth_ddb33aca-e881-421e-b64f-ca6aaa46298d/node-ca/0.log" Sep 17 00:34:01.806726 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:01.806695 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b27a182-232b-40f3-a781-c51db96fd889-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kqxk7\" (UID: \"3b27a182-232b-40f3-a781-c51db96fd889\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqxk7" Sep 17 00:34:01.806859 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:01.806808 2582 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 17 00:34:01.806904 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:01.806861 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b27a182-232b-40f3-a781-c51db96fd889-samples-operator-tls podName:3b27a182-232b-40f3-a781-c51db96fd889 nodeName:}" failed. No retries permitted until 2026-09-17 00:34:09.806848308 +0000 UTC m=+262.737481786 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/3b27a182-232b-40f3-a781-c51db96fd889-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-kqxk7" (UID: "3b27a182-232b-40f3-a781-c51db96fd889") : secret "samples-operator-tls" not found Sep 17 00:34:02.367088 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:02.367061 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6qb8z_446d7623-90a7-453f-b8cc-dc54feffd5b2/serve-healthcheck-canary/0.log" Sep 17 00:34:02.512453 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:02.512419 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:34:02.512646 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:02.512551 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 00:34:02.512646 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:02.512558 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/176754c0-81e9-405a-b0e4-6b9feb1f1320-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:34:02.512646 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:02.512621 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-rbac-proxy-cm podName:176754c0-81e9-405a-b0e4-6b9feb1f1320 nodeName:}" failed. No retries permitted until 2026-09-17 00:34:06.512606547 +0000 UTC m=+259.443240029 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5gblm" (UID: "176754c0-81e9-405a-b0e4-6b9feb1f1320") : configmap "kube-rbac-proxy" not found Sep 17 00:34:02.512770 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:02.512667 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 00:34:02.512770 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:02.512714 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/176754c0-81e9-405a-b0e4-6b9feb1f1320-insights-runtime-extractor-tls podName:176754c0-81e9-405a-b0e4-6b9feb1f1320 nodeName:}" failed. No retries permitted until 2026-09-17 00:34:06.512700084 +0000 UTC m=+259.443333570 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/176754c0-81e9-405a-b0e4-6b9feb1f1320-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5gblm" (UID: "176754c0-81e9-405a-b0e4-6b9feb1f1320") : secret "insights-runtime-extractor-tls" not found Sep 17 00:34:04.250849 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:04.250816 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-nbp24" Sep 17 00:34:04.251238 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:04.250868 2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-nbp24" Sep 17 00:34:04.251238 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:04.251176 2582 scope.go:117] "RemoveContainer" containerID="b371d4900bc5fc1a406c9fd6d789a6256b2ad28108e062e7916df8a22d9f37a5" Sep 17 00:34:04.251340 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:04.251321 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-nbp24_openshift-console-operator(8da41563-ba0d-45a3-a634-6d95efdf8857)\"" pod="openshift-console-operator/console-operator-696884fb9c-nbp24" podUID="8da41563-ba0d-45a3-a634-6d95efdf8857" Sep 17 00:34:06.542800 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:06.542764 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:34:06.543136 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:06.542852 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/176754c0-81e9-405a-b0e4-6b9feb1f1320-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:34:06.543136 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:06.542885 2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 17 00:34:06.543136 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:06.542937 2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 17 00:34:06.543136 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:06.542949 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-rbac-proxy-cm podName:176754c0-81e9-405a-b0e4-6b9feb1f1320 nodeName:}" failed. No retries permitted until 2026-09-17 00:34:14.542932024 +0000 UTC m=+267.473565489 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5gblm" (UID: "176754c0-81e9-405a-b0e4-6b9feb1f1320") : configmap "kube-rbac-proxy" not found Sep 17 00:34:06.543136 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:06.542976 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/176754c0-81e9-405a-b0e4-6b9feb1f1320-insights-runtime-extractor-tls podName:176754c0-81e9-405a-b0e4-6b9feb1f1320 nodeName:}" failed. No retries permitted until 2026-09-17 00:34:14.542965713 +0000 UTC m=+267.473599202 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/176754c0-81e9-405a-b0e4-6b9feb1f1320-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5gblm" (UID: "176754c0-81e9-405a-b0e4-6b9feb1f1320") : secret "insights-runtime-extractor-tls" not found Sep 17 00:34:09.868364 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:09.868291 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b27a182-232b-40f3-a781-c51db96fd889-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kqxk7\" (UID: \"3b27a182-232b-40f3-a781-c51db96fd889\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqxk7" Sep 17 00:34:09.870727 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:09.870707 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/3b27a182-232b-40f3-a781-c51db96fd889-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-kqxk7\" (UID: \"3b27a182-232b-40f3-a781-c51db96fd889\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqxk7" Sep 17 00:34:09.941128 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:09.941093 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqxk7" Sep 17 00:34:10.208239 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:10.208169 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqxk7"] Sep 17 00:34:10.586818 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:10.586787 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqxk7" event={"ID":"3b27a182-232b-40f3-a781-c51db96fd889","Type":"ContainerStarted","Data":"4406be8ccc76b2f986f6ef4f3ba6d2fd0c3668fbec20f0cb933b701cf27fccc9"} Sep 17 00:34:12.594160 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:12.594124 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqxk7" event={"ID":"3b27a182-232b-40f3-a781-c51db96fd889","Type":"ContainerStarted","Data":"b15a193b001f515e57119ab3bae12a230776fdb487f0716ede07e2fafb2081eb"} Sep 17 00:34:12.594548 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:12.594179 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqxk7" event={"ID":"3b27a182-232b-40f3-a781-c51db96fd889","Type":"ContainerStarted","Data":"687c24316a09560d802cdd5b37114195515774df360f75c121b1ac568c39763c"} Sep 17 00:34:12.608951 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:12.608906 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-kqxk7" podStartSLOduration=16.635492707 podStartE2EDuration="18.608891667s" totalImagesPullingTime="1.97339896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:12.608040301 +0000 UTC m=+265.538673789" watchObservedRunningTime="2026-09-17 00:34:12.608891667 +0000 UTC m=+265.539525154" Sep 17 00:34:14.606516 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:14.606476 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/176754c0-81e9-405a-b0e4-6b9feb1f1320-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:34:14.606851 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:14.606560 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:34:14.607039 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:14.607011 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/176754c0-81e9-405a-b0e4-6b9feb1f1320-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:34:14.608855 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:14.608835 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/176754c0-81e9-405a-b0e4-6b9feb1f1320-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5gblm\" (UID: \"176754c0-81e9-405a-b0e4-6b9feb1f1320\") " pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:34:14.734558 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:14.734518 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5gblm" Sep 17 00:34:15.022204 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:15.022134 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5gblm"] Sep 17 00:34:15.025120 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:34:15.025093 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod176754c0_81e9_405a_b0e4_6b9feb1f1320.slice/crio-de8dbdd45524bbc29d406865af3f320dbd234cac7903ce2683d79194da496716 WatchSource:0}: Error finding container de8dbdd45524bbc29d406865af3f320dbd234cac7903ce2683d79194da496716: Status 404 returned error can't find the container with id de8dbdd45524bbc29d406865af3f320dbd234cac7903ce2683d79194da496716 Sep 17 00:34:15.607375 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:15.607344 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5gblm" event={"ID":"176754c0-81e9-405a-b0e4-6b9feb1f1320","Type":"ContainerStarted","Data":"797558b450b7408aa3875d10accceee2b5c4627c5690fb8a42adda57ed6148fa"} Sep 17 00:34:15.607811 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:15.607395 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5gblm" event={"ID":"176754c0-81e9-405a-b0e4-6b9feb1f1320","Type":"ContainerStarted","Data":"de8dbdd45524bbc29d406865af3f320dbd234cac7903ce2683d79194da496716"} Sep 17 00:34:16.616421 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:16.616390 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5gblm" event={"ID":"176754c0-81e9-405a-b0e4-6b9feb1f1320","Type":"ContainerStarted","Data":"791bfb8dc973ecdba20871aae1f9f3e65c00db9318a63eece37f7ac3cb7909cb"} Sep 17 00:34:17.622070 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:17.622031 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5gblm" event={"ID":"176754c0-81e9-405a-b0e4-6b9feb1f1320","Type":"ContainerStarted","Data":"1e544ff15a0ab2e55121ba923361de535241969340082a445f2b88b9b46e0b68"} Sep 17 00:34:17.638516 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:17.638448 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5gblm" podStartSLOduration=17.384591673 podStartE2EDuration="19.638434643s" totalImagesPullingTime="2.25384297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 00:33:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:17.637470997 +0000 UTC m=+270.568104486" watchObservedRunningTime="2026-09-17 00:34:17.638434643 +0000 UTC m=+270.569068173" Sep 17 00:34:17.646168 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:17.646119 2582 scope.go:117] "RemoveContainer" containerID="b371d4900bc5fc1a406c9fd6d789a6256b2ad28108e062e7916df8a22d9f37a5" Sep 17 00:34:18.629558 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:18.629520 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/1.log" Sep 17 00:34:18.630721 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:18.630691 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-nbp24" event={"ID":"8da41563-ba0d-45a3-a634-6d95efdf8857","Type":"ContainerStarted","Data":"09eaecd25543a1d468c82939f334bbeb282573fb0335c30baaaf341c55beb495"} Sep 17 00:34:18.631108 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:18.631083 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-nbp24" Sep 17 00:34:18.637459 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:18.637435 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-nbp24" Sep 17 00:34:18.647092 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:18.647052 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-nbp24" podStartSLOduration=22.604027799 podStartE2EDuration="25.647040063s" totalImagesPullingTime="3.043012264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:18.645834444 +0000 UTC m=+271.576467930" watchObservedRunningTime="2026-09-17 00:34:18.647040063 +0000 UTC m=+271.577673553" Sep 17 00:34:23.121269 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.121235 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-758869bcb7-jtbqg"] Sep 17 00:34:23.121652 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:23.121446 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" podUID="b1b2f065-efd1-4d6c-8ec8-5c116b880c66" Sep 17 00:34:23.131149 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.131125 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-n2s4x"] Sep 17 00:34:23.134519 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.134467 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-n2s4x" Sep 17 00:34:23.134967 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.134948 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-cq9pb"] Sep 17 00:34:23.136669 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.136640 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 17 00:34:23.136669 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.136650 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-f87jf\"" Sep 17 00:34:23.137928 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.137914 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-cq9pb" Sep 17 00:34:23.139968 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.139949 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 17 00:34:23.140135 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.140036 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 17 00:34:23.140299 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.140270 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-9b6xq\"" Sep 17 00:34:23.145880 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.145857 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-n2s4x"] Sep 17 00:34:23.164324 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.164296 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-cq9pb"] Sep 17 00:34:23.276114 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.276086 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z7gm\" (UniqueName: \"kubernetes.io/projected/a7c70a6f-dd20-4a97-bc6c-79eaf9cd53e9-kube-api-access-2z7gm\") pod \"downloads-76f99584f4-cq9pb\" (UID: \"a7c70a6f-dd20-4a97-bc6c-79eaf9cd53e9\") " pod="openshift-console/downloads-76f99584f4-cq9pb" Sep 17 00:34:23.276114 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.276116 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dc8cc146-d16c-455c-96f3-28bf57efc91c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-n2s4x\" (UID: \"dc8cc146-d16c-455c-96f3-28bf57efc91c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-n2s4x" Sep 17 00:34:23.377045 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.376970 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z7gm\" (UniqueName: \"kubernetes.io/projected/a7c70a6f-dd20-4a97-bc6c-79eaf9cd53e9-kube-api-access-2z7gm\") pod \"downloads-76f99584f4-cq9pb\" (UID: \"a7c70a6f-dd20-4a97-bc6c-79eaf9cd53e9\") " pod="openshift-console/downloads-76f99584f4-cq9pb" Sep 17 00:34:23.377045 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.377007 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dc8cc146-d16c-455c-96f3-28bf57efc91c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-n2s4x\" (UID: \"dc8cc146-d16c-455c-96f3-28bf57efc91c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-n2s4x" Sep 17 00:34:23.379743 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.379720 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dc8cc146-d16c-455c-96f3-28bf57efc91c-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-n2s4x\" (UID: \"dc8cc146-d16c-455c-96f3-28bf57efc91c\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-n2s4x" Sep 17 00:34:23.386018 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.385998 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z7gm\" (UniqueName: \"kubernetes.io/projected/a7c70a6f-dd20-4a97-bc6c-79eaf9cd53e9-kube-api-access-2z7gm\") pod \"downloads-76f99584f4-cq9pb\" (UID: \"a7c70a6f-dd20-4a97-bc6c-79eaf9cd53e9\") " pod="openshift-console/downloads-76f99584f4-cq9pb" Sep 17 00:34:23.446184 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.446159 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-n2s4x" Sep 17 00:34:23.452193 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.452170 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-cq9pb" Sep 17 00:34:23.647322 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.647255 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:34:23.654849 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.654827 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:34:23.713866 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.713835 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-n2s4x"] Sep 17 00:34:23.717355 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:34:23.717333 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8cc146_d16c_455c_96f3_28bf57efc91c.slice/crio-dc11261096d15e5cdfb3fa7687a1ee9b9c4b06d28d3b2a7c2cef1f9b5dc36106 WatchSource:0}: Error finding container dc11261096d15e5cdfb3fa7687a1ee9b9c4b06d28d3b2a7c2cef1f9b5dc36106: Status 404 returned error can't find the container with id dc11261096d15e5cdfb3fa7687a1ee9b9c4b06d28d3b2a7c2cef1f9b5dc36106 Sep 17 00:34:23.721773 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.721751 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-cq9pb"] Sep 17 00:34:23.724306 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:34:23.724284 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c70a6f_dd20_4a97_bc6c_79eaf9cd53e9.slice/crio-d094dda4b1e517ef318afa51748b9d721e4472df142f6f2722d1081bafa7bcbb WatchSource:0}: Error finding container d094dda4b1e517ef318afa51748b9d721e4472df142f6f2722d1081bafa7bcbb: Status 404 returned error can't find the container with id d094dda4b1e517ef318afa51748b9d721e4472df142f6f2722d1081bafa7bcbb Sep 17 00:34:23.780640 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.780621 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-trusted-ca\") pod \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " Sep 17 00:34:23.780739 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.780654 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-bound-sa-token\") pod \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " Sep 17 00:34:23.780739 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.780696 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxkdr\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-kube-api-access-qxkdr\") pod \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " Sep 17 00:34:23.780739 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.780721 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-image-registry-private-configuration\") pod \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " Sep 17 00:34:23.780739 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.780736 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-certificates\") pod \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " Sep 17 00:34:23.780902 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.780763 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-installation-pull-secrets\") pod \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " Sep 17 00:34:23.780902 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.780797 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-ca-trust-extracted\") pod \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\" (UID: \"b1b2f065-efd1-4d6c-8ec8-5c116b880c66\") " Sep 17 00:34:23.781291 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.781250 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b1b2f065-efd1-4d6c-8ec8-5c116b880c66" (UID: "b1b2f065-efd1-4d6c-8ec8-5c116b880c66"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 00:34:23.781380 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.781315 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b1b2f065-efd1-4d6c-8ec8-5c116b880c66" (UID: "b1b2f065-efd1-4d6c-8ec8-5c116b880c66"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:34:23.781447 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.781412 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b1b2f065-efd1-4d6c-8ec8-5c116b880c66" (UID: "b1b2f065-efd1-4d6c-8ec8-5c116b880c66"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 00:34:23.783165 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.783143 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b1b2f065-efd1-4d6c-8ec8-5c116b880c66" (UID: "b1b2f065-efd1-4d6c-8ec8-5c116b880c66"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:34:23.783231 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.783181 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-kube-api-access-qxkdr" (OuterVolumeSpecName: "kube-api-access-qxkdr") pod "b1b2f065-efd1-4d6c-8ec8-5c116b880c66" (UID: "b1b2f065-efd1-4d6c-8ec8-5c116b880c66"). InnerVolumeSpecName "kube-api-access-qxkdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:34:23.783395 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.783379 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b1b2f065-efd1-4d6c-8ec8-5c116b880c66" (UID: "b1b2f065-efd1-4d6c-8ec8-5c116b880c66"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:34:23.783447 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.783395 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b1b2f065-efd1-4d6c-8ec8-5c116b880c66" (UID: "b1b2f065-efd1-4d6c-8ec8-5c116b880c66"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:34:23.882193 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.882164 2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-installation-pull-secrets\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:34:23.882193 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.882191 2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-ca-trust-extracted\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:34:23.882193 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.882201 2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-trusted-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:34:23.882407 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.882210 2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-bound-sa-token\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:34:23.882407 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.882219 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxkdr\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-kube-api-access-qxkdr\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:34:23.882407 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.882228 2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-image-registry-private-configuration\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:34:23.882407 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:23.882236 2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-certificates\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:34:24.653560 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:24.653453 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-cq9pb" event={"ID":"a7c70a6f-dd20-4a97-bc6c-79eaf9cd53e9","Type":"ContainerStarted","Data":"d094dda4b1e517ef318afa51748b9d721e4472df142f6f2722d1081bafa7bcbb"} Sep 17 00:34:24.655363 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:24.655303 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-n2s4x" event={"ID":"dc8cc146-d16c-455c-96f3-28bf57efc91c","Type":"ContainerStarted","Data":"dc11261096d15e5cdfb3fa7687a1ee9b9c4b06d28d3b2a7c2cef1f9b5dc36106"} Sep 17 00:34:24.655363 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:24.655320 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-758869bcb7-jtbqg" Sep 17 00:34:24.687180 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:24.687146 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-758869bcb7-jtbqg"] Sep 17 00:34:24.690746 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:24.690715 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-758869bcb7-jtbqg"] Sep 17 00:34:24.790584 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:24.790553 2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b1b2f065-efd1-4d6c-8ec8-5c116b880c66-registry-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:34:25.649759 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:25.649725 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1b2f065-efd1-4d6c-8ec8-5c116b880c66" path="/var/lib/kubelet/pods/b1b2f065-efd1-4d6c-8ec8-5c116b880c66/volumes" Sep 17 00:34:25.660767 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:25.660729 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-n2s4x" event={"ID":"dc8cc146-d16c-455c-96f3-28bf57efc91c","Type":"ContainerStarted","Data":"00598071436dd403188d6a87bc34af4f8917ea523e15d5c1399bb2ca66503e93"} Sep 17 00:34:25.661201 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:25.660993 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-n2s4x" Sep 17 00:34:25.666212 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:25.666170 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-n2s4x" Sep 17 00:34:25.674463 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:25.674421 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-n2s4x" podStartSLOduration=1.569148327 podStartE2EDuration="2.674406157s" totalImagesPullingTime="1.10525783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:34:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:25.673446118 +0000 UTC m=+278.604079605" watchObservedRunningTime="2026-09-17 00:34:25.674406157 +0000 UTC m=+278.605039644" Sep 17 00:34:26.533076 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.533034 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9"] Sep 17 00:34:26.538979 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.538957 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9" Sep 17 00:34:26.541779 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.541588 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 17 00:34:26.541779 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.541629 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 17 00:34:26.541779 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.541645 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 17 00:34:26.541779 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.541659 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 17 00:34:26.541779 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.541594 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 17 00:34:26.542116 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.541968 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-slclv\"" Sep 17 00:34:26.546053 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.546004 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9"] Sep 17 00:34:26.705628 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.705594 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/913b3f67-2d48-42ca-879a-7406e4bc0abe-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-kqtm9\" (UID: \"913b3f67-2d48-42ca-879a-7406e4bc0abe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9" Sep 17 00:34:26.706009 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.705691 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/913b3f67-2d48-42ca-879a-7406e4bc0abe-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-kqtm9\" (UID: \"913b3f67-2d48-42ca-879a-7406e4bc0abe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9" Sep 17 00:34:26.706009 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.705720 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlsnd\" (UniqueName: \"kubernetes.io/projected/913b3f67-2d48-42ca-879a-7406e4bc0abe-kube-api-access-nlsnd\") pod \"prometheus-operator-68bcd7f987-kqtm9\" (UID: \"913b3f67-2d48-42ca-879a-7406e4bc0abe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9" Sep 17 00:34:26.706009 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.705772 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/913b3f67-2d48-42ca-879a-7406e4bc0abe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-kqtm9\" (UID: \"913b3f67-2d48-42ca-879a-7406e4bc0abe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9" Sep 17 00:34:26.806924 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.806885 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/913b3f67-2d48-42ca-879a-7406e4bc0abe-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-kqtm9\" (UID: \"913b3f67-2d48-42ca-879a-7406e4bc0abe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9" Sep 17 00:34:26.807100 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.806936 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlsnd\" (UniqueName: \"kubernetes.io/projected/913b3f67-2d48-42ca-879a-7406e4bc0abe-kube-api-access-nlsnd\") pod \"prometheus-operator-68bcd7f987-kqtm9\" (UID: \"913b3f67-2d48-42ca-879a-7406e4bc0abe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9" Sep 17 00:34:26.807100 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.806968 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/913b3f67-2d48-42ca-879a-7406e4bc0abe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-kqtm9\" (UID: \"913b3f67-2d48-42ca-879a-7406e4bc0abe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9" Sep 17 00:34:26.807100 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.807048 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/913b3f67-2d48-42ca-879a-7406e4bc0abe-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-kqtm9\" (UID: \"913b3f67-2d48-42ca-879a-7406e4bc0abe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9" Sep 17 00:34:26.807753 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.807727 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/913b3f67-2d48-42ca-879a-7406e4bc0abe-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-kqtm9\" (UID: \"913b3f67-2d48-42ca-879a-7406e4bc0abe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9" Sep 17 00:34:26.810034 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.810011 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/913b3f67-2d48-42ca-879a-7406e4bc0abe-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-kqtm9\" (UID: \"913b3f67-2d48-42ca-879a-7406e4bc0abe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9" Sep 17 00:34:26.810146 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.810018 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/913b3f67-2d48-42ca-879a-7406e4bc0abe-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-kqtm9\" (UID: \"913b3f67-2d48-42ca-879a-7406e4bc0abe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9" Sep 17 00:34:26.814898 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.814872 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlsnd\" (UniqueName: \"kubernetes.io/projected/913b3f67-2d48-42ca-879a-7406e4bc0abe-kube-api-access-nlsnd\") pod \"prometheus-operator-68bcd7f987-kqtm9\" (UID: \"913b3f67-2d48-42ca-879a-7406e4bc0abe\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9" Sep 17 00:34:26.849559 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:26.849528 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9" Sep 17 00:34:27.129665 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:27.129590 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9"] Sep 17 00:34:27.133054 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:34:27.133021 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod913b3f67_2d48_42ca_879a_7406e4bc0abe.slice/crio-f0089dab92adfa3f9999378cd4b74096991ca8d1a0a2e99b4e9759db84258ac4 WatchSource:0}: Error finding container f0089dab92adfa3f9999378cd4b74096991ca8d1a0a2e99b4e9759db84258ac4: Status 404 returned error can't find the container with id f0089dab92adfa3f9999378cd4b74096991ca8d1a0a2e99b4e9759db84258ac4 Sep 17 00:34:27.670890 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:27.670852 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9" event={"ID":"913b3f67-2d48-42ca-879a-7406e4bc0abe","Type":"ContainerStarted","Data":"f0089dab92adfa3f9999378cd4b74096991ca8d1a0a2e99b4e9759db84258ac4"} Sep 17 00:34:28.298288 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:28.298230 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" podUID="5cc2bc3b-4179-4230-9a3b-de67ca98ad50" Sep 17 00:34:28.298288 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:28.298231 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8vx7h" podUID="3f05fc1f-6243-45c9-8f0b-d8d6a40b6225" Sep 17 00:34:28.677141 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:28.677109 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9" event={"ID":"913b3f67-2d48-42ca-879a-7406e4bc0abe","Type":"ContainerStarted","Data":"7e13ee297998245a1debd89a2aa818244d57b304bbced26d0a6ce5b3864e1ef0"} Sep 17 00:34:28.677259 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:28.677139 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8vx7h" Sep 17 00:34:28.677259 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:28.677160 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" Sep 17 00:34:29.682514 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:29.682463 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9" event={"ID":"913b3f67-2d48-42ca-879a-7406e4bc0abe","Type":"ContainerStarted","Data":"c73e01b86fb50cdc6159492a81d3f9e870c63e4939a7304cf922f015f6e198a0"} Sep 17 00:34:29.704320 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:29.704260 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-kqtm9" podStartSLOduration=2.250788927 podStartE2EDuration="3.704245068s" totalImagesPullingTime="1.453456141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:29.702349286 +0000 UTC m=+282.632982772" watchObservedRunningTime="2026-09-17 00:34:29.704245068 +0000 UTC m=+282.634878554" Sep 17 00:34:31.859319 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.859285 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5"] Sep 17 00:34:31.896874 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.896839 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5"] Sep 17 00:34:31.897022 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.896892 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km"] Sep 17 00:34:31.897022 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.896966 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5" Sep 17 00:34:31.899024 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.898812 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 17 00:34:31.899024 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.898984 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-z6zjt\"" Sep 17 00:34:31.899754 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.899562 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 17 00:34:31.912385 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.912358 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km"] Sep 17 00:34:31.912518 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.912401 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tsscd"] Sep 17 00:34:31.912518 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.912466 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:31.914283 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.914265 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-vsjz5\"" Sep 17 00:34:31.914603 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.914580 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 17 00:34:31.914689 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.914648 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 17 00:34:31.914944 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.914922 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 17 00:34:31.930577 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.930559 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:31.932761 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.932702 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 17 00:34:31.932761 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.932711 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 17 00:34:31.932932 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.932819 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 17 00:34:31.932932 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.932912 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s8d7w\"" Sep 17 00:34:31.952972 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.952944 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-b6l4k\" (UID: \"5cc2bc3b-4179-4230-9a3b-de67ca98ad50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" Sep 17 00:34:31.953087 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.952998 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:34:31.955342 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.955322 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3f05fc1f-6243-45c9-8f0b-d8d6a40b6225-metrics-tls\") pod \"dns-default-8vx7h\" (UID: \"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225\") " pod="openshift-dns/dns-default-8vx7h" Sep 17 00:34:31.955529 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.955486 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5cc2bc3b-4179-4230-9a3b-de67ca98ad50-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-b6l4k\" (UID: \"5cc2bc3b-4179-4230-9a3b-de67ca98ad50\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" Sep 17 00:34:31.979275 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.979249 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-n9svt\"" Sep 17 00:34:31.979377 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.979296 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-m9f5h\"" Sep 17 00:34:31.988478 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.988458 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" Sep 17 00:34:31.988579 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:31.988550 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8vx7h" Sep 17 00:34:32.054255 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.054212 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0af8806-ec80-41b8-bb64-73837f89b294-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.054405 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.054275 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8593ed54-bd76-4ca8-9f9e-78be23ae0f77-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-sn9w5\" (UID: \"8593ed54-bd76-4ca8-9f9e-78be23ae0f77\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5" Sep 17 00:34:32.054405 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.054314 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/81ae202e-68da-46af-84c4-572272397c69-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rp4km\" (UID: \"81ae202e-68da-46af-84c4-572272397c69\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:32.054550 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.054412 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld4br\" (UniqueName: \"kubernetes.io/projected/81ae202e-68da-46af-84c4-572272397c69-kube-api-access-ld4br\") pod \"kube-state-metrics-59fbdd77d-rp4km\" (UID: \"81ae202e-68da-46af-84c4-572272397c69\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:32.054550 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.054486 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0af8806-ec80-41b8-bb64-73837f89b294-metrics-client-ca\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.054665 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.054582 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81ae202e-68da-46af-84c4-572272397c69-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rp4km\" (UID: \"81ae202e-68da-46af-84c4-572272397c69\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:32.054665 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.054609 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f0af8806-ec80-41b8-bb64-73837f89b294-node-exporter-tls\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.054665 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.054647 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8593ed54-bd76-4ca8-9f9e-78be23ae0f77-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-sn9w5\" (UID: \"8593ed54-bd76-4ca8-9f9e-78be23ae0f77\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5" Sep 17 00:34:32.054813 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.054677 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/81ae202e-68da-46af-84c4-572272397c69-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rp4km\" (UID: \"81ae202e-68da-46af-84c4-572272397c69\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:32.054813 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.054726 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f0af8806-ec80-41b8-bb64-73837f89b294-node-exporter-accelerators-collector-config\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.054813 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.054757 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0af8806-ec80-41b8-bb64-73837f89b294-node-exporter-wtmp\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.054813 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.054780 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/81ae202e-68da-46af-84c4-572272397c69-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rp4km\" (UID: \"81ae202e-68da-46af-84c4-572272397c69\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:32.054813 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.054808 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81ae202e-68da-46af-84c4-572272397c69-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rp4km\" (UID: \"81ae202e-68da-46af-84c4-572272397c69\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:32.055089 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.054836 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhxgs\" (UniqueName: \"kubernetes.io/projected/8593ed54-bd76-4ca8-9f9e-78be23ae0f77-kube-api-access-dhxgs\") pod \"openshift-state-metrics-67dcf8c979-sn9w5\" (UID: \"8593ed54-bd76-4ca8-9f9e-78be23ae0f77\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5" Sep 17 00:34:32.055089 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.054870 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8593ed54-bd76-4ca8-9f9e-78be23ae0f77-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-sn9w5\" (UID: \"8593ed54-bd76-4ca8-9f9e-78be23ae0f77\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5" Sep 17 00:34:32.055089 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.054890 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0af8806-ec80-41b8-bb64-73837f89b294-sys\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.055089 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.054914 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0af8806-ec80-41b8-bb64-73837f89b294-root\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.055089 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.054934 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k6gg\" (UniqueName: \"kubernetes.io/projected/f0af8806-ec80-41b8-bb64-73837f89b294-kube-api-access-6k6gg\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.055089 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.054960 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f0af8806-ec80-41b8-bb64-73837f89b294-node-exporter-textfile\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.156175 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156087 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0af8806-ec80-41b8-bb64-73837f89b294-metrics-client-ca\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.156175 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156136 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81ae202e-68da-46af-84c4-572272397c69-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rp4km\" (UID: \"81ae202e-68da-46af-84c4-572272397c69\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:32.156175 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156152 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f0af8806-ec80-41b8-bb64-73837f89b294-node-exporter-tls\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.156388 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156226 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8593ed54-bd76-4ca8-9f9e-78be23ae0f77-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-sn9w5\" (UID: \"8593ed54-bd76-4ca8-9f9e-78be23ae0f77\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5" Sep 17 00:34:32.156388 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156259 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/81ae202e-68da-46af-84c4-572272397c69-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rp4km\" (UID: \"81ae202e-68da-46af-84c4-572272397c69\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:32.156388 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156309 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f0af8806-ec80-41b8-bb64-73837f89b294-node-exporter-accelerators-collector-config\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.156388 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156333 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0af8806-ec80-41b8-bb64-73837f89b294-node-exporter-wtmp\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.156388 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156356 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/81ae202e-68da-46af-84c4-572272397c69-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rp4km\" (UID: \"81ae202e-68da-46af-84c4-572272397c69\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:32.156388 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156387 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81ae202e-68da-46af-84c4-572272397c69-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rp4km\" (UID: \"81ae202e-68da-46af-84c4-572272397c69\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:32.156634 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156417 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhxgs\" (UniqueName: \"kubernetes.io/projected/8593ed54-bd76-4ca8-9f9e-78be23ae0f77-kube-api-access-dhxgs\") pod \"openshift-state-metrics-67dcf8c979-sn9w5\" (UID: \"8593ed54-bd76-4ca8-9f9e-78be23ae0f77\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5" Sep 17 00:34:32.156634 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156454 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8593ed54-bd76-4ca8-9f9e-78be23ae0f77-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-sn9w5\" (UID: \"8593ed54-bd76-4ca8-9f9e-78be23ae0f77\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5" Sep 17 00:34:32.156634 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156480 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0af8806-ec80-41b8-bb64-73837f89b294-sys\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.156634 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156536 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0af8806-ec80-41b8-bb64-73837f89b294-root\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.156634 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156560 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k6gg\" (UniqueName: \"kubernetes.io/projected/f0af8806-ec80-41b8-bb64-73837f89b294-kube-api-access-6k6gg\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.156634 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156587 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f0af8806-ec80-41b8-bb64-73837f89b294-node-exporter-textfile\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.156634 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156624 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0af8806-ec80-41b8-bb64-73837f89b294-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.156968 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156662 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8593ed54-bd76-4ca8-9f9e-78be23ae0f77-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-sn9w5\" (UID: \"8593ed54-bd76-4ca8-9f9e-78be23ae0f77\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5" Sep 17 00:34:32.156968 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156697 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/81ae202e-68da-46af-84c4-572272397c69-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rp4km\" (UID: \"81ae202e-68da-46af-84c4-572272397c69\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:32.156968 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156735 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld4br\" (UniqueName: \"kubernetes.io/projected/81ae202e-68da-46af-84c4-572272397c69-kube-api-access-ld4br\") pod \"kube-state-metrics-59fbdd77d-rp4km\" (UID: \"81ae202e-68da-46af-84c4-572272397c69\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:32.156968 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156764 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f0af8806-ec80-41b8-bb64-73837f89b294-node-exporter-wtmp\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.156968 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156806 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f0af8806-ec80-41b8-bb64-73837f89b294-metrics-client-ca\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.156968 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156871 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f0af8806-ec80-41b8-bb64-73837f89b294-sys\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.156968 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156915 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f0af8806-ec80-41b8-bb64-73837f89b294-root\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.157307 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.156968 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/81ae202e-68da-46af-84c4-572272397c69-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-rp4km\" (UID: \"81ae202e-68da-46af-84c4-572272397c69\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:32.157307 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:32.157058 2582 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 17 00:34:32.157307 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:32.157128 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81ae202e-68da-46af-84c4-572272397c69-kube-state-metrics-tls podName:81ae202e-68da-46af-84c4-572272397c69 nodeName:}" failed. No retries permitted until 2026-09-17 00:34:32.657107832 +0000 UTC m=+285.587741329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/81ae202e-68da-46af-84c4-572272397c69-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-rp4km" (UID: "81ae202e-68da-46af-84c4-572272397c69") : secret "kube-state-metrics-tls" not found Sep 17 00:34:32.157477 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.157381 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81ae202e-68da-46af-84c4-572272397c69-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-rp4km\" (UID: \"81ae202e-68da-46af-84c4-572272397c69\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:32.157661 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.157638 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f0af8806-ec80-41b8-bb64-73837f89b294-node-exporter-textfile\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.158015 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.157778 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/81ae202e-68da-46af-84c4-572272397c69-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-rp4km\" (UID: \"81ae202e-68da-46af-84c4-572272397c69\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:32.158015 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.157969 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8593ed54-bd76-4ca8-9f9e-78be23ae0f77-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-sn9w5\" (UID: \"8593ed54-bd76-4ca8-9f9e-78be23ae0f77\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5" Sep 17 00:34:32.158367 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.158342 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f0af8806-ec80-41b8-bb64-73837f89b294-node-exporter-accelerators-collector-config\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.159191 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.159168 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f0af8806-ec80-41b8-bb64-73837f89b294-node-exporter-tls\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.160019 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.159994 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f0af8806-ec80-41b8-bb64-73837f89b294-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.160122 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.160022 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8593ed54-bd76-4ca8-9f9e-78be23ae0f77-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-sn9w5\" (UID: \"8593ed54-bd76-4ca8-9f9e-78be23ae0f77\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5" Sep 17 00:34:32.160347 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.160322 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8593ed54-bd76-4ca8-9f9e-78be23ae0f77-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-sn9w5\" (UID: \"8593ed54-bd76-4ca8-9f9e-78be23ae0f77\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5" Sep 17 00:34:32.160779 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.160754 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/81ae202e-68da-46af-84c4-572272397c69-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-rp4km\" (UID: \"81ae202e-68da-46af-84c4-572272397c69\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:32.165699 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.165676 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k6gg\" (UniqueName: \"kubernetes.io/projected/f0af8806-ec80-41b8-bb64-73837f89b294-kube-api-access-6k6gg\") pod \"node-exporter-tsscd\" (UID: \"f0af8806-ec80-41b8-bb64-73837f89b294\") " pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.166271 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.166250 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld4br\" (UniqueName: \"kubernetes.io/projected/81ae202e-68da-46af-84c4-572272397c69-kube-api-access-ld4br\") pod \"kube-state-metrics-59fbdd77d-rp4km\" (UID: \"81ae202e-68da-46af-84c4-572272397c69\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:32.166337 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.166283 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhxgs\" (UniqueName: \"kubernetes.io/projected/8593ed54-bd76-4ca8-9f9e-78be23ae0f77-kube-api-access-dhxgs\") pod \"openshift-state-metrics-67dcf8c979-sn9w5\" (UID: \"8593ed54-bd76-4ca8-9f9e-78be23ae0f77\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5" Sep 17 00:34:32.209868 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.209840 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5" Sep 17 00:34:32.241239 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.241212 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tsscd" Sep 17 00:34:32.271267 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:34:32.271237 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0af8806_ec80_41b8_bb64_73837f89b294.slice/crio-c96851f8baa40099a3038105c7282c3e140f02ee570157c571226445f0e08b9c WatchSource:0}: Error finding container c96851f8baa40099a3038105c7282c3e140f02ee570157c571226445f0e08b9c: Status 404 returned error can't find the container with id c96851f8baa40099a3038105c7282c3e140f02ee570157c571226445f0e08b9c Sep 17 00:34:32.290284 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.290038 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k"] Sep 17 00:34:32.297382 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:34:32.297346 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc2bc3b_4179_4230_9a3b_de67ca98ad50.slice/crio-c1b35e7dd9e9f3c23138bc4817ba0d45770bfed6d3c16c97048b53fa6ad39ab6 WatchSource:0}: Error finding container c1b35e7dd9e9f3c23138bc4817ba0d45770bfed6d3c16c97048b53fa6ad39ab6: Status 404 returned error can't find the container with id c1b35e7dd9e9f3c23138bc4817ba0d45770bfed6d3c16c97048b53fa6ad39ab6 Sep 17 00:34:32.318229 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.318199 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8vx7h"] Sep 17 00:34:32.322059 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:34:32.322031 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f05fc1f_6243_45c9_8f0b_d8d6a40b6225.slice/crio-bcabd562c87c5d31b6b1a48c888b74729a06be1c9865eadf41a6bc4794b5ee7f WatchSource:0}: Error finding container bcabd562c87c5d31b6b1a48c888b74729a06be1c9865eadf41a6bc4794b5ee7f: Status 404 returned error can't find the container with id bcabd562c87c5d31b6b1a48c888b74729a06be1c9865eadf41a6bc4794b5ee7f Sep 17 00:34:32.515617 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.515539 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5"] Sep 17 00:34:32.518565 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:34:32.518541 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8593ed54_bd76_4ca8_9f9e_78be23ae0f77.slice/crio-461e5d96955bf1487a3d02200fadde0f9ed81f60152fb1442e64dfad0e5d3137 WatchSource:0}: Error finding container 461e5d96955bf1487a3d02200fadde0f9ed81f60152fb1442e64dfad0e5d3137: Status 404 returned error can't find the container with id 461e5d96955bf1487a3d02200fadde0f9ed81f60152fb1442e64dfad0e5d3137 Sep 17 00:34:32.660722 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.660698 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/81ae202e-68da-46af-84c4-572272397c69-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rp4km\" (UID: \"81ae202e-68da-46af-84c4-572272397c69\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:32.663670 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.663639 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/81ae202e-68da-46af-84c4-572272397c69-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-rp4km\" (UID: \"81ae202e-68da-46af-84c4-572272397c69\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:32.694854 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.694822 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tsscd" event={"ID":"f0af8806-ec80-41b8-bb64-73837f89b294","Type":"ContainerStarted","Data":"c96851f8baa40099a3038105c7282c3e140f02ee570157c571226445f0e08b9c"} Sep 17 00:34:32.696723 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.696699 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5" event={"ID":"8593ed54-bd76-4ca8-9f9e-78be23ae0f77","Type":"ContainerStarted","Data":"ad5a3abdd34c9c23c87c5f199bda19ae8f67b0e20f47e03e203594b41c38b6d5"} Sep 17 00:34:32.696831 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.696753 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5" event={"ID":"8593ed54-bd76-4ca8-9f9e-78be23ae0f77","Type":"ContainerStarted","Data":"461e5d96955bf1487a3d02200fadde0f9ed81f60152fb1442e64dfad0e5d3137"} Sep 17 00:34:32.698210 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.698189 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8vx7h" event={"ID":"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225","Type":"ContainerStarted","Data":"bcabd562c87c5d31b6b1a48c888b74729a06be1c9865eadf41a6bc4794b5ee7f"} Sep 17 00:34:32.699572 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.699540 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" event={"ID":"5cc2bc3b-4179-4230-9a3b-de67ca98ad50","Type":"ContainerStarted","Data":"c1b35e7dd9e9f3c23138bc4817ba0d45770bfed6d3c16c97048b53fa6ad39ab6"} Sep 17 00:34:32.824197 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.823342 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" Sep 17 00:34:32.919373 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.919344 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 00:34:32.942320 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.942274 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 00:34:32.942463 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.942431 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:32.945199 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.944714 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 17 00:34:32.945199 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.944762 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 17 00:34:32.945199 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.944714 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 17 00:34:32.945199 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.944718 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 17 00:34:32.945199 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.945089 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 17 00:34:32.945199 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.945125 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cw7rj\"" Sep 17 00:34:32.945849 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.945489 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 17 00:34:32.945849 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.945743 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 17 00:34:32.946039 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.946024 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 17 00:34:32.946142 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:32.946120 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 17 00:34:33.063835 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.063801 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.064057 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.063857 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.064057 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.063912 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.064057 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.063942 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-config-volume\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.064057 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.063967 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f95g\" (UniqueName: \"kubernetes.io/projected/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-kube-api-access-9f95g\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.064057 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.063996 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.064057 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.064018 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.064057 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.064050 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.064384 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.064102 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-config-out\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.064384 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.064135 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.064384 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.064281 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.064384 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.064307 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-web-config\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.064384 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.064333 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.138147 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.137779 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km"] Sep 17 00:34:33.140971 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:34:33.140941 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ae202e_68da_46af_84c4_572272397c69.slice/crio-3d86c8de910086d782c8e4341764fcf92e613ef40f22fe7594e59258aac902e3 WatchSource:0}: Error finding container 3d86c8de910086d782c8e4341764fcf92e613ef40f22fe7594e59258aac902e3: Status 404 returned error can't find the container with id 3d86c8de910086d782c8e4341764fcf92e613ef40f22fe7594e59258aac902e3 Sep 17 00:34:33.165077 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.164987 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.165077 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.165023 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-config-volume\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.165077 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.165044 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f95g\" (UniqueName: \"kubernetes.io/projected/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-kube-api-access-9f95g\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.165292 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.165102 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.165292 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.165128 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.165292 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.165160 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.165292 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.165207 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-config-out\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.165292 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.165242 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.165292 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.165285 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.165611 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.165307 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-web-config\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.165611 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:33.165312 2582 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Sep 17 00:34:33.165611 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.165332 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.165611 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:33.165377 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-main-tls podName:9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08 nodeName:}" failed. No retries permitted until 2026-09-17 00:34:33.665357788 +0000 UTC m=+286.595991269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08") : secret "alertmanager-main-tls" not found Sep 17 00:34:33.165611 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.165420 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.165611 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.165463 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.165973 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.165773 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.166769 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.166426 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.166769 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.166767 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.169390 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.169345 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.169846 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.169776 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-config-out\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.169968 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.169941 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.170417 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.170379 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.170680 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.170649 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-config-volume\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.171206 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.171186 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-web-config\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.171293 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.171264 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.172326 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.172285 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.178064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.178018 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f95g\" (UniqueName: \"kubernetes.io/projected/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-kube-api-access-9f95g\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.671292 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.671131 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.674364 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.674333 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.706700 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.706663 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5" event={"ID":"8593ed54-bd76-4ca8-9f9e-78be23ae0f77","Type":"ContainerStarted","Data":"f79b13dfa40023034c400484d28c76e95a17c02384e70b047cb0f9f9e4104545"} Sep 17 00:34:33.708088 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.708060 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" event={"ID":"81ae202e-68da-46af-84c4-572272397c69","Type":"ContainerStarted","Data":"3d86c8de910086d782c8e4341764fcf92e613ef40f22fe7594e59258aac902e3"} Sep 17 00:34:33.853729 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.853693 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-645649dfc7-f88xn"] Sep 17 00:34:33.856054 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.855769 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:34:33.888016 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.887987 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-645649dfc7-f88xn"] Sep 17 00:34:33.888164 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.888096 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:33.890601 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.890578 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 17 00:34:33.890771 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.890755 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 17 00:34:33.890972 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.890952 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 17 00:34:33.891089 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.891068 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ercgcvc67nfia\"" Sep 17 00:34:33.891089 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.891081 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 17 00:34:33.891186 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.891113 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 17 00:34:33.891230 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:33.891193 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-k6rx6\"" Sep 17 00:34:34.074682 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.074644 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wkzn\" (UniqueName: \"kubernetes.io/projected/6dc1abe4-a68e-4cf7-890f-473426223b55-kube-api-access-4wkzn\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.075099 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.074718 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dc1abe4-a68e-4cf7-890f-473426223b55-metrics-client-ca\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.075099 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.074749 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6dc1abe4-a68e-4cf7-890f-473426223b55-secret-grpc-tls\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.075099 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.074802 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6dc1abe4-a68e-4cf7-890f-473426223b55-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.075099 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.074839 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6dc1abe4-a68e-4cf7-890f-473426223b55-secret-thanos-querier-tls\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.075099 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.074877 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6dc1abe4-a68e-4cf7-890f-473426223b55-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.075099 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.074961 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6dc1abe4-a68e-4cf7-890f-473426223b55-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.075099 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.074990 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6dc1abe4-a68e-4cf7-890f-473426223b55-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.175510 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.175471 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dc1abe4-a68e-4cf7-890f-473426223b55-metrics-client-ca\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.175693 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.175537 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6dc1abe4-a68e-4cf7-890f-473426223b55-secret-grpc-tls\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.175693 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.175574 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6dc1abe4-a68e-4cf7-890f-473426223b55-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.175693 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.175618 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6dc1abe4-a68e-4cf7-890f-473426223b55-secret-thanos-querier-tls\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.175693 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.175658 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6dc1abe4-a68e-4cf7-890f-473426223b55-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.175904 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.175710 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6dc1abe4-a68e-4cf7-890f-473426223b55-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.175904 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.175743 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6dc1abe4-a68e-4cf7-890f-473426223b55-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.175904 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.175800 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wkzn\" (UniqueName: \"kubernetes.io/projected/6dc1abe4-a68e-4cf7-890f-473426223b55-kube-api-access-4wkzn\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.176318 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.176284 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dc1abe4-a68e-4cf7-890f-473426223b55-metrics-client-ca\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.178694 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.178666 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6dc1abe4-a68e-4cf7-890f-473426223b55-secret-grpc-tls\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.179190 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.179161 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6dc1abe4-a68e-4cf7-890f-473426223b55-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.179293 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.179216 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6dc1abe4-a68e-4cf7-890f-473426223b55-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.179293 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.179214 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6dc1abe4-a68e-4cf7-890f-473426223b55-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.179373 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.179289 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6dc1abe4-a68e-4cf7-890f-473426223b55-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.179637 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.179621 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6dc1abe4-a68e-4cf7-890f-473426223b55-secret-thanos-querier-tls\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.182984 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.182958 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wkzn\" (UniqueName: \"kubernetes.io/projected/6dc1abe4-a68e-4cf7-890f-473426223b55-kube-api-access-4wkzn\") pod \"thanos-querier-645649dfc7-f88xn\" (UID: \"6dc1abe4-a68e-4cf7-890f-473426223b55\") " pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:34.198896 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:34.198875 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:36.269734 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.269698 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-76f456b74c-f8bcc"] Sep 17 00:34:36.283917 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.283882 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76f456b74c-f8bcc"] Sep 17 00:34:36.284071 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.284030 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.286990 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.286771 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-v9nvh\"" Sep 17 00:34:36.286990 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.286785 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 17 00:34:36.286990 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.286790 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 17 00:34:36.286990 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.286846 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 17 00:34:36.286990 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.286783 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 17 00:34:36.287544 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.287396 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2csnd9aoj6jgh\"" Sep 17 00:34:36.298680 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.298656 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/22b3bd80-9c5d-4926-9331-9bf124eb903c-metrics-server-audit-profiles\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.298779 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.298713 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22b3bd80-9c5d-4926-9331-9bf124eb903c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.298825 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.298796 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/22b3bd80-9c5d-4926-9331-9bf124eb903c-secret-metrics-server-tls\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.298859 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.298845 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/22b3bd80-9c5d-4926-9331-9bf124eb903c-client-ca-bundle\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.298897 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.298866 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/22b3bd80-9c5d-4926-9331-9bf124eb903c-secret-metrics-server-client-certs\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.298948 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.298929 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxh9s\" (UniqueName: \"kubernetes.io/projected/22b3bd80-9c5d-4926-9331-9bf124eb903c-kube-api-access-vxh9s\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.299008 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.298971 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/22b3bd80-9c5d-4926-9331-9bf124eb903c-audit-log\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.399780 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.399742 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/22b3bd80-9c5d-4926-9331-9bf124eb903c-secret-metrics-server-client-certs\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.399956 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.399802 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxh9s\" (UniqueName: \"kubernetes.io/projected/22b3bd80-9c5d-4926-9331-9bf124eb903c-kube-api-access-vxh9s\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.399956 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.399833 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/22b3bd80-9c5d-4926-9331-9bf124eb903c-audit-log\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.399956 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.399869 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/22b3bd80-9c5d-4926-9331-9bf124eb903c-metrics-server-audit-profiles\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.399956 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.399919 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22b3bd80-9c5d-4926-9331-9bf124eb903c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.400162 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.399967 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/22b3bd80-9c5d-4926-9331-9bf124eb903c-secret-metrics-server-tls\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.400162 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.400026 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/22b3bd80-9c5d-4926-9331-9bf124eb903c-client-ca-bundle\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.400366 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.400316 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/22b3bd80-9c5d-4926-9331-9bf124eb903c-audit-log\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.400880 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.400833 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22b3bd80-9c5d-4926-9331-9bf124eb903c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.401055 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.401037 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/22b3bd80-9c5d-4926-9331-9bf124eb903c-metrics-server-audit-profiles\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.403012 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.402974 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/22b3bd80-9c5d-4926-9331-9bf124eb903c-secret-metrics-server-tls\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.403135 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.403115 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/22b3bd80-9c5d-4926-9331-9bf124eb903c-client-ca-bundle\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.403290 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.403271 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/22b3bd80-9c5d-4926-9331-9bf124eb903c-secret-metrics-server-client-certs\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.407474 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.407450 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxh9s\" (UniqueName: \"kubernetes.io/projected/22b3bd80-9c5d-4926-9331-9bf124eb903c-kube-api-access-vxh9s\") pod \"metrics-server-76f456b74c-f8bcc\" (UID: \"22b3bd80-9c5d-4926-9331-9bf124eb903c\") " pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:36.595767 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:36.595731 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:42.199281 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.199171 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-645649dfc7-f88xn"] Sep 17 00:34:42.209572 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:34:42.209540 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc1abe4_a68e_4cf7_890f_473426223b55.slice/crio-b3a171ebecb92c7e99756aa181ff49c442f21efb6783b8f29095c15361d17e2c WatchSource:0}: Error finding container b3a171ebecb92c7e99756aa181ff49c442f21efb6783b8f29095c15361d17e2c: Status 404 returned error can't find the container with id b3a171ebecb92c7e99756aa181ff49c442f21efb6783b8f29095c15361d17e2c Sep 17 00:34:42.449799 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.449722 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-76f456b74c-f8bcc"] Sep 17 00:34:42.486746 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:34:42.486710 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b3bd80_9c5d_4926_9331_9bf124eb903c.slice/crio-2b0087ed35e302b3a1dac14f412eb49be48bbf16c0a1d9246bf101ecc225f826 WatchSource:0}: Error finding container 2b0087ed35e302b3a1dac14f412eb49be48bbf16c0a1d9246bf101ecc225f826: Status 404 returned error can't find the container with id 2b0087ed35e302b3a1dac14f412eb49be48bbf16c0a1d9246bf101ecc225f826 Sep 17 00:34:42.500151 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.500054 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 00:34:42.512679 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:34:42.512644 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fcb49c8_bd4b_498e_b6d1_6ed6b0823a08.slice/crio-fcd0165cf5e911ed7a7501220cffdba44fe9bfa3ea7e087ffbde2fecba9dfa7e WatchSource:0}: Error finding container fcd0165cf5e911ed7a7501220cffdba44fe9bfa3ea7e087ffbde2fecba9dfa7e: Status 404 returned error can't find the container with id fcd0165cf5e911ed7a7501220cffdba44fe9bfa3ea7e087ffbde2fecba9dfa7e Sep 17 00:34:42.748771 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.748683 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" event={"ID":"81ae202e-68da-46af-84c4-572272397c69","Type":"ContainerStarted","Data":"578379393cbf5b40962eebdab4cafd2e281b4927639a4d9cb7edfa66ec6cfcb9"} Sep 17 00:34:42.749060 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.749033 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" event={"ID":"81ae202e-68da-46af-84c4-572272397c69","Type":"ContainerStarted","Data":"e79b78be036f8139440f287666280064b45fa1d673abb1507febf92ac3c1b204"} Sep 17 00:34:42.749183 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.749074 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" event={"ID":"81ae202e-68da-46af-84c4-572272397c69","Type":"ContainerStarted","Data":"dacadd2a26dea4057ee0ebc6f5b830e15430847c349b555708758bb90925109a"} Sep 17 00:34:42.752377 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.752337 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5" event={"ID":"8593ed54-bd76-4ca8-9f9e-78be23ae0f77","Type":"ContainerStarted","Data":"523731b3bd7d945d7c8b48674b8d12a3deb32132ab6da57e3b69b3d49fbd9a65"} Sep 17 00:34:42.754604 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.754582 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8vx7h" event={"ID":"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225","Type":"ContainerStarted","Data":"3c9759a0fef3d39c3aca930be07668a982beeec43b31de961932d3fdf35a17d6"} Sep 17 00:34:42.754773 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.754753 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8vx7h" event={"ID":"3f05fc1f-6243-45c9-8f0b-d8d6a40b6225","Type":"ContainerStarted","Data":"bca84ca27e9f01285eb7cc88bdf0267933df3b7ed97c0c6790993aa081105ae7"} Sep 17 00:34:42.754909 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.754894 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8vx7h" Sep 17 00:34:42.756811 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.756784 2582 generic.go:356] "Generic (PLEG): container finished" podID="f0af8806-ec80-41b8-bb64-73837f89b294" containerID="1f56f6e7f378495f31d647b0532e77ab810a3436b56276f1ffe1b0a69ecf6754" exitCode=0 Sep 17 00:34:42.756917 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.756872 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tsscd" event={"ID":"f0af8806-ec80-41b8-bb64-73837f89b294","Type":"ContainerDied","Data":"1f56f6e7f378495f31d647b0532e77ab810a3436b56276f1ffe1b0a69ecf6754"} Sep 17 00:34:42.758399 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.758379 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" event={"ID":"22b3bd80-9c5d-4926-9331-9bf124eb903c","Type":"ContainerStarted","Data":"2b0087ed35e302b3a1dac14f412eb49be48bbf16c0a1d9246bf101ecc225f826"} Sep 17 00:34:42.760692 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.760666 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" event={"ID":"5cc2bc3b-4179-4230-9a3b-de67ca98ad50","Type":"ContainerStarted","Data":"44fa646dd13ecde88d12f3b80a832589fcb41703c6282305aa3fbda41b0737ed"} Sep 17 00:34:42.764830 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.764795 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08","Type":"ContainerStarted","Data":"fcd0165cf5e911ed7a7501220cffdba44fe9bfa3ea7e087ffbde2fecba9dfa7e"} Sep 17 00:34:42.766625 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.766582 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-rp4km" podStartSLOduration=2.688524358 podStartE2EDuration="11.766566153s" totalImagesPullingTime="9.078041795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:42.764953359 +0000 UTC m=+295.695586858" watchObservedRunningTime="2026-09-17 00:34:42.766566153 +0000 UTC m=+295.697199642" Sep 17 00:34:42.767343 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.767290 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" event={"ID":"6dc1abe4-a68e-4cf7-890f-473426223b55","Type":"ContainerStarted","Data":"b3a171ebecb92c7e99756aa181ff49c442f21efb6783b8f29095c15361d17e2c"} Sep 17 00:34:42.769308 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.769287 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-cq9pb" event={"ID":"a7c70a6f-dd20-4a97-bc6c-79eaf9cd53e9","Type":"ContainerStarted","Data":"36bb3928147214f115089873f455a00870ba19b29829a32c7977860cbad064ef"} Sep 17 00:34:42.769633 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.769614 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-cq9pb" Sep 17 00:34:42.780327 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.780301 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-cq9pb" Sep 17 00:34:42.811150 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.811012 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-b6l4k" podStartSLOduration=264.188643908 podStartE2EDuration="4m33.810989133s" totalImagesPullingTime="9.622345225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:30:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:42.810782309 +0000 UTC m=+295.741415799" watchObservedRunningTime="2026-09-17 00:34:42.810989133 +0000 UTC m=+295.741622620" Sep 17 00:34:42.811364 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.811325 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8vx7h" podStartSLOduration=251.927797884 podStartE2EDuration="4m21.811314961s" totalImagesPullingTime="9.883517077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:30:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:42.79866947 +0000 UTC m=+295.729302958" watchObservedRunningTime="2026-09-17 00:34:42.811314961 +0000 UTC m=+295.741948449" Sep 17 00:34:42.843532 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.843138 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sn9w5" podStartSLOduration=2.397673397 podStartE2EDuration="11.843118561s" totalImagesPullingTime="9.445445164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:42.826845886 +0000 UTC m=+295.757479374" watchObservedRunningTime="2026-09-17 00:34:42.843118561 +0000 UTC m=+295.773752049" Sep 17 00:34:42.848520 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:42.843684 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-cq9pb" podStartSLOduration=1.3609341879999999 podStartE2EDuration="19.843675092s" totalImagesPullingTime="18.482740904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:34:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:42.842824211 +0000 UTC m=+295.773457700" watchObservedRunningTime="2026-09-17 00:34:42.843675092 +0000 UTC m=+295.774308581" Sep 17 00:34:43.648533 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:34:43.648449 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-rh4cz" podUID="6155b312-55da-43b1-9b1f-4d51dfbb0d42" Sep 17 00:34:43.781511 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:43.781470 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tsscd" event={"ID":"f0af8806-ec80-41b8-bb64-73837f89b294","Type":"ContainerStarted","Data":"f456614867dfc5a4bfe1fd6713a0c0b970d45e81a7802a69a28aa10189a03061"} Sep 17 00:34:43.781686 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:43.781560 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tsscd" event={"ID":"f0af8806-ec80-41b8-bb64-73837f89b294","Type":"ContainerStarted","Data":"d4804e2efab1b35dd042dbf909b6236d382b1766450e639a8275491803762e41"} Sep 17 00:34:43.801469 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:43.801405 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tsscd" podStartSLOduration=2.750137958 podStartE2EDuration="12.801382642s" totalImagesPullingTime="9.934255837s" totalInitContainerRuntime="116.988847ms" isStatefulPod=true podCreationTimestamp="2026-09-17 00:34:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:43.799521138 +0000 UTC m=+296.730154622" watchObservedRunningTime="2026-09-17 00:34:43.801382642 +0000 UTC m=+296.732016131" Sep 17 00:34:46.797762 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:46.797729 2582 generic.go:356] "Generic (PLEG): container finished" podID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerID="46c16b4e926bb0f3629802e3eaddf5b0319c665ced861f50682abbff3ea43cfb" exitCode=0 Sep 17 00:34:46.798114 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:46.797971 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08","Type":"ContainerDied","Data":"46c16b4e926bb0f3629802e3eaddf5b0319c665ced861f50682abbff3ea43cfb"} Sep 17 00:34:46.800862 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:46.800832 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" event={"ID":"6dc1abe4-a68e-4cf7-890f-473426223b55","Type":"ContainerStarted","Data":"6aa3b59f309975904b0573b45c2efd075a08e9cfc98087acac6a6c4d22bac0be"} Sep 17 00:34:46.800965 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:46.800903 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" event={"ID":"6dc1abe4-a68e-4cf7-890f-473426223b55","Type":"ContainerStarted","Data":"5c81a4f020770d0e41afb96b8ca5d554309f83ef906be26bd030c3fe5fca179a"} Sep 17 00:34:46.805340 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:46.805303 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" event={"ID":"22b3bd80-9c5d-4926-9331-9bf124eb903c","Type":"ContainerStarted","Data":"b4f5899d18988fdbfcfc1789f0509feda2798379faa6cde8d181d93cffa0df72"} Sep 17 00:34:46.835989 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:46.835934 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" podStartSLOduration=6.915628323 podStartE2EDuration="10.83591493s" totalImagesPullingTime="3.920286607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:46.835346701 +0000 UTC m=+299.765980189" watchObservedRunningTime="2026-09-17 00:34:46.83591493 +0000 UTC m=+299.766548414" Sep 17 00:34:47.558782 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:47.558752 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/1.log" Sep 17 00:34:48.572787 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:47.564334 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/1.log" Sep 17 00:34:48.572787 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:47.579526 2582 kubelet.go:1648] "Image garbage collection succeeded" Sep 17 00:34:48.572787 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:47.815710 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" event={"ID":"6dc1abe4-a68e-4cf7-890f-473426223b55","Type":"ContainerStarted","Data":"4e08aa778d4ac436bb85d9e65c81bc62d14f118a4a1d8ea575af5314b410963d"} Sep 17 00:34:49.830138 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:49.830111 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08","Type":"ContainerStarted","Data":"7fd1a84d5c22ee6dacb5d3b7456f5f7e0d5a7284e9aa0b2c66c5c917582be7e8"} Sep 17 00:34:49.830565 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:49.830195 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08","Type":"ContainerStarted","Data":"a0bd09e8dbe0cc6bafcf9926231a9a798c62e50d643b0e4d78db9e9c7aaf75e5"} Sep 17 00:34:49.830565 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:49.830243 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08","Type":"ContainerStarted","Data":"10b11d264c42a249fb14fa2b6eb1b2ffbc428e64641b09805699b6a274603e98"} Sep 17 00:34:49.830565 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:49.830282 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08","Type":"ContainerStarted","Data":"785332a8f065870c0837b48e98d3070d160c5b072bb37f122bcc05431112cad1"} Sep 17 00:34:49.830565 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:49.830311 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08","Type":"ContainerStarted","Data":"ac4f18589b2f2ec31ed17966b06b6752bd0bd608237566076ba7360486685430"} Sep 17 00:34:49.833691 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:49.833663 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" event={"ID":"6dc1abe4-a68e-4cf7-890f-473426223b55","Type":"ContainerStarted","Data":"cb2d505612936246b6f498f3e78e2727a5f42a6443b221f6a7f13cc5091cff59"} Sep 17 00:34:49.833800 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:49.833739 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" event={"ID":"6dc1abe4-a68e-4cf7-890f-473426223b55","Type":"ContainerStarted","Data":"aa79976d53ea2892c434f2a36536b15c724c5c9aa5f3705db84017a7fd70a46f"} Sep 17 00:34:49.833800 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:49.833786 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" event={"ID":"6dc1abe4-a68e-4cf7-890f-473426223b55","Type":"ContainerStarted","Data":"94db80553f2011426ee9fc165c63e0ebbae653796b261a2b2b983f21f2c5b356"} Sep 17 00:34:49.833960 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:49.833926 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:49.854877 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:49.854819 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" podStartSLOduration=10.789993195 podStartE2EDuration="16.854798116s" totalImagesPullingTime="6.064804921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:34:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:49.852580365 +0000 UTC m=+302.783213851" watchObservedRunningTime="2026-09-17 00:34:49.854798116 +0000 UTC m=+302.785431604" Sep 17 00:34:50.843310 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:50.843271 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08","Type":"ContainerStarted","Data":"2f445d1cc6a69f65df85b5c76cf1d45c90f880cf8e82294600c29c72bedef2cc"} Sep 17 00:34:50.867303 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:50.867254 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=12.640369361 podStartE2EDuration="18.867231924s" totalImagesPullingTime="6.20150622s" totalInitContainerRuntime="25.356343ms" isStatefulPod=false podCreationTimestamp="2026-09-17 00:34:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:34:50.864885194 +0000 UTC m=+303.795518694" watchObservedRunningTime="2026-09-17 00:34:50.867231924 +0000 UTC m=+303.797865412" Sep 17 00:34:53.787988 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:53.787961 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8vx7h" Sep 17 00:34:54.644386 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:54.644354 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:34:55.850943 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:55.850916 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-645649dfc7-f88xn" Sep 17 00:34:56.596819 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:56.596784 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:34:56.597047 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:34:56.596875 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:35:02.893268 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:02.893233 2582 generic.go:356] "Generic (PLEG): container finished" podID="c7a57e9f-7ddf-46db-86f6-6772869c49d1" containerID="28bb97ca0a8c4f68b59a3d4d468f8e111c36db3823be355810d83835fa900416" exitCode=0 Sep 17 00:35:02.893706 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:02.893286 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-gcss2" event={"ID":"c7a57e9f-7ddf-46db-86f6-6772869c49d1","Type":"ContainerDied","Data":"28bb97ca0a8c4f68b59a3d4d468f8e111c36db3823be355810d83835fa900416"} Sep 17 00:35:02.893776 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:02.893705 2582 scope.go:117] "RemoveContainer" containerID="28bb97ca0a8c4f68b59a3d4d468f8e111c36db3823be355810d83835fa900416" Sep 17 00:35:03.635879 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:03.635851 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8vx7h_3f05fc1f-6243-45c9-8f0b-d8d6a40b6225/dns/0.log" Sep 17 00:35:03.835052 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:03.835021 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8vx7h_3f05fc1f-6243-45c9-8f0b-d8d6a40b6225/kube-rbac-proxy/0.log" Sep 17 00:35:03.898878 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:03.898816 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-gcss2" event={"ID":"c7a57e9f-7ddf-46db-86f6-6772869c49d1","Type":"ContainerStarted","Data":"83e6c65ff57278a57aae69a072dd9c8804ee985955abb24e320e8467ecf89c70"} Sep 17 00:35:05.034599 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:05.034566 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kzvtd_6d6d5c16-9a78-4f6e-b33b-bd62d16302b0/dns-node-resolver/0.log" Sep 17 00:35:05.634917 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:05.634892 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6qb8z_446d7623-90a7-453f-b8cc-dc54feffd5b2/serve-healthcheck-canary/0.log" Sep 17 00:35:16.603579 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:16.603491 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:35:16.611183 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:16.611155 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-76f456b74c-f8bcc" Sep 17 00:35:29.010795 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:29.010706 2582 generic.go:356] "Generic (PLEG): container finished" podID="2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c" containerID="ff5101a0ca80650d7e77df7fdcd5464acfc864549c8a72a50d2896f3abc1c1e6" exitCode=0 Sep 17 00:35:29.010795 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:29.010780 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69rz8" event={"ID":"2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c","Type":"ContainerDied","Data":"ff5101a0ca80650d7e77df7fdcd5464acfc864549c8a72a50d2896f3abc1c1e6"} Sep 17 00:35:29.011205 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:29.011084 2582 scope.go:117] "RemoveContainer" containerID="ff5101a0ca80650d7e77df7fdcd5464acfc864549c8a72a50d2896f3abc1c1e6" Sep 17 00:35:30.016128 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:30.016095 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-69rz8" event={"ID":"2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c","Type":"ContainerStarted","Data":"79e484ea06e00e5a855a50e453bdc4e63b97fabb27596a348ac4e4163ceb4b80"} Sep 17 00:35:52.079539 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:52.079443 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 00:35:52.080567 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:52.080530 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="alertmanager" containerID="cri-o://ac4f18589b2f2ec31ed17966b06b6752bd0bd608237566076ba7360486685430" gracePeriod=120 Sep 17 00:35:52.081381 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:52.081092 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="prom-label-proxy" containerID="cri-o://2f445d1cc6a69f65df85b5c76cf1d45c90f880cf8e82294600c29c72bedef2cc" gracePeriod=120 Sep 17 00:35:52.081381 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:52.081242 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="kube-rbac-proxy-metric" containerID="cri-o://7fd1a84d5c22ee6dacb5d3b7456f5f7e0d5a7284e9aa0b2c66c5c917582be7e8" gracePeriod=120 Sep 17 00:35:52.081381 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:52.081338 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="kube-rbac-proxy" containerID="cri-o://a0bd09e8dbe0cc6bafcf9926231a9a798c62e50d643b0e4d78db9e9c7aaf75e5" gracePeriod=120 Sep 17 00:35:52.081910 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:52.081418 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="kube-rbac-proxy-web" containerID="cri-o://10b11d264c42a249fb14fa2b6eb1b2ffbc428e64641b09805699b6a274603e98" gracePeriod=120 Sep 17 00:35:52.081910 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:52.081485 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="config-reloader" containerID="cri-o://785332a8f065870c0837b48e98d3070d160c5b072bb37f122bcc05431112cad1" gracePeriod=120 Sep 17 00:35:53.117044 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.117010 2582 generic.go:356] "Generic (PLEG): container finished" podID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerID="2f445d1cc6a69f65df85b5c76cf1d45c90f880cf8e82294600c29c72bedef2cc" exitCode=0 Sep 17 00:35:53.117044 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.117035 2582 generic.go:356] "Generic (PLEG): container finished" podID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerID="a0bd09e8dbe0cc6bafcf9926231a9a798c62e50d643b0e4d78db9e9c7aaf75e5" exitCode=0 Sep 17 00:35:53.117044 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.117041 2582 generic.go:356] "Generic (PLEG): container finished" podID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerID="785332a8f065870c0837b48e98d3070d160c5b072bb37f122bcc05431112cad1" exitCode=0 Sep 17 00:35:53.117044 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.117048 2582 generic.go:356] "Generic (PLEG): container finished" podID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerID="ac4f18589b2f2ec31ed17966b06b6752bd0bd608237566076ba7360486685430" exitCode=0 Sep 17 00:35:53.117556 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.117083 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08","Type":"ContainerDied","Data":"2f445d1cc6a69f65df85b5c76cf1d45c90f880cf8e82294600c29c72bedef2cc"} Sep 17 00:35:53.117556 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.117148 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08","Type":"ContainerDied","Data":"a0bd09e8dbe0cc6bafcf9926231a9a798c62e50d643b0e4d78db9e9c7aaf75e5"} Sep 17 00:35:53.117556 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.117175 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08","Type":"ContainerDied","Data":"785332a8f065870c0837b48e98d3070d160c5b072bb37f122bcc05431112cad1"} Sep 17 00:35:53.117556 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.117199 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08","Type":"ContainerDied","Data":"ac4f18589b2f2ec31ed17966b06b6752bd0bd608237566076ba7360486685430"} Sep 17 00:35:53.346482 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.346458 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:53.508918 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.508842 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-main-tls\") pod \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " Sep 17 00:35:53.508918 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.508879 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-tls-assets\") pod \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " Sep 17 00:35:53.508918 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.508914 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-kube-rbac-proxy-web\") pod \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " Sep 17 00:35:53.509192 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.508933 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-alertmanager-trusted-ca-bundle\") pod \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " Sep 17 00:35:53.509192 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.508980 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " Sep 17 00:35:53.509192 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.509013 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-config-volume\") pod \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " Sep 17 00:35:53.509192 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.509030 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-cluster-tls-config\") pod \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " Sep 17 00:35:53.509192 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.509059 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-config-out\") pod \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " Sep 17 00:35:53.509541 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.509402 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" (UID: "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 00:35:53.509692 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.509619 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-metrics-client-ca\") pod \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " Sep 17 00:35:53.509692 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.509657 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9f95g\" (UniqueName: \"kubernetes.io/projected/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-kube-api-access-9f95g\") pod \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " Sep 17 00:35:53.509692 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.509695 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-web-config\") pod \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " Sep 17 00:35:53.509692 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.509734 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-alertmanager-main-db\") pod \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " Sep 17 00:35:53.509968 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.509782 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-kube-rbac-proxy\") pod \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\" (UID: \"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08\") " Sep 17 00:35:53.511149 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.510181 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" (UID: "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 00:35:53.511149 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.510275 2582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:35:53.511149 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.510297 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-metrics-client-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:35:53.511149 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.510616 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" (UID: "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:35:53.511943 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.511903 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-config-out" (OuterVolumeSpecName: "config-out") pod "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" (UID: "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:35:53.511943 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.511906 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" (UID: "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:35:53.512114 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.512037 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-config-volume" (OuterVolumeSpecName: "config-volume") pod "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" (UID: "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:35:53.512407 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.512256 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" (UID: "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:35:53.513133 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.513095 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" (UID: "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:35:53.513226 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.513162 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" (UID: "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:35:53.513469 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.513447 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-kube-api-access-9f95g" (OuterVolumeSpecName: "kube-api-access-9f95g") pod "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" (UID: "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08"). InnerVolumeSpecName "kube-api-access-9f95g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:35:53.514264 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.514230 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" (UID: "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:35:53.516887 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.516863 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" (UID: "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:35:53.524212 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.524190 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-web-config" (OuterVolumeSpecName: "web-config") pod "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" (UID: "9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:35:53.611268 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.611245 2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-main-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:35:53.611268 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.611266 2582 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-tls-assets\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:35:53.611380 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.611276 2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:35:53.611380 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.611286 2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:35:53.611380 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.611294 2582 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-config-volume\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:35:53.611380 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.611302 2582 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-cluster-tls-config\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:35:53.611380 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.611311 2582 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-config-out\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:35:53.611380 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.611320 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f95g\" (UniqueName: \"kubernetes.io/projected/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-kube-api-access-9f95g\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:35:53.611380 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.611336 2582 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-web-config\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:35:53.611380 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.611344 2582 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-alertmanager-main-db\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:35:53.611380 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:53.611352 2582 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:35:54.124457 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.124426 2582 generic.go:356] "Generic (PLEG): container finished" podID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerID="7fd1a84d5c22ee6dacb5d3b7456f5f7e0d5a7284e9aa0b2c66c5c917582be7e8" exitCode=0 Sep 17 00:35:54.124457 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.124451 2582 generic.go:356] "Generic (PLEG): container finished" podID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerID="10b11d264c42a249fb14fa2b6eb1b2ffbc428e64641b09805699b6a274603e98" exitCode=0 Sep 17 00:35:54.124904 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.124528 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08","Type":"ContainerDied","Data":"7fd1a84d5c22ee6dacb5d3b7456f5f7e0d5a7284e9aa0b2c66c5c917582be7e8"} Sep 17 00:35:54.124904 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.124590 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.124904 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.124601 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08","Type":"ContainerDied","Data":"10b11d264c42a249fb14fa2b6eb1b2ffbc428e64641b09805699b6a274603e98"} Sep 17 00:35:54.124904 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.124628 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08","Type":"ContainerDied","Data":"fcd0165cf5e911ed7a7501220cffdba44fe9bfa3ea7e087ffbde2fecba9dfa7e"} Sep 17 00:35:54.124904 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.124659 2582 scope.go:117] "RemoveContainer" containerID="2f445d1cc6a69f65df85b5c76cf1d45c90f880cf8e82294600c29c72bedef2cc" Sep 17 00:35:54.132389 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.132374 2582 scope.go:117] "RemoveContainer" containerID="7fd1a84d5c22ee6dacb5d3b7456f5f7e0d5a7284e9aa0b2c66c5c917582be7e8" Sep 17 00:35:54.139552 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.139511 2582 scope.go:117] "RemoveContainer" containerID="a0bd09e8dbe0cc6bafcf9926231a9a798c62e50d643b0e4d78db9e9c7aaf75e5" Sep 17 00:35:54.142290 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.142268 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 00:35:54.145843 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.145793 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 00:35:54.149373 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.149356 2582 scope.go:117] "RemoveContainer" containerID="10b11d264c42a249fb14fa2b6eb1b2ffbc428e64641b09805699b6a274603e98" Sep 17 00:35:54.156115 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.156100 2582 scope.go:117] "RemoveContainer" containerID="785332a8f065870c0837b48e98d3070d160c5b072bb37f122bcc05431112cad1" Sep 17 00:35:54.163550 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.163535 2582 scope.go:117] "RemoveContainer" containerID="ac4f18589b2f2ec31ed17966b06b6752bd0bd608237566076ba7360486685430" Sep 17 00:35:54.168184 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168166 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 00:35:54.168468 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168457 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="kube-rbac-proxy-metric" Sep 17 00:35:54.168529 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168470 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="kube-rbac-proxy-metric" Sep 17 00:35:54.168529 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168483 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="config-reloader" Sep 17 00:35:54.168529 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168489 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="config-reloader" Sep 17 00:35:54.168529 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168512 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="kube-rbac-proxy" Sep 17 00:35:54.168529 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168519 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="kube-rbac-proxy" Sep 17 00:35:54.168529 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168530 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="alertmanager" Sep 17 00:35:54.168705 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168536 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="alertmanager" Sep 17 00:35:54.168705 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168542 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="kube-rbac-proxy-web" Sep 17 00:35:54.168705 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168547 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="kube-rbac-proxy-web" Sep 17 00:35:54.168705 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168555 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="prom-label-proxy" Sep 17 00:35:54.168705 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168561 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="prom-label-proxy" Sep 17 00:35:54.168705 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168569 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="init-config-reloader" Sep 17 00:35:54.168705 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168574 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="init-config-reloader" Sep 17 00:35:54.168705 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168620 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="config-reloader" Sep 17 00:35:54.168705 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168627 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="prom-label-proxy" Sep 17 00:35:54.168705 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168633 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="kube-rbac-proxy-metric" Sep 17 00:35:54.168705 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168638 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="kube-rbac-proxy-web" Sep 17 00:35:54.168705 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168644 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="kube-rbac-proxy" Sep 17 00:35:54.168705 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.168651 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" containerName="alertmanager" Sep 17 00:35:54.171243 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.171228 2582 scope.go:117] "RemoveContainer" containerID="46c16b4e926bb0f3629802e3eaddf5b0319c665ced861f50682abbff3ea43cfb" Sep 17 00:35:54.173924 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.173908 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.175748 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.175728 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 17 00:35:54.175980 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.175963 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 17 00:35:54.176061 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.176009 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 17 00:35:54.176061 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.176047 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 17 00:35:54.176234 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.176217 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 17 00:35:54.176302 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.176232 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 17 00:35:54.176451 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.176435 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 17 00:35:54.176555 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.176447 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 17 00:35:54.176555 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.176447 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cw7rj\"" Sep 17 00:35:54.179281 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.179264 2582 scope.go:117] "RemoveContainer" containerID="2f445d1cc6a69f65df85b5c76cf1d45c90f880cf8e82294600c29c72bedef2cc" Sep 17 00:35:54.179604 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:35:54.179570 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f445d1cc6a69f65df85b5c76cf1d45c90f880cf8e82294600c29c72bedef2cc\": container with ID starting with 2f445d1cc6a69f65df85b5c76cf1d45c90f880cf8e82294600c29c72bedef2cc not found: ID does not exist" containerID="2f445d1cc6a69f65df85b5c76cf1d45c90f880cf8e82294600c29c72bedef2cc" Sep 17 00:35:54.179715 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.179608 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f445d1cc6a69f65df85b5c76cf1d45c90f880cf8e82294600c29c72bedef2cc"} err="failed to get container status \"2f445d1cc6a69f65df85b5c76cf1d45c90f880cf8e82294600c29c72bedef2cc\": rpc error: code = NotFound desc = could not find container \"2f445d1cc6a69f65df85b5c76cf1d45c90f880cf8e82294600c29c72bedef2cc\": container with ID starting with 2f445d1cc6a69f65df85b5c76cf1d45c90f880cf8e82294600c29c72bedef2cc not found: ID does not exist" Sep 17 00:35:54.179715 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.179645 2582 scope.go:117] "RemoveContainer" containerID="7fd1a84d5c22ee6dacb5d3b7456f5f7e0d5a7284e9aa0b2c66c5c917582be7e8" Sep 17 00:35:54.179937 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:35:54.179919 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd1a84d5c22ee6dacb5d3b7456f5f7e0d5a7284e9aa0b2c66c5c917582be7e8\": container with ID starting with 7fd1a84d5c22ee6dacb5d3b7456f5f7e0d5a7284e9aa0b2c66c5c917582be7e8 not found: ID does not exist" containerID="7fd1a84d5c22ee6dacb5d3b7456f5f7e0d5a7284e9aa0b2c66c5c917582be7e8" Sep 17 00:35:54.180001 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.179942 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd1a84d5c22ee6dacb5d3b7456f5f7e0d5a7284e9aa0b2c66c5c917582be7e8"} err="failed to get container status \"7fd1a84d5c22ee6dacb5d3b7456f5f7e0d5a7284e9aa0b2c66c5c917582be7e8\": rpc error: code = NotFound desc = could not find container \"7fd1a84d5c22ee6dacb5d3b7456f5f7e0d5a7284e9aa0b2c66c5c917582be7e8\": container with ID starting with 7fd1a84d5c22ee6dacb5d3b7456f5f7e0d5a7284e9aa0b2c66c5c917582be7e8 not found: ID does not exist" Sep 17 00:35:54.180001 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.179964 2582 scope.go:117] "RemoveContainer" containerID="a0bd09e8dbe0cc6bafcf9926231a9a798c62e50d643b0e4d78db9e9c7aaf75e5" Sep 17 00:35:54.180246 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:35:54.180222 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0bd09e8dbe0cc6bafcf9926231a9a798c62e50d643b0e4d78db9e9c7aaf75e5\": container with ID starting with a0bd09e8dbe0cc6bafcf9926231a9a798c62e50d643b0e4d78db9e9c7aaf75e5 not found: ID does not exist" containerID="a0bd09e8dbe0cc6bafcf9926231a9a798c62e50d643b0e4d78db9e9c7aaf75e5" Sep 17 00:35:54.180317 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.180255 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0bd09e8dbe0cc6bafcf9926231a9a798c62e50d643b0e4d78db9e9c7aaf75e5"} err="failed to get container status \"a0bd09e8dbe0cc6bafcf9926231a9a798c62e50d643b0e4d78db9e9c7aaf75e5\": rpc error: code = NotFound desc = could not find container \"a0bd09e8dbe0cc6bafcf9926231a9a798c62e50d643b0e4d78db9e9c7aaf75e5\": container with ID starting with a0bd09e8dbe0cc6bafcf9926231a9a798c62e50d643b0e4d78db9e9c7aaf75e5 not found: ID does not exist" Sep 17 00:35:54.180317 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.180278 2582 scope.go:117] "RemoveContainer" containerID="10b11d264c42a249fb14fa2b6eb1b2ffbc428e64641b09805699b6a274603e98" Sep 17 00:35:54.180468 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.180434 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 17 00:35:54.180578 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:35:54.180560 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10b11d264c42a249fb14fa2b6eb1b2ffbc428e64641b09805699b6a274603e98\": container with ID starting with 10b11d264c42a249fb14fa2b6eb1b2ffbc428e64641b09805699b6a274603e98 not found: ID does not exist" containerID="10b11d264c42a249fb14fa2b6eb1b2ffbc428e64641b09805699b6a274603e98" Sep 17 00:35:54.180639 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.180580 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b11d264c42a249fb14fa2b6eb1b2ffbc428e64641b09805699b6a274603e98"} err="failed to get container status \"10b11d264c42a249fb14fa2b6eb1b2ffbc428e64641b09805699b6a274603e98\": rpc error: code = NotFound desc = could not find container \"10b11d264c42a249fb14fa2b6eb1b2ffbc428e64641b09805699b6a274603e98\": container with ID starting with 10b11d264c42a249fb14fa2b6eb1b2ffbc428e64641b09805699b6a274603e98 not found: ID does not exist" Sep 17 00:35:54.180639 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.180593 2582 scope.go:117] "RemoveContainer" containerID="785332a8f065870c0837b48e98d3070d160c5b072bb37f122bcc05431112cad1" Sep 17 00:35:54.180886 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:35:54.180868 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"785332a8f065870c0837b48e98d3070d160c5b072bb37f122bcc05431112cad1\": container with ID starting with 785332a8f065870c0837b48e98d3070d160c5b072bb37f122bcc05431112cad1 not found: ID does not exist" containerID="785332a8f065870c0837b48e98d3070d160c5b072bb37f122bcc05431112cad1" Sep 17 00:35:54.180922 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.180890 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"785332a8f065870c0837b48e98d3070d160c5b072bb37f122bcc05431112cad1"} err="failed to get container status \"785332a8f065870c0837b48e98d3070d160c5b072bb37f122bcc05431112cad1\": rpc error: code = NotFound desc = could not find container \"785332a8f065870c0837b48e98d3070d160c5b072bb37f122bcc05431112cad1\": container with ID starting with 785332a8f065870c0837b48e98d3070d160c5b072bb37f122bcc05431112cad1 not found: ID does not exist" Sep 17 00:35:54.180922 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.180902 2582 scope.go:117] "RemoveContainer" containerID="ac4f18589b2f2ec31ed17966b06b6752bd0bd608237566076ba7360486685430" Sep 17 00:35:54.181120 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:35:54.181107 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac4f18589b2f2ec31ed17966b06b6752bd0bd608237566076ba7360486685430\": container with ID starting with ac4f18589b2f2ec31ed17966b06b6752bd0bd608237566076ba7360486685430 not found: ID does not exist" containerID="ac4f18589b2f2ec31ed17966b06b6752bd0bd608237566076ba7360486685430" Sep 17 00:35:54.181159 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.181124 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4f18589b2f2ec31ed17966b06b6752bd0bd608237566076ba7360486685430"} err="failed to get container status \"ac4f18589b2f2ec31ed17966b06b6752bd0bd608237566076ba7360486685430\": rpc error: code = NotFound desc = could not find container \"ac4f18589b2f2ec31ed17966b06b6752bd0bd608237566076ba7360486685430\": container with ID starting with ac4f18589b2f2ec31ed17966b06b6752bd0bd608237566076ba7360486685430 not found: ID does not exist" Sep 17 00:35:54.181159 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.181134 2582 scope.go:117] "RemoveContainer" containerID="46c16b4e926bb0f3629802e3eaddf5b0319c665ced861f50682abbff3ea43cfb" Sep 17 00:35:54.181358 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:35:54.181343 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46c16b4e926bb0f3629802e3eaddf5b0319c665ced861f50682abbff3ea43cfb\": container with ID starting with 46c16b4e926bb0f3629802e3eaddf5b0319c665ced861f50682abbff3ea43cfb not found: ID does not exist" containerID="46c16b4e926bb0f3629802e3eaddf5b0319c665ced861f50682abbff3ea43cfb" Sep 17 00:35:54.181404 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.181363 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46c16b4e926bb0f3629802e3eaddf5b0319c665ced861f50682abbff3ea43cfb"} err="failed to get container status \"46c16b4e926bb0f3629802e3eaddf5b0319c665ced861f50682abbff3ea43cfb\": rpc error: code = NotFound desc = could not find container \"46c16b4e926bb0f3629802e3eaddf5b0319c665ced861f50682abbff3ea43cfb\": container with ID starting with 46c16b4e926bb0f3629802e3eaddf5b0319c665ced861f50682abbff3ea43cfb not found: ID does not exist" Sep 17 00:35:54.181404 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.181375 2582 scope.go:117] "RemoveContainer" containerID="2f445d1cc6a69f65df85b5c76cf1d45c90f880cf8e82294600c29c72bedef2cc" Sep 17 00:35:54.181643 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.181620 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f445d1cc6a69f65df85b5c76cf1d45c90f880cf8e82294600c29c72bedef2cc"} err="failed to get container status \"2f445d1cc6a69f65df85b5c76cf1d45c90f880cf8e82294600c29c72bedef2cc\": rpc error: code = NotFound desc = could not find container \"2f445d1cc6a69f65df85b5c76cf1d45c90f880cf8e82294600c29c72bedef2cc\": container with ID starting with 2f445d1cc6a69f65df85b5c76cf1d45c90f880cf8e82294600c29c72bedef2cc not found: ID does not exist" Sep 17 00:35:54.181702 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.181645 2582 scope.go:117] "RemoveContainer" containerID="7fd1a84d5c22ee6dacb5d3b7456f5f7e0d5a7284e9aa0b2c66c5c917582be7e8" Sep 17 00:35:54.181861 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.181844 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd1a84d5c22ee6dacb5d3b7456f5f7e0d5a7284e9aa0b2c66c5c917582be7e8"} err="failed to get container status \"7fd1a84d5c22ee6dacb5d3b7456f5f7e0d5a7284e9aa0b2c66c5c917582be7e8\": rpc error: code = NotFound desc = could not find container \"7fd1a84d5c22ee6dacb5d3b7456f5f7e0d5a7284e9aa0b2c66c5c917582be7e8\": container with ID starting with 7fd1a84d5c22ee6dacb5d3b7456f5f7e0d5a7284e9aa0b2c66c5c917582be7e8 not found: ID does not exist" Sep 17 00:35:54.181861 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.181860 2582 scope.go:117] "RemoveContainer" containerID="a0bd09e8dbe0cc6bafcf9926231a9a798c62e50d643b0e4d78db9e9c7aaf75e5" Sep 17 00:35:54.182075 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.182048 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0bd09e8dbe0cc6bafcf9926231a9a798c62e50d643b0e4d78db9e9c7aaf75e5"} err="failed to get container status \"a0bd09e8dbe0cc6bafcf9926231a9a798c62e50d643b0e4d78db9e9c7aaf75e5\": rpc error: code = NotFound desc = could not find container \"a0bd09e8dbe0cc6bafcf9926231a9a798c62e50d643b0e4d78db9e9c7aaf75e5\": container with ID starting with a0bd09e8dbe0cc6bafcf9926231a9a798c62e50d643b0e4d78db9e9c7aaf75e5 not found: ID does not exist" Sep 17 00:35:54.182163 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.182076 2582 scope.go:117] "RemoveContainer" containerID="10b11d264c42a249fb14fa2b6eb1b2ffbc428e64641b09805699b6a274603e98" Sep 17 00:35:54.182426 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.182389 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10b11d264c42a249fb14fa2b6eb1b2ffbc428e64641b09805699b6a274603e98"} err="failed to get container status \"10b11d264c42a249fb14fa2b6eb1b2ffbc428e64641b09805699b6a274603e98\": rpc error: code = NotFound desc = could not find container \"10b11d264c42a249fb14fa2b6eb1b2ffbc428e64641b09805699b6a274603e98\": container with ID starting with 10b11d264c42a249fb14fa2b6eb1b2ffbc428e64641b09805699b6a274603e98 not found: ID does not exist" Sep 17 00:35:54.182426 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.182416 2582 scope.go:117] "RemoveContainer" containerID="785332a8f065870c0837b48e98d3070d160c5b072bb37f122bcc05431112cad1" Sep 17 00:35:54.183338 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.183316 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"785332a8f065870c0837b48e98d3070d160c5b072bb37f122bcc05431112cad1"} err="failed to get container status \"785332a8f065870c0837b48e98d3070d160c5b072bb37f122bcc05431112cad1\": rpc error: code = NotFound desc = could not find container \"785332a8f065870c0837b48e98d3070d160c5b072bb37f122bcc05431112cad1\": container with ID starting with 785332a8f065870c0837b48e98d3070d160c5b072bb37f122bcc05431112cad1 not found: ID does not exist" Sep 17 00:35:54.183338 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.183337 2582 scope.go:117] "RemoveContainer" containerID="ac4f18589b2f2ec31ed17966b06b6752bd0bd608237566076ba7360486685430" Sep 17 00:35:54.183894 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.183626 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4f18589b2f2ec31ed17966b06b6752bd0bd608237566076ba7360486685430"} err="failed to get container status \"ac4f18589b2f2ec31ed17966b06b6752bd0bd608237566076ba7360486685430\": rpc error: code = NotFound desc = could not find container \"ac4f18589b2f2ec31ed17966b06b6752bd0bd608237566076ba7360486685430\": container with ID starting with ac4f18589b2f2ec31ed17966b06b6752bd0bd608237566076ba7360486685430 not found: ID does not exist" Sep 17 00:35:54.183894 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.183648 2582 scope.go:117] "RemoveContainer" containerID="46c16b4e926bb0f3629802e3eaddf5b0319c665ced861f50682abbff3ea43cfb" Sep 17 00:35:54.184136 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.183997 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46c16b4e926bb0f3629802e3eaddf5b0319c665ced861f50682abbff3ea43cfb"} err="failed to get container status \"46c16b4e926bb0f3629802e3eaddf5b0319c665ced861f50682abbff3ea43cfb\": rpc error: code = NotFound desc = could not find container \"46c16b4e926bb0f3629802e3eaddf5b0319c665ced861f50682abbff3ea43cfb\": container with ID starting with 46c16b4e926bb0f3629802e3eaddf5b0319c665ced861f50682abbff3ea43cfb not found: ID does not exist" Sep 17 00:35:54.193831 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.193807 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 00:35:54.315598 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.315574 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3ad5398-9310-4305-89e7-fd2edb38f0a0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.315734 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.315603 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvl7d\" (UniqueName: \"kubernetes.io/projected/a3ad5398-9310-4305-89e7-fd2edb38f0a0-kube-api-access-cvl7d\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.315734 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.315637 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3ad5398-9310-4305-89e7-fd2edb38f0a0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.315734 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.315671 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-config-volume\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.315734 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.315723 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.315905 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.315745 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-web-config\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.315905 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.315764 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3ad5398-9310-4305-89e7-fd2edb38f0a0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.315905 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.315776 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3ad5398-9310-4305-89e7-fd2edb38f0a0-config-out\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.315905 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.315796 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.315905 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.315860 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.316046 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.315913 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.316046 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.315946 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a3ad5398-9310-4305-89e7-fd2edb38f0a0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.316046 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.315982 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.416632 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.416569 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3ad5398-9310-4305-89e7-fd2edb38f0a0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.416632 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.416600 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvl7d\" (UniqueName: \"kubernetes.io/projected/a3ad5398-9310-4305-89e7-fd2edb38f0a0-kube-api-access-cvl7d\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.416632 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.416631 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3ad5398-9310-4305-89e7-fd2edb38f0a0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.416842 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.416657 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-config-volume\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.416842 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.416688 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.416842 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.416713 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-web-config\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.416842 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.416742 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3ad5398-9310-4305-89e7-fd2edb38f0a0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.416842 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.416762 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3ad5398-9310-4305-89e7-fd2edb38f0a0-config-out\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.416842 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.416786 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.417752 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.417725 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3ad5398-9310-4305-89e7-fd2edb38f0a0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.417863 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.417788 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.417863 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.417829 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.417984 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.417870 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a3ad5398-9310-4305-89e7-fd2edb38f0a0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.417984 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.417896 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.418261 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.418114 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3ad5398-9310-4305-89e7-fd2edb38f0a0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.420197 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.420056 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-web-config\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.420197 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.420113 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a3ad5398-9310-4305-89e7-fd2edb38f0a0-config-out\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.420197 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.420166 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-config-volume\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.420402 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.420383 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a3ad5398-9310-4305-89e7-fd2edb38f0a0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.420534 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.420489 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.420696 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.420677 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a3ad5398-9310-4305-89e7-fd2edb38f0a0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.420875 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.420859 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.420910 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.420861 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.420985 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.420962 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.422236 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.422213 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a3ad5398-9310-4305-89e7-fd2edb38f0a0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.424048 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.424030 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvl7d\" (UniqueName: \"kubernetes.io/projected/a3ad5398-9310-4305-89e7-fd2edb38f0a0-kube-api-access-cvl7d\") pod \"alertmanager-main-0\" (UID: \"a3ad5398-9310-4305-89e7-fd2edb38f0a0\") " pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.495270 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.495247 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 17 00:35:54.771139 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:54.771116 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 17 00:35:54.773287 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:35:54.773263 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ad5398_9310_4305_89e7_fd2edb38f0a0.slice/crio-4ab743be3a185f4582df38fac2e741062e4094e5d1b24541729ee2fbb92c0bb3 WatchSource:0}: Error finding container 4ab743be3a185f4582df38fac2e741062e4094e5d1b24541729ee2fbb92c0bb3: Status 404 returned error can't find the container with id 4ab743be3a185f4582df38fac2e741062e4094e5d1b24541729ee2fbb92c0bb3 Sep 17 00:35:55.131033 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:55.130998 2582 generic.go:356] "Generic (PLEG): container finished" podID="a3ad5398-9310-4305-89e7-fd2edb38f0a0" containerID="7f58c4cf0cfd1257b593b20f487c1720ef3d6fa7a0fe4cd104180c089b6540d4" exitCode=0 Sep 17 00:35:55.131437 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:55.131086 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3ad5398-9310-4305-89e7-fd2edb38f0a0","Type":"ContainerDied","Data":"7f58c4cf0cfd1257b593b20f487c1720ef3d6fa7a0fe4cd104180c089b6540d4"} Sep 17 00:35:55.131437 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:55.131161 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3ad5398-9310-4305-89e7-fd2edb38f0a0","Type":"ContainerStarted","Data":"4ab743be3a185f4582df38fac2e741062e4094e5d1b24541729ee2fbb92c0bb3"} Sep 17 00:35:55.649977 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:55.649886 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08" path="/var/lib/kubelet/pods/9fcb49c8-bd4b-498e-b6d1-6ed6b0823a08/volumes" Sep 17 00:35:56.098137 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.098105 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5554f8dfff-27h4m"] Sep 17 00:35:56.106547 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.106490 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.108537 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.108493 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 17 00:35:56.109003 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.108976 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 17 00:35:56.109109 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.109081 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-sdfsf\"" Sep 17 00:35:56.109530 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.109340 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 17 00:35:56.109737 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.109662 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 17 00:35:56.110278 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.110258 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 17 00:35:56.113945 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.113570 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 17 00:35:56.115423 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.115398 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5554f8dfff-27h4m"] Sep 17 00:35:56.133459 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.133434 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bfac6f7-c098-40bd-9d2d-8654c39e178c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.133897 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.133545 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bfac6f7-c098-40bd-9d2d-8654c39e178c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.133897 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.133582 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9bfac6f7-c098-40bd-9d2d-8654c39e178c-secret-telemeter-client\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.133897 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.133627 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bfac6f7-c098-40bd-9d2d-8654c39e178c-metrics-client-ca\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.133897 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.133654 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9bfac6f7-c098-40bd-9d2d-8654c39e178c-telemeter-client-tls\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.133897 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.133712 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn6jb\" (UniqueName: \"kubernetes.io/projected/9bfac6f7-c098-40bd-9d2d-8654c39e178c-kube-api-access-wn6jb\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.133897 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.133746 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bfac6f7-c098-40bd-9d2d-8654c39e178c-serving-certs-ca-bundle\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.133897 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.133762 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9bfac6f7-c098-40bd-9d2d-8654c39e178c-federate-client-tls\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.142884 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.142851 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3ad5398-9310-4305-89e7-fd2edb38f0a0","Type":"ContainerStarted","Data":"af70a976805e45f09fbbf94ffcb91beecccb4959953f4cb2812e2c72f87ab5e3"} Sep 17 00:35:56.142994 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.142929 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3ad5398-9310-4305-89e7-fd2edb38f0a0","Type":"ContainerStarted","Data":"ab3492fbb11d11c8b8dc0eed121cacd48c236969d1d554282ad1f1bc6bfeb155"} Sep 17 00:35:56.142994 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.142973 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3ad5398-9310-4305-89e7-fd2edb38f0a0","Type":"ContainerStarted","Data":"281e6549fb8e91470a1e6576b233804761b15c980d158ba363f362080454eb96"} Sep 17 00:35:56.143062 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.143014 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3ad5398-9310-4305-89e7-fd2edb38f0a0","Type":"ContainerStarted","Data":"a6cf699f217f05483678866a6f33b1e0d9409815481cd4fbbdf7d37b85547889"} Sep 17 00:35:56.143062 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.143048 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3ad5398-9310-4305-89e7-fd2edb38f0a0","Type":"ContainerStarted","Data":"31ab66df353d55633257bb10f827eb3ad564bb218401bd5a9ae7e3830ff6a275"} Sep 17 00:35:56.143123 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.143083 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a3ad5398-9310-4305-89e7-fd2edb38f0a0","Type":"ContainerStarted","Data":"eadfcd65beae4db291daac746c0cc673d24282c2b364a2e327cdc8a8438b7187"} Sep 17 00:35:56.171790 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.171743 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.154345303 podStartE2EDuration="2.171728766s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.383463ms" isStatefulPod=false podCreationTimestamp="2026-09-17 00:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:35:56.169749415 +0000 UTC m=+369.100382902" watchObservedRunningTime="2026-09-17 00:35:56.171728766 +0000 UTC m=+369.102362253" Sep 17 00:35:56.235086 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.235042 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bfac6f7-c098-40bd-9d2d-8654c39e178c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.235270 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.235132 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9bfac6f7-c098-40bd-9d2d-8654c39e178c-secret-telemeter-client\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.235270 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.235207 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bfac6f7-c098-40bd-9d2d-8654c39e178c-metrics-client-ca\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.235270 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.235237 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9bfac6f7-c098-40bd-9d2d-8654c39e178c-telemeter-client-tls\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.235436 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.235272 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn6jb\" (UniqueName: \"kubernetes.io/projected/9bfac6f7-c098-40bd-9d2d-8654c39e178c-kube-api-access-wn6jb\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.235436 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.235390 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bfac6f7-c098-40bd-9d2d-8654c39e178c-serving-certs-ca-bundle\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.235436 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.235427 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9bfac6f7-c098-40bd-9d2d-8654c39e178c-federate-client-tls\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.235611 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.235539 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bfac6f7-c098-40bd-9d2d-8654c39e178c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.243695 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.243654 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bfac6f7-c098-40bd-9d2d-8654c39e178c-serving-certs-ca-bundle\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.244533 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.244190 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bfac6f7-c098-40bd-9d2d-8654c39e178c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.244533 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.244227 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bfac6f7-c098-40bd-9d2d-8654c39e178c-metrics-client-ca\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.249248 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.247359 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9bfac6f7-c098-40bd-9d2d-8654c39e178c-secret-telemeter-client\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.250250 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.250221 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9bfac6f7-c098-40bd-9d2d-8654c39e178c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.250553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.250533 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9bfac6f7-c098-40bd-9d2d-8654c39e178c-federate-client-tls\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.251346 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.251320 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn6jb\" (UniqueName: \"kubernetes.io/projected/9bfac6f7-c098-40bd-9d2d-8654c39e178c-kube-api-access-wn6jb\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.251483 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.251469 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9bfac6f7-c098-40bd-9d2d-8654c39e178c-telemeter-client-tls\") pod \"telemeter-client-5554f8dfff-27h4m\" (UID: \"9bfac6f7-c098-40bd-9d2d-8654c39e178c\") " pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.419895 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.419815 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" Sep 17 00:35:56.706630 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.706542 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5554f8dfff-27h4m"] Sep 17 00:35:56.712641 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:35:56.712611 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bfac6f7_c098_40bd_9d2d_8654c39e178c.slice/crio-43ed122650814efa06b982d886fe51bae3d3b42ba78fff865080a748cbf3c98e WatchSource:0}: Error finding container 43ed122650814efa06b982d886fe51bae3d3b42ba78fff865080a748cbf3c98e: Status 404 returned error can't find the container with id 43ed122650814efa06b982d886fe51bae3d3b42ba78fff865080a748cbf3c98e Sep 17 00:35:56.714714 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:56.714693 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 00:35:57.148791 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:57.148753 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" event={"ID":"9bfac6f7-c098-40bd-9d2d-8654c39e178c","Type":"ContainerStarted","Data":"43ed122650814efa06b982d886fe51bae3d3b42ba78fff865080a748cbf3c98e"} Sep 17 00:35:59.159383 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:59.159348 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" event={"ID":"9bfac6f7-c098-40bd-9d2d-8654c39e178c","Type":"ContainerStarted","Data":"8b25435269d74a0ddfbc0f5f313e25027aca9a2c7b23453145a4145d9f1caaf0"} Sep 17 00:35:59.159860 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:59.159409 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" event={"ID":"9bfac6f7-c098-40bd-9d2d-8654c39e178c","Type":"ContainerStarted","Data":"de05816f6546be1276e05ee4c7dfe34abb52db5fafa003e5d165d05435386743"} Sep 17 00:35:59.159860 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:59.159434 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" event={"ID":"9bfac6f7-c098-40bd-9d2d-8654c39e178c","Type":"ContainerStarted","Data":"fe8cc08ce2a1805ffb2eeaff81c3b4df58038916cdfd1340a22e547d3cfa5577"} Sep 17 00:35:59.179766 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:35:59.179713 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5554f8dfff-27h4m" podStartSLOduration=1.2573431529999999 podStartE2EDuration="3.179696492s" totalImagesPullingTime="1.922353339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:35:59.178152298 +0000 UTC m=+372.108785786" watchObservedRunningTime="2026-09-17 00:35:59.179696492 +0000 UTC m=+372.110329973" Sep 17 00:36:00.474748 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:36:00.474695 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs\") pod \"network-metrics-daemon-rh4cz\" (UID: \"6155b312-55da-43b1-9b1f-4d51dfbb0d42\") " pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:36:00.477288 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:36:00.477224 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6155b312-55da-43b1-9b1f-4d51dfbb0d42-metrics-certs\") pod \"network-metrics-daemon-rh4cz\" (UID: \"6155b312-55da-43b1-9b1f-4d51dfbb0d42\") " pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:36:00.646879 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:36:00.646847 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-plv5t\"" Sep 17 00:36:00.655393 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:36:00.655364 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rh4cz" Sep 17 00:36:00.924278 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:36:00.924246 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rh4cz"] Sep 17 00:36:00.927360 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:36:00.927332 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6155b312_55da_43b1_9b1f_4d51dfbb0d42.slice/crio-ba027c7fcf8735371b7a9975218f0ca89d3211a77f67386bc0a2b33dbb5c1fe4 WatchSource:0}: Error finding container ba027c7fcf8735371b7a9975218f0ca89d3211a77f67386bc0a2b33dbb5c1fe4: Status 404 returned error can't find the container with id ba027c7fcf8735371b7a9975218f0ca89d3211a77f67386bc0a2b33dbb5c1fe4 Sep 17 00:36:01.168907 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:36:01.168868 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rh4cz" event={"ID":"6155b312-55da-43b1-9b1f-4d51dfbb0d42","Type":"ContainerStarted","Data":"ba027c7fcf8735371b7a9975218f0ca89d3211a77f67386bc0a2b33dbb5c1fe4"} Sep 17 00:36:03.180807 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:36:03.180773 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rh4cz" event={"ID":"6155b312-55da-43b1-9b1f-4d51dfbb0d42","Type":"ContainerStarted","Data":"7783ff7178409059f2a4a353efb0e0dedcb4db656a3d0fb4fc76a7118743efc9"} Sep 17 00:36:03.181223 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:36:03.180823 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rh4cz" event={"ID":"6155b312-55da-43b1-9b1f-4d51dfbb0d42","Type":"ContainerStarted","Data":"42ecc2d5d87d778592856b10d7b8ca27d1d296dc5a14a0c27eb276f46a10cc54"} Sep 17 00:36:03.195661 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:36:03.195610 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rh4cz" podStartSLOduration=373.988620282 podStartE2EDuration="6m15.195596034s" totalImagesPullingTime="1.206975752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:36:03.193542315 +0000 UTC m=+376.124175799" watchObservedRunningTime="2026-09-17 00:36:03.195596034 +0000 UTC m=+376.126229570" Sep 17 00:39:10.577971 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:10.577896 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j6vxm"] Sep 17 00:39:10.581040 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:10.581022 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j6vxm" Sep 17 00:39:10.583563 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:10.583540 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 17 00:39:10.583986 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:10.583607 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 17 00:39:10.589247 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:10.583636 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-jqqmw\"" Sep 17 00:39:10.592749 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:10.592714 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j6vxm"] Sep 17 00:39:10.742805 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:10.742768 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghbbs\" (UniqueName: \"kubernetes.io/projected/b210fd21-25a3-4006-b9de-d43d9243d936-kube-api-access-ghbbs\") pod \"cert-manager-operator-controller-manager-55c444bf86-j6vxm\" (UID: \"b210fd21-25a3-4006-b9de-d43d9243d936\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j6vxm" Sep 17 00:39:10.742958 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:10.742896 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b210fd21-25a3-4006-b9de-d43d9243d936-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-j6vxm\" (UID: \"b210fd21-25a3-4006-b9de-d43d9243d936\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j6vxm" Sep 17 00:39:10.843841 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:10.843748 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b210fd21-25a3-4006-b9de-d43d9243d936-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-j6vxm\" (UID: \"b210fd21-25a3-4006-b9de-d43d9243d936\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j6vxm" Sep 17 00:39:10.843841 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:10.843818 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghbbs\" (UniqueName: \"kubernetes.io/projected/b210fd21-25a3-4006-b9de-d43d9243d936-kube-api-access-ghbbs\") pod \"cert-manager-operator-controller-manager-55c444bf86-j6vxm\" (UID: \"b210fd21-25a3-4006-b9de-d43d9243d936\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j6vxm" Sep 17 00:39:10.844190 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:10.844163 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b210fd21-25a3-4006-b9de-d43d9243d936-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-j6vxm\" (UID: \"b210fd21-25a3-4006-b9de-d43d9243d936\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j6vxm" Sep 17 00:39:10.852787 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:10.852756 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghbbs\" (UniqueName: \"kubernetes.io/projected/b210fd21-25a3-4006-b9de-d43d9243d936-kube-api-access-ghbbs\") pod \"cert-manager-operator-controller-manager-55c444bf86-j6vxm\" (UID: \"b210fd21-25a3-4006-b9de-d43d9243d936\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j6vxm" Sep 17 00:39:10.897345 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:10.897311 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j6vxm" Sep 17 00:39:11.174374 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:11.174346 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j6vxm"] Sep 17 00:39:11.176956 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:39:11.176925 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb210fd21_25a3_4006_b9de_d43d9243d936.slice/crio-a5a736bc4ac0ca148124799030551ad556b0058631be029dc2eb690f7c58dcaa WatchSource:0}: Error finding container a5a736bc4ac0ca148124799030551ad556b0058631be029dc2eb690f7c58dcaa: Status 404 returned error can't find the container with id a5a736bc4ac0ca148124799030551ad556b0058631be029dc2eb690f7c58dcaa Sep 17 00:39:11.985029 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:11.984999 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j6vxm" event={"ID":"b210fd21-25a3-4006-b9de-d43d9243d936","Type":"ContainerStarted","Data":"a5a736bc4ac0ca148124799030551ad556b0058631be029dc2eb690f7c58dcaa"} Sep 17 00:39:15.001241 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:15.001204 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j6vxm" event={"ID":"b210fd21-25a3-4006-b9de-d43d9243d936","Type":"ContainerStarted","Data":"c3742bc116cf7270dbfb385aa4d58019ff1411085c74e04d63bc111fea50cb26"} Sep 17 00:39:15.021805 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:15.021759 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j6vxm" podStartSLOduration=1.770853882 podStartE2EDuration="5.021742499s" totalImagesPullingTime="3.250888617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:39:15.019602039 +0000 UTC m=+567.950235526" watchObservedRunningTime="2026-09-17 00:39:15.021742499 +0000 UTC m=+567.952375985" Sep 17 00:39:16.403289 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:16.403251 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2nvwk"] Sep 17 00:39:16.406826 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:16.406809 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-2nvwk" Sep 17 00:39:16.408726 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:16.408703 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 17 00:39:16.408838 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:16.408710 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 17 00:39:16.408838 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:16.408753 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-5zzf7\"" Sep 17 00:39:16.414551 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:16.414529 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2nvwk"] Sep 17 00:39:16.594538 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:16.594481 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4932c2fa-c36a-4b48-ad24-e760a69a45a2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2nvwk\" (UID: \"4932c2fa-c36a-4b48-ad24-e760a69a45a2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2nvwk" Sep 17 00:39:16.594700 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:16.594560 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gp79\" (UniqueName: \"kubernetes.io/projected/4932c2fa-c36a-4b48-ad24-e760a69a45a2-kube-api-access-4gp79\") pod \"cert-manager-webhook-6486ffd997-2nvwk\" (UID: \"4932c2fa-c36a-4b48-ad24-e760a69a45a2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2nvwk" Sep 17 00:39:16.695412 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:16.695332 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4932c2fa-c36a-4b48-ad24-e760a69a45a2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2nvwk\" (UID: \"4932c2fa-c36a-4b48-ad24-e760a69a45a2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2nvwk" Sep 17 00:39:16.695412 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:16.695381 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gp79\" (UniqueName: \"kubernetes.io/projected/4932c2fa-c36a-4b48-ad24-e760a69a45a2-kube-api-access-4gp79\") pod \"cert-manager-webhook-6486ffd997-2nvwk\" (UID: \"4932c2fa-c36a-4b48-ad24-e760a69a45a2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2nvwk" Sep 17 00:39:16.702556 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:16.702534 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4932c2fa-c36a-4b48-ad24-e760a69a45a2-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-2nvwk\" (UID: \"4932c2fa-c36a-4b48-ad24-e760a69a45a2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2nvwk" Sep 17 00:39:16.702697 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:16.702680 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gp79\" (UniqueName: \"kubernetes.io/projected/4932c2fa-c36a-4b48-ad24-e760a69a45a2-kube-api-access-4gp79\") pod \"cert-manager-webhook-6486ffd997-2nvwk\" (UID: \"4932c2fa-c36a-4b48-ad24-e760a69a45a2\") " pod="cert-manager/cert-manager-webhook-6486ffd997-2nvwk" Sep 17 00:39:16.729423 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:16.729391 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-2nvwk" Sep 17 00:39:16.997538 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:16.997514 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-2nvwk"] Sep 17 00:39:17.000221 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:39:17.000194 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4932c2fa_c36a_4b48_ad24_e760a69a45a2.slice/crio-30e9f14654199f68c6b58d44ae5fd73621ce760df92c14a773cb253d58c42c9c WatchSource:0}: Error finding container 30e9f14654199f68c6b58d44ae5fd73621ce760df92c14a773cb253d58c42c9c: Status 404 returned error can't find the container with id 30e9f14654199f68c6b58d44ae5fd73621ce760df92c14a773cb253d58c42c9c Sep 17 00:39:17.011060 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:17.011035 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-2nvwk" event={"ID":"4932c2fa-c36a-4b48-ad24-e760a69a45a2","Type":"ContainerStarted","Data":"30e9f14654199f68c6b58d44ae5fd73621ce760df92c14a773cb253d58c42c9c"} Sep 17 00:39:20.028085 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:20.027992 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-2nvwk" event={"ID":"4932c2fa-c36a-4b48-ad24-e760a69a45a2","Type":"ContainerStarted","Data":"ec91300b0aef3f40f560a25e0ae6cdfd6cf190ae70b769dd32e7b4afc0ae3404"} Sep 17 00:39:20.028518 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:20.028113 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-2nvwk" Sep 17 00:39:20.044003 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:20.043962 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-2nvwk" podStartSLOduration=1.335201206 podStartE2EDuration="4.043945161s" totalImagesPullingTime="2.708743955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:39:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:39:20.042515053 +0000 UTC m=+572.973148540" watchObservedRunningTime="2026-09-17 00:39:20.043945161 +0000 UTC m=+572.974578648" Sep 17 00:39:26.035448 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:26.035415 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-2nvwk" Sep 17 00:39:28.074333 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:28.074300 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pxpn9"] Sep 17 00:39:28.077883 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:28.077861 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pxpn9" Sep 17 00:39:28.079612 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:28.079589 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-2s6z2\"" Sep 17 00:39:28.087216 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:28.087195 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pxpn9"] Sep 17 00:39:28.093976 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:28.093951 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ddbefd1-6d61-4708-8676-b14cd31bb3a4-bound-sa-token\") pod \"cert-manager-55b4d954d5-pxpn9\" (UID: \"2ddbefd1-6d61-4708-8676-b14cd31bb3a4\") " pod="cert-manager/cert-manager-55b4d954d5-pxpn9" Sep 17 00:39:28.094092 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:28.094076 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-265nl\" (UniqueName: \"kubernetes.io/projected/2ddbefd1-6d61-4708-8676-b14cd31bb3a4-kube-api-access-265nl\") pod \"cert-manager-55b4d954d5-pxpn9\" (UID: \"2ddbefd1-6d61-4708-8676-b14cd31bb3a4\") " pod="cert-manager/cert-manager-55b4d954d5-pxpn9" Sep 17 00:39:28.195269 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:28.195232 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-265nl\" (UniqueName: \"kubernetes.io/projected/2ddbefd1-6d61-4708-8676-b14cd31bb3a4-kube-api-access-265nl\") pod \"cert-manager-55b4d954d5-pxpn9\" (UID: \"2ddbefd1-6d61-4708-8676-b14cd31bb3a4\") " pod="cert-manager/cert-manager-55b4d954d5-pxpn9" Sep 17 00:39:28.195445 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:28.195279 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ddbefd1-6d61-4708-8676-b14cd31bb3a4-bound-sa-token\") pod \"cert-manager-55b4d954d5-pxpn9\" (UID: \"2ddbefd1-6d61-4708-8676-b14cd31bb3a4\") " pod="cert-manager/cert-manager-55b4d954d5-pxpn9" Sep 17 00:39:28.202890 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:28.202859 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2ddbefd1-6d61-4708-8676-b14cd31bb3a4-bound-sa-token\") pod \"cert-manager-55b4d954d5-pxpn9\" (UID: \"2ddbefd1-6d61-4708-8676-b14cd31bb3a4\") " pod="cert-manager/cert-manager-55b4d954d5-pxpn9" Sep 17 00:39:28.203056 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:28.203038 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-265nl\" (UniqueName: \"kubernetes.io/projected/2ddbefd1-6d61-4708-8676-b14cd31bb3a4-kube-api-access-265nl\") pod \"cert-manager-55b4d954d5-pxpn9\" (UID: \"2ddbefd1-6d61-4708-8676-b14cd31bb3a4\") " pod="cert-manager/cert-manager-55b4d954d5-pxpn9" Sep 17 00:39:28.387312 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:28.387231 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-pxpn9" Sep 17 00:39:28.654714 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:28.654692 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-pxpn9"] Sep 17 00:39:28.657291 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:39:28.657267 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ddbefd1_6d61_4708_8676_b14cd31bb3a4.slice/crio-8ea91b66a2cfafd2c6adb7fef351dd393df3af762b1abdf2659fd42c15091987 WatchSource:0}: Error finding container 8ea91b66a2cfafd2c6adb7fef351dd393df3af762b1abdf2659fd42c15091987: Status 404 returned error can't find the container with id 8ea91b66a2cfafd2c6adb7fef351dd393df3af762b1abdf2659fd42c15091987 Sep 17 00:39:29.075368 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:29.075329 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pxpn9" event={"ID":"2ddbefd1-6d61-4708-8676-b14cd31bb3a4","Type":"ContainerStarted","Data":"fd96ec1bec4e901b15488e203b06b5729d1dfae46f01f6a38957dea54797ff4f"} Sep 17 00:39:29.075368 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:29.075373 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-pxpn9" event={"ID":"2ddbefd1-6d61-4708-8676-b14cd31bb3a4","Type":"ContainerStarted","Data":"8ea91b66a2cfafd2c6adb7fef351dd393df3af762b1abdf2659fd42c15091987"} Sep 17 00:39:29.089339 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:29.089291 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-pxpn9" podStartSLOduration=1.089276474 podStartE2EDuration="1.089276474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:39:29.08763874 +0000 UTC m=+582.018272229" watchObservedRunningTime="2026-09-17 00:39:29.089276474 +0000 UTC m=+582.019909962" Sep 17 00:39:47.601063 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:47.600969 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/1.log" Sep 17 00:39:47.851666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:47.609603 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/1.log" Sep 17 00:39:54.439234 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.439187 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh"] Sep 17 00:39:54.453111 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.452485 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh" Sep 17 00:39:54.453111 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.453038 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh"] Sep 17 00:39:54.455332 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.455296 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 17 00:39:54.455773 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.455750 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 17 00:39:54.455872 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.455758 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 17 00:39:54.455872 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.455832 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-6cghx\"" Sep 17 00:39:54.456082 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.456062 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 17 00:39:54.456296 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.456280 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 17 00:39:54.624719 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.624686 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e11fff9b-5466-4dd4-a845-51b2a07aefe5-manager-config\") pod \"lws-controller-manager-68945d9444-x5wrh\" (UID: \"e11fff9b-5466-4dd4-a845-51b2a07aefe5\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh" Sep 17 00:39:54.624719 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.624725 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e11fff9b-5466-4dd4-a845-51b2a07aefe5-cert\") pod \"lws-controller-manager-68945d9444-x5wrh\" (UID: \"e11fff9b-5466-4dd4-a845-51b2a07aefe5\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh" Sep 17 00:39:54.624949 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.624751 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e11fff9b-5466-4dd4-a845-51b2a07aefe5-metrics-cert\") pod \"lws-controller-manager-68945d9444-x5wrh\" (UID: \"e11fff9b-5466-4dd4-a845-51b2a07aefe5\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh" Sep 17 00:39:54.624949 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.624771 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh6xw\" (UniqueName: \"kubernetes.io/projected/e11fff9b-5466-4dd4-a845-51b2a07aefe5-kube-api-access-zh6xw\") pod \"lws-controller-manager-68945d9444-x5wrh\" (UID: \"e11fff9b-5466-4dd4-a845-51b2a07aefe5\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh" Sep 17 00:39:54.726012 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.725913 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e11fff9b-5466-4dd4-a845-51b2a07aefe5-manager-config\") pod \"lws-controller-manager-68945d9444-x5wrh\" (UID: \"e11fff9b-5466-4dd4-a845-51b2a07aefe5\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh" Sep 17 00:39:54.726012 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.725980 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e11fff9b-5466-4dd4-a845-51b2a07aefe5-cert\") pod \"lws-controller-manager-68945d9444-x5wrh\" (UID: \"e11fff9b-5466-4dd4-a845-51b2a07aefe5\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh" Sep 17 00:39:54.726311 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.726021 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e11fff9b-5466-4dd4-a845-51b2a07aefe5-metrics-cert\") pod \"lws-controller-manager-68945d9444-x5wrh\" (UID: \"e11fff9b-5466-4dd4-a845-51b2a07aefe5\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh" Sep 17 00:39:54.726311 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.726051 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh6xw\" (UniqueName: \"kubernetes.io/projected/e11fff9b-5466-4dd4-a845-51b2a07aefe5-kube-api-access-zh6xw\") pod \"lws-controller-manager-68945d9444-x5wrh\" (UID: \"e11fff9b-5466-4dd4-a845-51b2a07aefe5\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh" Sep 17 00:39:54.727091 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.727060 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e11fff9b-5466-4dd4-a845-51b2a07aefe5-manager-config\") pod \"lws-controller-manager-68945d9444-x5wrh\" (UID: \"e11fff9b-5466-4dd4-a845-51b2a07aefe5\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh" Sep 17 00:39:54.730836 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.730805 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e11fff9b-5466-4dd4-a845-51b2a07aefe5-cert\") pod \"lws-controller-manager-68945d9444-x5wrh\" (UID: \"e11fff9b-5466-4dd4-a845-51b2a07aefe5\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh" Sep 17 00:39:54.732030 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.732008 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e11fff9b-5466-4dd4-a845-51b2a07aefe5-metrics-cert\") pod \"lws-controller-manager-68945d9444-x5wrh\" (UID: \"e11fff9b-5466-4dd4-a845-51b2a07aefe5\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh" Sep 17 00:39:54.733067 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.733047 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh6xw\" (UniqueName: \"kubernetes.io/projected/e11fff9b-5466-4dd4-a845-51b2a07aefe5-kube-api-access-zh6xw\") pod \"lws-controller-manager-68945d9444-x5wrh\" (UID: \"e11fff9b-5466-4dd4-a845-51b2a07aefe5\") " pod="openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh" Sep 17 00:39:54.765096 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:54.765059 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh" Sep 17 00:39:55.035033 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:55.034820 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh"] Sep 17 00:39:55.037849 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:39:55.037822 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11fff9b_5466_4dd4_a845_51b2a07aefe5.slice/crio-32350cd8ebe53ee403e0843f67b4c7b9523309b6ea2b1a1d283dcebc03b77003 WatchSource:0}: Error finding container 32350cd8ebe53ee403e0843f67b4c7b9523309b6ea2b1a1d283dcebc03b77003: Status 404 returned error can't find the container with id 32350cd8ebe53ee403e0843f67b4c7b9523309b6ea2b1a1d283dcebc03b77003 Sep 17 00:39:55.198113 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:55.198078 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh" event={"ID":"e11fff9b-5466-4dd4-a845-51b2a07aefe5","Type":"ContainerStarted","Data":"32350cd8ebe53ee403e0843f67b4c7b9523309b6ea2b1a1d283dcebc03b77003"} Sep 17 00:39:59.222195 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:59.222163 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh" event={"ID":"e11fff9b-5466-4dd4-a845-51b2a07aefe5","Type":"ContainerStarted","Data":"79d1e79830dfc68c475e59ab7af65ab573e4662045fa77eb1e3c886497eac1cd"} Sep 17 00:39:59.222590 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:59.222218 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh" Sep 17 00:39:59.247480 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:39:59.247438 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh" podStartSLOduration=1.776722092 podStartE2EDuration="5.247422656s" totalImagesPullingTime="3.470700564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:39:59.246179698 +0000 UTC m=+612.176813188" watchObservedRunningTime="2026-09-17 00:39:59.247422656 +0000 UTC m=+612.178056144" Sep 17 00:40:01.282653 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.282616 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67"] Sep 17 00:40:01.286345 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.286329 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" Sep 17 00:40:01.288882 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.288863 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 17 00:40:01.289010 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.288869 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 17 00:40:01.289010 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.288974 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mmxsx\"" Sep 17 00:40:01.289225 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.289211 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 17 00:40:01.289396 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.289380 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 17 00:40:01.317287 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.317265 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67"] Sep 17 00:40:01.386812 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.386773 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g54g\" (UniqueName: \"kubernetes.io/projected/5942742c-211c-4a48-ad10-effed1266242-kube-api-access-8g54g\") pod \"opendatahub-operator-controller-manager-596c59f645-s9t67\" (UID: \"5942742c-211c-4a48-ad10-effed1266242\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" Sep 17 00:40:01.386812 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.386812 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5942742c-211c-4a48-ad10-effed1266242-manifests\") pod \"opendatahub-operator-controller-manager-596c59f645-s9t67\" (UID: \"5942742c-211c-4a48-ad10-effed1266242\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" Sep 17 00:40:01.387024 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.386832 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5942742c-211c-4a48-ad10-effed1266242-webhook-cert\") pod \"opendatahub-operator-controller-manager-596c59f645-s9t67\" (UID: \"5942742c-211c-4a48-ad10-effed1266242\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" Sep 17 00:40:01.387024 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.386922 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5942742c-211c-4a48-ad10-effed1266242-apiservice-cert\") pod \"opendatahub-operator-controller-manager-596c59f645-s9t67\" (UID: \"5942742c-211c-4a48-ad10-effed1266242\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" Sep 17 00:40:01.387024 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.386985 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5942742c-211c-4a48-ad10-effed1266242-tmp\") pod \"opendatahub-operator-controller-manager-596c59f645-s9t67\" (UID: \"5942742c-211c-4a48-ad10-effed1266242\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" Sep 17 00:40:01.488361 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.488317 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5942742c-211c-4a48-ad10-effed1266242-apiservice-cert\") pod \"opendatahub-operator-controller-manager-596c59f645-s9t67\" (UID: \"5942742c-211c-4a48-ad10-effed1266242\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" Sep 17 00:40:01.488582 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.488384 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5942742c-211c-4a48-ad10-effed1266242-tmp\") pod \"opendatahub-operator-controller-manager-596c59f645-s9t67\" (UID: \"5942742c-211c-4a48-ad10-effed1266242\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" Sep 17 00:40:01.488582 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.488435 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g54g\" (UniqueName: \"kubernetes.io/projected/5942742c-211c-4a48-ad10-effed1266242-kube-api-access-8g54g\") pod \"opendatahub-operator-controller-manager-596c59f645-s9t67\" (UID: \"5942742c-211c-4a48-ad10-effed1266242\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" Sep 17 00:40:01.488582 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.488462 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5942742c-211c-4a48-ad10-effed1266242-manifests\") pod \"opendatahub-operator-controller-manager-596c59f645-s9t67\" (UID: \"5942742c-211c-4a48-ad10-effed1266242\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" Sep 17 00:40:01.488582 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.488493 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5942742c-211c-4a48-ad10-effed1266242-webhook-cert\") pod \"opendatahub-operator-controller-manager-596c59f645-s9t67\" (UID: \"5942742c-211c-4a48-ad10-effed1266242\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" Sep 17 00:40:01.489058 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.489025 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5942742c-211c-4a48-ad10-effed1266242-manifests\") pod \"opendatahub-operator-controller-manager-596c59f645-s9t67\" (UID: \"5942742c-211c-4a48-ad10-effed1266242\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" Sep 17 00:40:01.491393 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.491359 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5942742c-211c-4a48-ad10-effed1266242-apiservice-cert\") pod \"opendatahub-operator-controller-manager-596c59f645-s9t67\" (UID: \"5942742c-211c-4a48-ad10-effed1266242\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" Sep 17 00:40:01.491489 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.491407 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5942742c-211c-4a48-ad10-effed1266242-tmp\") pod \"opendatahub-operator-controller-manager-596c59f645-s9t67\" (UID: \"5942742c-211c-4a48-ad10-effed1266242\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" Sep 17 00:40:01.491569 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.491547 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5942742c-211c-4a48-ad10-effed1266242-webhook-cert\") pod \"opendatahub-operator-controller-manager-596c59f645-s9t67\" (UID: \"5942742c-211c-4a48-ad10-effed1266242\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" Sep 17 00:40:01.499123 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.499100 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g54g\" (UniqueName: \"kubernetes.io/projected/5942742c-211c-4a48-ad10-effed1266242-kube-api-access-8g54g\") pod \"opendatahub-operator-controller-manager-596c59f645-s9t67\" (UID: \"5942742c-211c-4a48-ad10-effed1266242\") " pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" Sep 17 00:40:01.597247 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.597216 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" Sep 17 00:40:01.873696 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:01.873673 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67"] Sep 17 00:40:01.876600 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:40:01.876569 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5942742c_211c_4a48_ad10_effed1266242.slice/crio-850f9203b3ef6065f0949d7178962d296366dc43c5e60293a30417b7a64c65e6 WatchSource:0}: Error finding container 850f9203b3ef6065f0949d7178962d296366dc43c5e60293a30417b7a64c65e6: Status 404 returned error can't find the container with id 850f9203b3ef6065f0949d7178962d296366dc43c5e60293a30417b7a64c65e6 Sep 17 00:40:02.236856 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:02.236772 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" event={"ID":"5942742c-211c-4a48-ad10-effed1266242","Type":"ContainerStarted","Data":"850f9203b3ef6065f0949d7178962d296366dc43c5e60293a30417b7a64c65e6"} Sep 17 00:40:05.256119 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:05.256023 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" event={"ID":"5942742c-211c-4a48-ad10-effed1266242","Type":"ContainerStarted","Data":"3493e38abe0187210b556557901b25b3f1bb3ffeba4190c7f76fa6151c0bf252"} Sep 17 00:40:06.267952 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:06.267919 2582 generic.go:356] "Generic (PLEG): container finished" podID="5942742c-211c-4a48-ad10-effed1266242" containerID="3493e38abe0187210b556557901b25b3f1bb3ffeba4190c7f76fa6151c0bf252" exitCode=0 Sep 17 00:40:06.268366 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:06.268008 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" event={"ID":"5942742c-211c-4a48-ad10-effed1266242","Type":"ContainerDied","Data":"3493e38abe0187210b556557901b25b3f1bb3ffeba4190c7f76fa6151c0bf252"} Sep 17 00:40:07.275226 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:07.275142 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" event={"ID":"5942742c-211c-4a48-ad10-effed1266242","Type":"ContainerStarted","Data":"966154cd9388ed4f8d9545b39d12f2741578ddf24499d50907c63cfeeb3a5ac1"} Sep 17 00:40:07.275596 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:07.275248 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" Sep 17 00:40:07.295380 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:07.295334 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" podStartSLOduration=1.534020844 podStartE2EDuration="6.295319176s" totalImagesPullingTime="3.737531289s" totalInitContainerRuntime="1.023767043s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:40:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:40:07.293670965 +0000 UTC m=+620.224304453" watchObservedRunningTime="2026-09-17 00:40:07.295319176 +0000 UTC m=+620.225952663" Sep 17 00:40:10.228641 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:10.228566 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-68945d9444-x5wrh" Sep 17 00:40:18.280883 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:18.280854 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-596c59f645-s9t67" Sep 17 00:40:20.598636 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:20.598593 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-67475995c-q8sn2"] Sep 17 00:40:20.605479 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:20.605456 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67475995c-q8sn2" Sep 17 00:40:20.607600 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:20.607579 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 17 00:40:20.607600 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:20.607594 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 17 00:40:20.607743 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:20.607576 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 17 00:40:20.608184 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:20.608158 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 17 00:40:20.608333 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:20.608196 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-p4g7r\"" Sep 17 00:40:20.610698 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:20.610677 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67475995c-q8sn2"] Sep 17 00:40:20.650485 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:20.650454 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84e2eca0-6325-479c-83ef-93d23a0d5c9f-tmp\") pod \"kube-auth-proxy-67475995c-q8sn2\" (UID: \"84e2eca0-6325-479c-83ef-93d23a0d5c9f\") " pod="openshift-ingress/kube-auth-proxy-67475995c-q8sn2" Sep 17 00:40:20.650643 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:20.650515 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84e2eca0-6325-479c-83ef-93d23a0d5c9f-tls-certs\") pod \"kube-auth-proxy-67475995c-q8sn2\" (UID: \"84e2eca0-6325-479c-83ef-93d23a0d5c9f\") " pod="openshift-ingress/kube-auth-proxy-67475995c-q8sn2" Sep 17 00:40:20.650643 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:20.650584 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfmdr\" (UniqueName: \"kubernetes.io/projected/84e2eca0-6325-479c-83ef-93d23a0d5c9f-kube-api-access-cfmdr\") pod \"kube-auth-proxy-67475995c-q8sn2\" (UID: \"84e2eca0-6325-479c-83ef-93d23a0d5c9f\") " pod="openshift-ingress/kube-auth-proxy-67475995c-q8sn2" Sep 17 00:40:20.751940 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:20.751904 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84e2eca0-6325-479c-83ef-93d23a0d5c9f-tmp\") pod \"kube-auth-proxy-67475995c-q8sn2\" (UID: \"84e2eca0-6325-479c-83ef-93d23a0d5c9f\") " pod="openshift-ingress/kube-auth-proxy-67475995c-q8sn2" Sep 17 00:40:20.752097 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:20.751949 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84e2eca0-6325-479c-83ef-93d23a0d5c9f-tls-certs\") pod \"kube-auth-proxy-67475995c-q8sn2\" (UID: \"84e2eca0-6325-479c-83ef-93d23a0d5c9f\") " pod="openshift-ingress/kube-auth-proxy-67475995c-q8sn2" Sep 17 00:40:20.752097 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:20.751995 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfmdr\" (UniqueName: \"kubernetes.io/projected/84e2eca0-6325-479c-83ef-93d23a0d5c9f-kube-api-access-cfmdr\") pod \"kube-auth-proxy-67475995c-q8sn2\" (UID: \"84e2eca0-6325-479c-83ef-93d23a0d5c9f\") " pod="openshift-ingress/kube-auth-proxy-67475995c-q8sn2" Sep 17 00:40:20.752212 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:40:20.752137 2582 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 17 00:40:20.752263 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:40:20.752223 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84e2eca0-6325-479c-83ef-93d23a0d5c9f-tls-certs podName:84e2eca0-6325-479c-83ef-93d23a0d5c9f nodeName:}" failed. No retries permitted until 2026-09-17 00:40:21.252201853 +0000 UTC m=+634.182835338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/84e2eca0-6325-479c-83ef-93d23a0d5c9f-tls-certs") pod "kube-auth-proxy-67475995c-q8sn2" (UID: "84e2eca0-6325-479c-83ef-93d23a0d5c9f") : secret "kube-auth-proxy-tls" not found Sep 17 00:40:20.754336 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:20.754317 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/84e2eca0-6325-479c-83ef-93d23a0d5c9f-tmp\") pod \"kube-auth-proxy-67475995c-q8sn2\" (UID: \"84e2eca0-6325-479c-83ef-93d23a0d5c9f\") " pod="openshift-ingress/kube-auth-proxy-67475995c-q8sn2" Sep 17 00:40:20.759856 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:20.759833 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfmdr\" (UniqueName: \"kubernetes.io/projected/84e2eca0-6325-479c-83ef-93d23a0d5c9f-kube-api-access-cfmdr\") pod \"kube-auth-proxy-67475995c-q8sn2\" (UID: \"84e2eca0-6325-479c-83ef-93d23a0d5c9f\") " pod="openshift-ingress/kube-auth-proxy-67475995c-q8sn2" Sep 17 00:40:21.255928 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:21.255891 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84e2eca0-6325-479c-83ef-93d23a0d5c9f-tls-certs\") pod \"kube-auth-proxy-67475995c-q8sn2\" (UID: \"84e2eca0-6325-479c-83ef-93d23a0d5c9f\") " pod="openshift-ingress/kube-auth-proxy-67475995c-q8sn2" Sep 17 00:40:21.258545 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:21.258516 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84e2eca0-6325-479c-83ef-93d23a0d5c9f-tls-certs\") pod \"kube-auth-proxy-67475995c-q8sn2\" (UID: \"84e2eca0-6325-479c-83ef-93d23a0d5c9f\") " pod="openshift-ingress/kube-auth-proxy-67475995c-q8sn2" Sep 17 00:40:21.517236 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:21.517153 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67475995c-q8sn2" Sep 17 00:40:21.847913 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:21.847881 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67475995c-q8sn2"] Sep 17 00:40:21.851236 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:40:21.851209 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e2eca0_6325_479c_83ef_93d23a0d5c9f.slice/crio-fcdddd92565b24ddc11fbec9057c2e8e08ef2287f9843a712df77e3e63f32c12 WatchSource:0}: Error finding container fcdddd92565b24ddc11fbec9057c2e8e08ef2287f9843a712df77e3e63f32c12: Status 404 returned error can't find the container with id fcdddd92565b24ddc11fbec9057c2e8e08ef2287f9843a712df77e3e63f32c12 Sep 17 00:40:22.348718 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:22.348676 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67475995c-q8sn2" event={"ID":"84e2eca0-6325-479c-83ef-93d23a0d5c9f","Type":"ContainerStarted","Data":"fcdddd92565b24ddc11fbec9057c2e8e08ef2287f9843a712df77e3e63f32c12"} Sep 17 00:40:25.370474 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:25.370440 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67475995c-q8sn2" event={"ID":"84e2eca0-6325-479c-83ef-93d23a0d5c9f","Type":"ContainerStarted","Data":"caca55b1160876f916587dceb542c4060ca03af28d611f28f568e2172fec6b61"} Sep 17 00:40:25.384799 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:40:25.384749 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-67475995c-q8sn2" podStartSLOduration=2.69527492 podStartE2EDuration="5.384732045s" totalImagesPullingTime="2.689457125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:40:25.383825399 +0000 UTC m=+638.314458887" watchObservedRunningTime="2026-09-17 00:40:25.384732045 +0000 UTC m=+638.315365534" Sep 17 00:42:00.171212 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:00.171140 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-zp6sg"] Sep 17 00:42:00.174833 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:00.174812 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zp6sg" Sep 17 00:42:00.176888 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:00.176867 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 17 00:42:00.176980 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:00.176910 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 17 00:42:00.176980 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:00.176962 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 17 00:42:00.177396 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:00.177379 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 17 00:42:00.177462 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:00.177387 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-l29pz\"" Sep 17 00:42:00.181344 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:00.181310 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-zp6sg"] Sep 17 00:42:00.249444 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:00.249408 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-zp6sg\" (UID: \"af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zp6sg" Sep 17 00:42:00.249660 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:00.249458 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-zp6sg\" (UID: \"af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zp6sg" Sep 17 00:42:00.249660 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:00.249553 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl2tr\" (UniqueName: \"kubernetes.io/projected/af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5-kube-api-access-nl2tr\") pod \"kuadrant-console-plugin-fbdd4f445-zp6sg\" (UID: \"af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zp6sg" Sep 17 00:42:00.350129 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:00.350095 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-zp6sg\" (UID: \"af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zp6sg" Sep 17 00:42:00.350129 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:00.350140 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-zp6sg\" (UID: \"af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zp6sg" Sep 17 00:42:00.350363 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:00.350179 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl2tr\" (UniqueName: \"kubernetes.io/projected/af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5-kube-api-access-nl2tr\") pod \"kuadrant-console-plugin-fbdd4f445-zp6sg\" (UID: \"af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zp6sg" Sep 17 00:42:00.350363 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:42:00.350264 2582 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Sep 17 00:42:00.350363 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:42:00.350337 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5-plugin-serving-cert podName:af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5 nodeName:}" failed. No retries permitted until 2026-09-17 00:42:00.850320393 +0000 UTC m=+733.780953858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-zp6sg" (UID: "af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5") : secret "plugin-serving-cert" not found Sep 17 00:42:00.350909 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:00.350888 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-zp6sg\" (UID: \"af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zp6sg" Sep 17 00:42:00.357967 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:00.357947 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl2tr\" (UniqueName: \"kubernetes.io/projected/af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5-kube-api-access-nl2tr\") pod \"kuadrant-console-plugin-fbdd4f445-zp6sg\" (UID: \"af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zp6sg" Sep 17 00:42:00.854859 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:00.854825 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-zp6sg\" (UID: \"af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zp6sg" Sep 17 00:42:00.857369 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:00.857350 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-zp6sg\" (UID: \"af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zp6sg" Sep 17 00:42:01.085717 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:01.085683 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zp6sg" Sep 17 00:42:01.357925 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:01.357892 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-zp6sg"] Sep 17 00:42:01.361557 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:42:01.361528 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf9ef9fe_5e78_4fcf_9d14_e58e7e2be3f5.slice/crio-89e52b852fe82e10b062139ed01fbbdb3e1b21b3dd5c03d85b1d93201fd4972c WatchSource:0}: Error finding container 89e52b852fe82e10b062139ed01fbbdb3e1b21b3dd5c03d85b1d93201fd4972c: Status 404 returned error can't find the container with id 89e52b852fe82e10b062139ed01fbbdb3e1b21b3dd5c03d85b1d93201fd4972c Sep 17 00:42:01.362973 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:01.362958 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 00:42:01.834379 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:01.834340 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zp6sg" event={"ID":"af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5","Type":"ContainerStarted","Data":"89e52b852fe82e10b062139ed01fbbdb3e1b21b3dd5c03d85b1d93201fd4972c"} Sep 17 00:42:06.863773 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:06.863738 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zp6sg" event={"ID":"af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5","Type":"ContainerStarted","Data":"1e977a9a34c1ed6421a5baff9f32bc01fe24c915195882b40fdb95a314ce7111"} Sep 17 00:42:06.879239 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:06.879195 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-zp6sg" podStartSLOduration=2.1990637140000002 podStartE2EDuration="6.879179892s" totalImagesPullingTime="4.680116178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:42:06.87781586 +0000 UTC m=+739.808449346" watchObservedRunningTime="2026-09-17 00:42:06.879179892 +0000 UTC m=+739.809813378" Sep 17 00:42:42.716667 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:42.716622 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:42:42.720230 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:42.720210 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-g59k9" Sep 17 00:42:42.721993 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:42.721977 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 17 00:42:42.729347 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:42.729324 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:42:42.752085 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:42.752055 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:42:42.835051 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:42.835019 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9xnk\" (UniqueName: \"kubernetes.io/projected/303a349d-cec8-4a13-b881-5c72bd957a37-kube-api-access-z9xnk\") pod \"limitador-limitador-84994f7ffc-g59k9\" (UID: \"303a349d-cec8-4a13-b881-5c72bd957a37\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-g59k9" Sep 17 00:42:42.835210 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:42.835074 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/303a349d-cec8-4a13-b881-5c72bd957a37-config-file\") pod \"limitador-limitador-84994f7ffc-g59k9\" (UID: \"303a349d-cec8-4a13-b881-5c72bd957a37\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-g59k9" Sep 17 00:42:42.935977 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:42.935947 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9xnk\" (UniqueName: \"kubernetes.io/projected/303a349d-cec8-4a13-b881-5c72bd957a37-kube-api-access-z9xnk\") pod \"limitador-limitador-84994f7ffc-g59k9\" (UID: \"303a349d-cec8-4a13-b881-5c72bd957a37\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-g59k9" Sep 17 00:42:42.936162 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:42.935997 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/303a349d-cec8-4a13-b881-5c72bd957a37-config-file\") pod \"limitador-limitador-84994f7ffc-g59k9\" (UID: \"303a349d-cec8-4a13-b881-5c72bd957a37\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-g59k9" Sep 17 00:42:42.936584 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:42.936567 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/303a349d-cec8-4a13-b881-5c72bd957a37-config-file\") pod \"limitador-limitador-84994f7ffc-g59k9\" (UID: \"303a349d-cec8-4a13-b881-5c72bd957a37\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-g59k9" Sep 17 00:42:42.943313 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:42.943288 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9xnk\" (UniqueName: \"kubernetes.io/projected/303a349d-cec8-4a13-b881-5c72bd957a37-kube-api-access-z9xnk\") pod \"limitador-limitador-84994f7ffc-g59k9\" (UID: \"303a349d-cec8-4a13-b881-5c72bd957a37\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-g59k9" Sep 17 00:42:43.031596 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:43.031534 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-g59k9" Sep 17 00:42:43.232547 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:43.232485 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4s6nn"] Sep 17 00:42:43.236224 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:43.236201 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-4s6nn" Sep 17 00:42:43.238043 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:43.238015 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5bclt\"" Sep 17 00:42:43.242148 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:43.242129 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4s6nn"] Sep 17 00:42:43.300320 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:43.300299 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:42:43.302875 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:42:43.302849 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303a349d_cec8_4a13_b881_5c72bd957a37.slice/crio-2d044d6ec3b02af4e7efd91095a691770141590473cce2a78056d275e611600f WatchSource:0}: Error finding container 2d044d6ec3b02af4e7efd91095a691770141590473cce2a78056d275e611600f: Status 404 returned error can't find the container with id 2d044d6ec3b02af4e7efd91095a691770141590473cce2a78056d275e611600f Sep 17 00:42:43.339708 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:43.339683 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lwdb\" (UniqueName: \"kubernetes.io/projected/a73987e3-c1b2-4b8c-a5ad-3b5c8167757f-kube-api-access-2lwdb\") pod \"authorino-7bcfd8f9cd-4s6nn\" (UID: \"a73987e3-c1b2-4b8c-a5ad-3b5c8167757f\") " pod="kuadrant-system/authorino-7bcfd8f9cd-4s6nn" Sep 17 00:42:43.440455 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:43.440416 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lwdb\" (UniqueName: \"kubernetes.io/projected/a73987e3-c1b2-4b8c-a5ad-3b5c8167757f-kube-api-access-2lwdb\") pod \"authorino-7bcfd8f9cd-4s6nn\" (UID: \"a73987e3-c1b2-4b8c-a5ad-3b5c8167757f\") " pod="kuadrant-system/authorino-7bcfd8f9cd-4s6nn" Sep 17 00:42:43.447893 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:43.447862 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lwdb\" (UniqueName: \"kubernetes.io/projected/a73987e3-c1b2-4b8c-a5ad-3b5c8167757f-kube-api-access-2lwdb\") pod \"authorino-7bcfd8f9cd-4s6nn\" (UID: \"a73987e3-c1b2-4b8c-a5ad-3b5c8167757f\") " pod="kuadrant-system/authorino-7bcfd8f9cd-4s6nn" Sep 17 00:42:43.546780 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:43.546755 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-4s6nn" Sep 17 00:42:43.812846 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:43.812821 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4s6nn"] Sep 17 00:42:43.814917 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:42:43.814895 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda73987e3_c1b2_4b8c_a5ad_3b5c8167757f.slice/crio-b8566850bb52c834a3fe145bc05f1245374ccd84acb6fb5749197c25abbb3211 WatchSource:0}: Error finding container b8566850bb52c834a3fe145bc05f1245374ccd84acb6fb5749197c25abbb3211: Status 404 returned error can't find the container with id b8566850bb52c834a3fe145bc05f1245374ccd84acb6fb5749197c25abbb3211 Sep 17 00:42:44.043153 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:44.043119 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-g59k9" event={"ID":"303a349d-cec8-4a13-b881-5c72bd957a37","Type":"ContainerStarted","Data":"2d044d6ec3b02af4e7efd91095a691770141590473cce2a78056d275e611600f"} Sep 17 00:42:44.044402 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:44.044377 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-4s6nn" event={"ID":"a73987e3-c1b2-4b8c-a5ad-3b5c8167757f","Type":"ContainerStarted","Data":"b8566850bb52c834a3fe145bc05f1245374ccd84acb6fb5749197c25abbb3211"} Sep 17 00:42:47.063810 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:47.063771 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-g59k9" event={"ID":"303a349d-cec8-4a13-b881-5c72bd957a37","Type":"ContainerStarted","Data":"ddb5662418c4c2827998a8f60fa95b11c710221edd864538f6fa82ba7a66ccef"} Sep 17 00:42:47.064238 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:47.063843 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-g59k9" Sep 17 00:42:47.065610 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:47.065582 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-4s6nn" event={"ID":"a73987e3-c1b2-4b8c-a5ad-3b5c8167757f","Type":"ContainerStarted","Data":"bd72c733c83c35b180c5c6c807df48081be5c14484a32238059bdaf29d4a9641"} Sep 17 00:42:47.079331 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:47.079285 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-g59k9" podStartSLOduration=2.1171294 podStartE2EDuration="5.079269344s" totalImagesPullingTime="2.962139944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:42:47.077850577 +0000 UTC m=+780.008484064" watchObservedRunningTime="2026-09-17 00:42:47.079269344 +0000 UTC m=+780.009902866" Sep 17 00:42:47.083379 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:47.083358 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4s6nn"] Sep 17 00:42:47.095313 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:47.095272 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-4s6nn" podStartSLOduration=1.638164054 podStartE2EDuration="4.095257529s" totalImagesPullingTime="2.457093475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:42:47.094191877 +0000 UTC m=+780.024825365" watchObservedRunningTime="2026-09-17 00:42:47.095257529 +0000 UTC m=+780.025891037" Sep 17 00:42:49.074876 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:49.074817 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-4s6nn" podUID="a73987e3-c1b2-4b8c-a5ad-3b5c8167757f" containerName="authorino" containerID="cri-o://bd72c733c83c35b180c5c6c807df48081be5c14484a32238059bdaf29d4a9641" gracePeriod=30 Sep 17 00:42:49.327586 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:49.327518 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-4s6nn" Sep 17 00:42:49.397460 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:49.397428 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lwdb\" (UniqueName: \"kubernetes.io/projected/a73987e3-c1b2-4b8c-a5ad-3b5c8167757f-kube-api-access-2lwdb\") pod \"a73987e3-c1b2-4b8c-a5ad-3b5c8167757f\" (UID: \"a73987e3-c1b2-4b8c-a5ad-3b5c8167757f\") " Sep 17 00:42:49.399905 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:49.399870 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a73987e3-c1b2-4b8c-a5ad-3b5c8167757f-kube-api-access-2lwdb" (OuterVolumeSpecName: "kube-api-access-2lwdb") pod "a73987e3-c1b2-4b8c-a5ad-3b5c8167757f" (UID: "a73987e3-c1b2-4b8c-a5ad-3b5c8167757f"). InnerVolumeSpecName "kube-api-access-2lwdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:42:49.498538 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:49.498467 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lwdb\" (UniqueName: \"kubernetes.io/projected/a73987e3-c1b2-4b8c-a5ad-3b5c8167757f-kube-api-access-2lwdb\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:42:50.080958 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:50.080923 2582 generic.go:356] "Generic (PLEG): container finished" podID="a73987e3-c1b2-4b8c-a5ad-3b5c8167757f" containerID="bd72c733c83c35b180c5c6c807df48081be5c14484a32238059bdaf29d4a9641" exitCode=0 Sep 17 00:42:50.081416 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:50.080971 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-4s6nn" event={"ID":"a73987e3-c1b2-4b8c-a5ad-3b5c8167757f","Type":"ContainerDied","Data":"bd72c733c83c35b180c5c6c807df48081be5c14484a32238059bdaf29d4a9641"} Sep 17 00:42:50.081416 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:50.080985 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-4s6nn" Sep 17 00:42:50.081416 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:50.081001 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-4s6nn" event={"ID":"a73987e3-c1b2-4b8c-a5ad-3b5c8167757f","Type":"ContainerDied","Data":"b8566850bb52c834a3fe145bc05f1245374ccd84acb6fb5749197c25abbb3211"} Sep 17 00:42:50.081416 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:50.081018 2582 scope.go:117] "RemoveContainer" containerID="bd72c733c83c35b180c5c6c807df48081be5c14484a32238059bdaf29d4a9641" Sep 17 00:42:50.089661 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:50.089643 2582 scope.go:117] "RemoveContainer" containerID="bd72c733c83c35b180c5c6c807df48081be5c14484a32238059bdaf29d4a9641" Sep 17 00:42:50.089936 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:42:50.089920 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd72c733c83c35b180c5c6c807df48081be5c14484a32238059bdaf29d4a9641\": container with ID starting with bd72c733c83c35b180c5c6c807df48081be5c14484a32238059bdaf29d4a9641 not found: ID does not exist" containerID="bd72c733c83c35b180c5c6c807df48081be5c14484a32238059bdaf29d4a9641" Sep 17 00:42:50.090011 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:50.089942 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd72c733c83c35b180c5c6c807df48081be5c14484a32238059bdaf29d4a9641"} err="failed to get container status \"bd72c733c83c35b180c5c6c807df48081be5c14484a32238059bdaf29d4a9641\": rpc error: code = NotFound desc = could not find container \"bd72c733c83c35b180c5c6c807df48081be5c14484a32238059bdaf29d4a9641\": container with ID starting with bd72c733c83c35b180c5c6c807df48081be5c14484a32238059bdaf29d4a9641 not found: ID does not exist" Sep 17 00:42:50.095012 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:50.094989 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4s6nn"] Sep 17 00:42:50.101489 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:50.101437 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-4s6nn"] Sep 17 00:42:51.651569 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:51.651536 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a73987e3-c1b2-4b8c-a5ad-3b5c8167757f" path="/var/lib/kubelet/pods/a73987e3-c1b2-4b8c-a5ad-3b5c8167757f/volumes" Sep 17 00:42:58.071684 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:42:58.071650 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-g59k9" Sep 17 00:44:04.903922 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:44:04.903893 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:44:47.635147 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:44:47.635036 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/1.log" Sep 17 00:44:47.641730 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:44:47.641710 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/1.log" Sep 17 00:44:55.195195 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:44:55.195163 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:45:23.895452 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:23.895417 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95"] Sep 17 00:45:23.895971 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:23.895849 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a73987e3-c1b2-4b8c-a5ad-3b5c8167757f" containerName="authorino" Sep 17 00:45:23.895971 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:23.895862 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a73987e3-c1b2-4b8c-a5ad-3b5c8167757f" containerName="authorino" Sep 17 00:45:23.895971 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:23.895942 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a73987e3-c1b2-4b8c-a5ad-3b5c8167757f" containerName="authorino" Sep 17 00:45:23.899146 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:23.899127 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:23.901772 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:23.901751 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 17 00:45:23.901772 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:23.901766 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 17 00:45:23.901958 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:23.901791 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vlt7q\"" Sep 17 00:45:23.901958 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:23.901821 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 17 00:45:23.908549 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:23.908527 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95"] Sep 17 00:45:24.039230 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.039198 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81343895-85b3-4e34-afb4-5cf2fc6466ef-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9vr95\" (UID: \"81343895-85b3-4e34-afb4-5cf2fc6466ef\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:24.039384 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.039241 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81343895-85b3-4e34-afb4-5cf2fc6466ef-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9vr95\" (UID: \"81343895-85b3-4e34-afb4-5cf2fc6466ef\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:24.039384 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.039264 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81343895-85b3-4e34-afb4-5cf2fc6466ef-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9vr95\" (UID: \"81343895-85b3-4e34-afb4-5cf2fc6466ef\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:24.039384 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.039355 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81343895-85b3-4e34-afb4-5cf2fc6466ef-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9vr95\" (UID: \"81343895-85b3-4e34-afb4-5cf2fc6466ef\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:24.039487 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.039411 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6pjb\" (UniqueName: \"kubernetes.io/projected/81343895-85b3-4e34-afb4-5cf2fc6466ef-kube-api-access-f6pjb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9vr95\" (UID: \"81343895-85b3-4e34-afb4-5cf2fc6466ef\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:24.039487 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.039456 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81343895-85b3-4e34-afb4-5cf2fc6466ef-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9vr95\" (UID: \"81343895-85b3-4e34-afb4-5cf2fc6466ef\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:24.140334 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.140300 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81343895-85b3-4e34-afb4-5cf2fc6466ef-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9vr95\" (UID: \"81343895-85b3-4e34-afb4-5cf2fc6466ef\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:24.140494 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.140358 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6pjb\" (UniqueName: \"kubernetes.io/projected/81343895-85b3-4e34-afb4-5cf2fc6466ef-kube-api-access-f6pjb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9vr95\" (UID: \"81343895-85b3-4e34-afb4-5cf2fc6466ef\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:24.140494 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.140389 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81343895-85b3-4e34-afb4-5cf2fc6466ef-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9vr95\" (UID: \"81343895-85b3-4e34-afb4-5cf2fc6466ef\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:24.140494 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.140438 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81343895-85b3-4e34-afb4-5cf2fc6466ef-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9vr95\" (UID: \"81343895-85b3-4e34-afb4-5cf2fc6466ef\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:24.140494 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.140477 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81343895-85b3-4e34-afb4-5cf2fc6466ef-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9vr95\" (UID: \"81343895-85b3-4e34-afb4-5cf2fc6466ef\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:24.140494 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.140522 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81343895-85b3-4e34-afb4-5cf2fc6466ef-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9vr95\" (UID: \"81343895-85b3-4e34-afb4-5cf2fc6466ef\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:24.140879 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.140849 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81343895-85b3-4e34-afb4-5cf2fc6466ef-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9vr95\" (UID: \"81343895-85b3-4e34-afb4-5cf2fc6466ef\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:24.140949 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.140883 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81343895-85b3-4e34-afb4-5cf2fc6466ef-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9vr95\" (UID: \"81343895-85b3-4e34-afb4-5cf2fc6466ef\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:24.140949 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.140897 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81343895-85b3-4e34-afb4-5cf2fc6466ef-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9vr95\" (UID: \"81343895-85b3-4e34-afb4-5cf2fc6466ef\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:24.142823 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.142803 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81343895-85b3-4e34-afb4-5cf2fc6466ef-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9vr95\" (UID: \"81343895-85b3-4e34-afb4-5cf2fc6466ef\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:24.142964 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.142948 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81343895-85b3-4e34-afb4-5cf2fc6466ef-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9vr95\" (UID: \"81343895-85b3-4e34-afb4-5cf2fc6466ef\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:24.146963 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.146920 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6pjb\" (UniqueName: \"kubernetes.io/projected/81343895-85b3-4e34-afb4-5cf2fc6466ef-kube-api-access-f6pjb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-9vr95\" (UID: \"81343895-85b3-4e34-afb4-5cf2fc6466ef\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:24.210160 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.210134 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:24.480739 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.480712 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95"] Sep 17 00:45:24.483244 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:45:24.483207 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81343895_85b3_4e34_afb4_5cf2fc6466ef.slice/crio-8f8b05b58137ef8b025d547ebb89bfa62980a4f75a57d408f7024ed20ee40d44 WatchSource:0}: Error finding container 8f8b05b58137ef8b025d547ebb89bfa62980a4f75a57d408f7024ed20ee40d44: Status 404 returned error can't find the container with id 8f8b05b58137ef8b025d547ebb89bfa62980a4f75a57d408f7024ed20ee40d44 Sep 17 00:45:24.589467 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.589435 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:45:24.848156 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:24.848081 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" event={"ID":"81343895-85b3-4e34-afb4-5cf2fc6466ef","Type":"ContainerStarted","Data":"8f8b05b58137ef8b025d547ebb89bfa62980a4f75a57d408f7024ed20ee40d44"} Sep 17 00:45:25.854723 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:25.854691 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" event={"ID":"81343895-85b3-4e34-afb4-5cf2fc6466ef","Type":"ContainerStarted","Data":"2d9d204b0adc063ac642def619b5e2ca4bbdca653d2d9a0d5fd85810d5002ef4"} Sep 17 00:45:25.870802 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:25.870757 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" podStartSLOduration=1.716198566 podStartE2EDuration="2.870741221s" totalImagesPullingTime="1.154542655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:45:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:45:25.868306419 +0000 UTC m=+938.798939906" watchObservedRunningTime="2026-09-17 00:45:25.870741221 +0000 UTC m=+938.801374752" Sep 17 00:45:34.210472 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:34.210435 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:34.211024 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:34.210549 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:34.223064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:34.223037 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:34.909139 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:34.909113 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-9vr95" Sep 17 00:45:40.489214 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:40.489178 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:45:48.199096 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:45:48.199055 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:46:07.183021 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.182985 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz"] Sep 17 00:46:07.186679 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.186660 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:07.188641 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.188623 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 17 00:46:07.196306 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.196281 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz"] Sep 17 00:46:07.231405 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.231379 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwrlw\" (UniqueName: \"kubernetes.io/projected/184376f5-676f-4d5b-b32e-0182b96cef13-kube-api-access-dwrlw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz\" (UID: \"184376f5-676f-4d5b-b32e-0182b96cef13\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:07.231534 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.231425 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/184376f5-676f-4d5b-b32e-0182b96cef13-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz\" (UID: \"184376f5-676f-4d5b-b32e-0182b96cef13\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:07.231534 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.231466 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/184376f5-676f-4d5b-b32e-0182b96cef13-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz\" (UID: \"184376f5-676f-4d5b-b32e-0182b96cef13\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:07.231534 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.231519 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/184376f5-676f-4d5b-b32e-0182b96cef13-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz\" (UID: \"184376f5-676f-4d5b-b32e-0182b96cef13\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:07.231658 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.231566 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/184376f5-676f-4d5b-b32e-0182b96cef13-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz\" (UID: \"184376f5-676f-4d5b-b32e-0182b96cef13\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:07.231658 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.231618 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/184376f5-676f-4d5b-b32e-0182b96cef13-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz\" (UID: \"184376f5-676f-4d5b-b32e-0182b96cef13\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:07.332647 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.332620 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/184376f5-676f-4d5b-b32e-0182b96cef13-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz\" (UID: \"184376f5-676f-4d5b-b32e-0182b96cef13\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:07.332755 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.332676 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/184376f5-676f-4d5b-b32e-0182b96cef13-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz\" (UID: \"184376f5-676f-4d5b-b32e-0182b96cef13\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:07.332755 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.332721 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwrlw\" (UniqueName: \"kubernetes.io/projected/184376f5-676f-4d5b-b32e-0182b96cef13-kube-api-access-dwrlw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz\" (UID: \"184376f5-676f-4d5b-b32e-0182b96cef13\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:07.332755 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.332749 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/184376f5-676f-4d5b-b32e-0182b96cef13-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz\" (UID: \"184376f5-676f-4d5b-b32e-0182b96cef13\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:07.332906 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.332788 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/184376f5-676f-4d5b-b32e-0182b96cef13-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz\" (UID: \"184376f5-676f-4d5b-b32e-0182b96cef13\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:07.333020 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.332992 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/184376f5-676f-4d5b-b32e-0182b96cef13-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz\" (UID: \"184376f5-676f-4d5b-b32e-0182b96cef13\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:07.333084 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.333025 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/184376f5-676f-4d5b-b32e-0182b96cef13-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz\" (UID: \"184376f5-676f-4d5b-b32e-0182b96cef13\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:07.333084 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.333065 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/184376f5-676f-4d5b-b32e-0182b96cef13-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz\" (UID: \"184376f5-676f-4d5b-b32e-0182b96cef13\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:07.333274 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.333254 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/184376f5-676f-4d5b-b32e-0182b96cef13-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz\" (UID: \"184376f5-676f-4d5b-b32e-0182b96cef13\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:07.335175 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.335157 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/184376f5-676f-4d5b-b32e-0182b96cef13-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz\" (UID: \"184376f5-676f-4d5b-b32e-0182b96cef13\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:07.335434 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.335412 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/184376f5-676f-4d5b-b32e-0182b96cef13-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz\" (UID: \"184376f5-676f-4d5b-b32e-0182b96cef13\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:07.340128 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.340101 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwrlw\" (UniqueName: \"kubernetes.io/projected/184376f5-676f-4d5b-b32e-0182b96cef13-kube-api-access-dwrlw\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz\" (UID: \"184376f5-676f-4d5b-b32e-0182b96cef13\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:07.497979 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.497907 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:07.768172 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:07.768090 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz"] Sep 17 00:46:07.771316 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:46:07.771284 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184376f5_676f_4d5b_b32e_0182b96cef13.slice/crio-938170040d5c5e06cb36e774d0c83fb990dfff99ea23309ec5d34a47c97030cc WatchSource:0}: Error finding container 938170040d5c5e06cb36e774d0c83fb990dfff99ea23309ec5d34a47c97030cc: Status 404 returned error can't find the container with id 938170040d5c5e06cb36e774d0c83fb990dfff99ea23309ec5d34a47c97030cc Sep 17 00:46:08.060953 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:08.060919 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" event={"ID":"184376f5-676f-4d5b-b32e-0182b96cef13","Type":"ContainerStarted","Data":"3c64ef3fdeed6bf7fe4d8ec9fa2c08538f677b0a51030fd5d06fb8355f76fb5f"} Sep 17 00:46:08.061080 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:08.060992 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" event={"ID":"184376f5-676f-4d5b-b32e-0182b96cef13","Type":"ContainerStarted","Data":"938170040d5c5e06cb36e774d0c83fb990dfff99ea23309ec5d34a47c97030cc"} Sep 17 00:46:08.076417 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:08.076363 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" podStartSLOduration=0.866152104 podStartE2EDuration="1.07634309s" totalImagesPullingTime="210.190986ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:46:08.074639886 +0000 UTC m=+981.005273373" watchObservedRunningTime="2026-09-17 00:46:08.07634309 +0000 UTC m=+981.006976577" Sep 17 00:46:08.694676 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:08.694600 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:46:17.196162 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:17.196126 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:46:17.498806 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:17.498721 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:17.498806 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:17.498799 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:17.511874 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:17.511843 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:18.120481 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:18.120448 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz" Sep 17 00:46:33.890241 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:46:33.890199 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:47:37.712747 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:37.712718 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cc5f98657-8w2m9"] Sep 17 00:47:37.716315 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:37.716297 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cc5f98657-8w2m9" Sep 17 00:47:37.718766 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:37.718743 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5bclt\"" Sep 17 00:47:37.718766 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:37.718765 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 17 00:47:37.723146 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:37.723124 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cc5f98657-8w2m9"] Sep 17 00:47:37.820787 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:37.820757 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z66nq\" (UniqueName: \"kubernetes.io/projected/291bccc7-15e3-4fc5-8b83-b2d0920151a5-kube-api-access-z66nq\") pod \"authorino-7cc5f98657-8w2m9\" (UID: \"291bccc7-15e3-4fc5-8b83-b2d0920151a5\") " pod="kuadrant-system/authorino-7cc5f98657-8w2m9" Sep 17 00:47:37.820949 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:37.820805 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/291bccc7-15e3-4fc5-8b83-b2d0920151a5-tls-cert\") pod \"authorino-7cc5f98657-8w2m9\" (UID: \"291bccc7-15e3-4fc5-8b83-b2d0920151a5\") " pod="kuadrant-system/authorino-7cc5f98657-8w2m9" Sep 17 00:47:37.922204 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:37.922175 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z66nq\" (UniqueName: \"kubernetes.io/projected/291bccc7-15e3-4fc5-8b83-b2d0920151a5-kube-api-access-z66nq\") pod \"authorino-7cc5f98657-8w2m9\" (UID: \"291bccc7-15e3-4fc5-8b83-b2d0920151a5\") " pod="kuadrant-system/authorino-7cc5f98657-8w2m9" Sep 17 00:47:37.922342 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:37.922215 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/291bccc7-15e3-4fc5-8b83-b2d0920151a5-tls-cert\") pod \"authorino-7cc5f98657-8w2m9\" (UID: \"291bccc7-15e3-4fc5-8b83-b2d0920151a5\") " pod="kuadrant-system/authorino-7cc5f98657-8w2m9" Sep 17 00:47:37.924846 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:37.924817 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/291bccc7-15e3-4fc5-8b83-b2d0920151a5-tls-cert\") pod \"authorino-7cc5f98657-8w2m9\" (UID: \"291bccc7-15e3-4fc5-8b83-b2d0920151a5\") " pod="kuadrant-system/authorino-7cc5f98657-8w2m9" Sep 17 00:47:37.929123 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:37.929099 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z66nq\" (UniqueName: \"kubernetes.io/projected/291bccc7-15e3-4fc5-8b83-b2d0920151a5-kube-api-access-z66nq\") pod \"authorino-7cc5f98657-8w2m9\" (UID: \"291bccc7-15e3-4fc5-8b83-b2d0920151a5\") " pod="kuadrant-system/authorino-7cc5f98657-8w2m9" Sep 17 00:47:38.026573 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:38.026457 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cc5f98657-8w2m9" Sep 17 00:47:38.297118 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:38.297094 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cc5f98657-8w2m9"] Sep 17 00:47:38.299352 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:47:38.299320 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod291bccc7_15e3_4fc5_8b83_b2d0920151a5.slice/crio-211ebf0985be552e466eb9a355cf2d6186e8ceb36254348590b9b310544abcef WatchSource:0}: Error finding container 211ebf0985be552e466eb9a355cf2d6186e8ceb36254348590b9b310544abcef: Status 404 returned error can't find the container with id 211ebf0985be552e466eb9a355cf2d6186e8ceb36254348590b9b310544abcef Sep 17 00:47:38.300672 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:38.300654 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 00:47:38.520153 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:38.520087 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cc5f98657-8w2m9" event={"ID":"291bccc7-15e3-4fc5-8b83-b2d0920151a5","Type":"ContainerStarted","Data":"211ebf0985be552e466eb9a355cf2d6186e8ceb36254348590b9b310544abcef"} Sep 17 00:47:39.526743 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:39.526647 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cc5f98657-8w2m9" event={"ID":"291bccc7-15e3-4fc5-8b83-b2d0920151a5","Type":"ContainerStarted","Data":"4068dbb73fd065abb2f609dfc978dfecbad8af46d09eec8dcae59d1a8874e3c7"} Sep 17 00:47:39.542137 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:39.542071 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cc5f98657-8w2m9" podStartSLOduration=1.765172661 podStartE2EDuration="2.542048294s" totalImagesPullingTime="776.875633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:47:39.540652615 +0000 UTC m=+1072.471286102" watchObservedRunningTime="2026-09-17 00:47:39.542048294 +0000 UTC m=+1072.472681783" Sep 17 00:47:58.404057 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:58.404022 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7705-7905b0"] Sep 17 00:47:58.407571 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:58.407554 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-7905b0" Sep 17 00:47:58.409434 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:58.409418 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bd8qq\"" Sep 17 00:47:58.412859 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:58.412837 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-7905b0"] Sep 17 00:47:58.512893 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:58.512862 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gss55\" (UniqueName: \"kubernetes.io/projected/fab35c9d-e845-47fe-8671-31e86483ceed-kube-api-access-gss55\") pod \"test-curl-7705-7905b0\" (UID: \"fab35c9d-e845-47fe-8671-31e86483ceed\") " pod="opendatahub/test-curl-7705-7905b0" Sep 17 00:47:58.614323 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:58.614298 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gss55\" (UniqueName: \"kubernetes.io/projected/fab35c9d-e845-47fe-8671-31e86483ceed-kube-api-access-gss55\") pod \"test-curl-7705-7905b0\" (UID: \"fab35c9d-e845-47fe-8671-31e86483ceed\") " pod="opendatahub/test-curl-7705-7905b0" Sep 17 00:47:58.621682 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:58.621661 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gss55\" (UniqueName: \"kubernetes.io/projected/fab35c9d-e845-47fe-8671-31e86483ceed-kube-api-access-gss55\") pod \"test-curl-7705-7905b0\" (UID: \"fab35c9d-e845-47fe-8671-31e86483ceed\") " pod="opendatahub/test-curl-7705-7905b0" Sep 17 00:47:58.717552 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:58.717469 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-7905b0" Sep 17 00:47:58.985879 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:58.985837 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-7905b0"] Sep 17 00:47:58.988452 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:47:58.988423 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfab35c9d_e845_47fe_8671_31e86483ceed.slice/crio-2cd8b740893e3567330434f1655689b41cf2c57f7e6d2fa10cbb399920b57baf WatchSource:0}: Error finding container 2cd8b740893e3567330434f1655689b41cf2c57f7e6d2fa10cbb399920b57baf: Status 404 returned error can't find the container with id 2cd8b740893e3567330434f1655689b41cf2c57f7e6d2fa10cbb399920b57baf Sep 17 00:47:59.636666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:47:59.636629 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-7905b0" event={"ID":"fab35c9d-e845-47fe-8671-31e86483ceed","Type":"ContainerStarted","Data":"2cd8b740893e3567330434f1655689b41cf2c57f7e6d2fa10cbb399920b57baf"} Sep 17 00:48:00.397342 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:00.397297 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:48:01.650578 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:01.650540 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-7905b0" event={"ID":"fab35c9d-e845-47fe-8671-31e86483ceed","Type":"ContainerStarted","Data":"4d188880ddf8038d9f477749046f0655d3e388d2a933725e726602842c43dd15"} Sep 17 00:48:01.668726 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:01.668624 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7705-7905b0" podStartSLOduration=1.247177511 podStartE2EDuration="3.668606645s" totalImagesPullingTime="2.421429134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:48:01.666714265 +0000 UTC m=+1094.597347753" watchObservedRunningTime="2026-09-17 00:48:01.668606645 +0000 UTC m=+1094.599240184" Sep 17 00:48:02.249616 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.249580 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7705-7905b0"] Sep 17 00:48:02.254146 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.254113 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7705-7905b0"] Sep 17 00:48:02.364476 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.364381 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7705-694783"] Sep 17 00:48:02.364833 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.364818 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fab35c9d-e845-47fe-8671-31e86483ceed" containerName="test-curl-7705-7905b0" Sep 17 00:48:02.364833 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.364834 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fab35c9d-e845-47fe-8671-31e86483ceed" containerName="test-curl-7705-7905b0" Sep 17 00:48:02.364963 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.364894 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fab35c9d-e845-47fe-8671-31e86483ceed" containerName="test-curl-7705-7905b0" Sep 17 00:48:02.368234 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.368211 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-694783" Sep 17 00:48:02.374360 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.374319 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-694783"] Sep 17 00:48:02.452125 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.452084 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmb88\" (UniqueName: \"kubernetes.io/projected/06981310-db4e-420a-9bc6-bcc9728f5b2b-kube-api-access-cmb88\") pod \"test-curl-7705-694783\" (UID: \"06981310-db4e-420a-9bc6-bcc9728f5b2b\") " pod="opendatahub/test-curl-7705-694783" Sep 17 00:48:02.553103 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.553062 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmb88\" (UniqueName: \"kubernetes.io/projected/06981310-db4e-420a-9bc6-bcc9728f5b2b-kube-api-access-cmb88\") pod \"test-curl-7705-694783\" (UID: \"06981310-db4e-420a-9bc6-bcc9728f5b2b\") " pod="opendatahub/test-curl-7705-694783" Sep 17 00:48:02.560541 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.560492 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmb88\" (UniqueName: \"kubernetes.io/projected/06981310-db4e-420a-9bc6-bcc9728f5b2b-kube-api-access-cmb88\") pod \"test-curl-7705-694783\" (UID: \"06981310-db4e-420a-9bc6-bcc9728f5b2b\") " pod="opendatahub/test-curl-7705-694783" Sep 17 00:48:02.623539 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.623438 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49"] Sep 17 00:48:02.627128 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.627107 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" Sep 17 00:48:02.630120 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.629211 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 00:48:02.630120 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.629845 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gxvb4\"" Sep 17 00:48:02.630120 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.630101 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 00:48:02.630364 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.630293 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-f40754-serving-cert\"" Sep 17 00:48:02.630427 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.630368 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 00:48:02.635064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.635036 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49"] Sep 17 00:48:02.657427 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:02.657397 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7705-7905b0" container="test-curl-7705-7905b0" Sep 17 00:48:02.679373 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.679337 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-694783" Sep 17 00:48:02.755183 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.755150 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49\" (UID: \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" Sep 17 00:48:02.755349 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.755241 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-metrics-certs\") pod \"maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49\" (UID: \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" Sep 17 00:48:02.755349 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.755313 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr47g\" (UniqueName: \"kubernetes.io/projected/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-kube-api-access-zr47g\") pod \"maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49\" (UID: \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" Sep 17 00:48:02.755349 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.755341 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49\" (UID: \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" Sep 17 00:48:02.856443 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.856400 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr47g\" (UniqueName: \"kubernetes.io/projected/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-kube-api-access-zr47g\") pod \"maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49\" (UID: \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" Sep 17 00:48:02.856684 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.856453 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49\" (UID: \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" Sep 17 00:48:02.856684 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.856544 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49\" (UID: \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" Sep 17 00:48:02.856684 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.856599 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-metrics-certs\") pod \"maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49\" (UID: \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" Sep 17 00:48:02.856684 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:02.856632 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-serving-cert: secret "maas-api-e2e-worker-w4-f40754-serving-cert" not found Sep 17 00:48:02.856855 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:02.856709 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-maas-api-tls podName:221c5b4f-3870-4aa3-8fa4-77a46596ffc2 nodeName:}" failed. No retries permitted until 2026-09-17 00:48:03.35668856 +0000 UTC m=+1096.287322030 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-maas-api-tls") pod "maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" (UID: "221c5b4f-3870-4aa3-8fa4-77a46596ffc2") : secret "maas-api-e2e-worker-w4-f40754-serving-cert" not found Sep 17 00:48:02.857291 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.857273 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49\" (UID: \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" Sep 17 00:48:02.859250 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.859232 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-metrics-certs\") pod \"maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49\" (UID: \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" Sep 17 00:48:02.864872 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.864846 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr47g\" (UniqueName: \"kubernetes.io/projected/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-kube-api-access-zr47g\") pod \"maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49\" (UID: \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" Sep 17 00:48:02.954990 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:02.954924 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7705-694783"] Sep 17 00:48:02.957699 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:48:02.957675 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06981310_db4e_420a_9bc6_bcc9728f5b2b.slice/crio-23e93947c26c641c2d064667b015bc7de6fff3a49f2e5e70b3f3ad86b8b580f1 WatchSource:0}: Error finding container 23e93947c26c641c2d064667b015bc7de6fff3a49f2e5e70b3f3ad86b8b580f1: Status 404 returned error can't find the container with id 23e93947c26c641c2d064667b015bc7de6fff3a49f2e5e70b3f3ad86b8b580f1 Sep 17 00:48:03.362756 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:03.362714 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49\" (UID: \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" Sep 17 00:48:03.365307 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:03.365287 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49\" (UID: \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" Sep 17 00:48:03.542383 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:03.542345 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" Sep 17 00:48:03.663247 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:03.663160 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-694783" event={"ID":"06981310-db4e-420a-9bc6-bcc9728f5b2b","Type":"ContainerStarted","Data":"f924335c3a723c94cad5134051215ac282ddbfa647970fdbe9d2cde0c4e82b0f"} Sep 17 00:48:03.663247 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:03.663204 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7705-694783" event={"ID":"06981310-db4e-420a-9bc6-bcc9728f5b2b","Type":"ContainerStarted","Data":"23e93947c26c641c2d064667b015bc7de6fff3a49f2e5e70b3f3ad86b8b580f1"} Sep 17 00:48:03.663802 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:03.663247 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7705-7905b0" podUID="fab35c9d-e845-47fe-8671-31e86483ceed" containerName="test-curl-7705-7905b0" containerID="cri-o://4d188880ddf8038d9f477749046f0655d3e388d2a933725e726602842c43dd15" gracePeriod=2 Sep 17 00:48:03.676465 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:03.676418 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7705-694783" podStartSLOduration=1.676402009 podStartE2EDuration="1.676402009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:48:03.675141497 +0000 UTC m=+1096.605774984" watchObservedRunningTime="2026-09-17 00:48:03.676402009 +0000 UTC m=+1096.607035496" Sep 17 00:48:03.814112 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:03.814086 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49"] Sep 17 00:48:03.816977 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:48:03.816941 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod221c5b4f_3870_4aa3_8fa4_77a46596ffc2.slice/crio-a424bec7fc5fea802fdab511c7ea7517cc4c334ae8d54dc663be4e61068b350a WatchSource:0}: Error finding container a424bec7fc5fea802fdab511c7ea7517cc4c334ae8d54dc663be4e61068b350a: Status 404 returned error can't find the container with id a424bec7fc5fea802fdab511c7ea7517cc4c334ae8d54dc663be4e61068b350a Sep 17 00:48:04.258189 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:04.258142 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7705-694783"] Sep 17 00:48:04.260628 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:04.260594 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7705-694783"] Sep 17 00:48:04.671980 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:04.671933 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" event={"ID":"221c5b4f-3870-4aa3-8fa4-77a46596ffc2","Type":"ContainerStarted","Data":"a424bec7fc5fea802fdab511c7ea7517cc4c334ae8d54dc663be4e61068b350a"} Sep 17 00:48:04.672457 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:04.672169 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7705-694783" container="test-curl-7705-694783" Sep 17 00:48:05.678424 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:05.678387 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7705-694783" podUID="06981310-db4e-420a-9bc6-bcc9728f5b2b" containerName="test-curl-7705-694783" containerID="cri-o://f924335c3a723c94cad5134051215ac282ddbfa647970fdbe9d2cde0c4e82b0f" gracePeriod=2 Sep 17 00:48:06.246390 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:06.246366 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-7905b0" Sep 17 00:48:06.393126 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:06.393047 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gss55\" (UniqueName: \"kubernetes.io/projected/fab35c9d-e845-47fe-8671-31e86483ceed-kube-api-access-gss55\") pod \"fab35c9d-e845-47fe-8671-31e86483ceed\" (UID: \"fab35c9d-e845-47fe-8671-31e86483ceed\") " Sep 17 00:48:06.395592 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:06.395560 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab35c9d-e845-47fe-8671-31e86483ceed-kube-api-access-gss55" (OuterVolumeSpecName: "kube-api-access-gss55") pod "fab35c9d-e845-47fe-8671-31e86483ceed" (UID: "fab35c9d-e845-47fe-8671-31e86483ceed"). InnerVolumeSpecName "kube-api-access-gss55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:48:06.493935 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:06.493900 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gss55\" (UniqueName: \"kubernetes.io/projected/fab35c9d-e845-47fe-8671-31e86483ceed-kube-api-access-gss55\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:48:06.684742 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:06.684649 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" event={"ID":"221c5b4f-3870-4aa3-8fa4-77a46596ffc2","Type":"ContainerStarted","Data":"b902e5204abe5ac092eddf465baa225448285edf3d190790c77822f95360096c"} Sep 17 00:48:06.685160 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:06.684898 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" Sep 17 00:48:06.686052 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:06.686028 2582 generic.go:356] "Generic (PLEG): container finished" podID="fab35c9d-e845-47fe-8671-31e86483ceed" containerID="4d188880ddf8038d9f477749046f0655d3e388d2a933725e726602842c43dd15" exitCode=137 Sep 17 00:48:06.686171 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:06.686062 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-7905b0" Sep 17 00:48:06.686171 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:06.686116 2582 scope.go:117] "RemoveContainer" containerID="4d188880ddf8038d9f477749046f0655d3e388d2a933725e726602842c43dd15" Sep 17 00:48:06.695323 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:06.695307 2582 scope.go:117] "RemoveContainer" containerID="4d188880ddf8038d9f477749046f0655d3e388d2a933725e726602842c43dd15" Sep 17 00:48:06.695783 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:06.695759 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d188880ddf8038d9f477749046f0655d3e388d2a933725e726602842c43dd15\": container with ID starting with 4d188880ddf8038d9f477749046f0655d3e388d2a933725e726602842c43dd15 not found: ID does not exist" containerID="4d188880ddf8038d9f477749046f0655d3e388d2a933725e726602842c43dd15" Sep 17 00:48:06.695843 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:06.695791 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d188880ddf8038d9f477749046f0655d3e388d2a933725e726602842c43dd15"} err="failed to get container status \"4d188880ddf8038d9f477749046f0655d3e388d2a933725e726602842c43dd15\": rpc error: code = NotFound desc = could not find container \"4d188880ddf8038d9f477749046f0655d3e388d2a933725e726602842c43dd15\": container with ID starting with 4d188880ddf8038d9f477749046f0655d3e388d2a933725e726602842c43dd15 not found: ID does not exist" Sep 17 00:48:06.700488 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:06.700434 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7705-7905b0\" is forbidden: User \"system:node:ip-10-0-134-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-177.ec2.internal' and this object" podUID="fab35c9d-e845-47fe-8671-31e86483ceed" pod="opendatahub/test-curl-7705-7905b0" Sep 17 00:48:06.700695 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:06.700654 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" podStartSLOduration=2.3562140080000002 podStartE2EDuration="4.700640091s" totalImagesPullingTime="2.344426083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:48:06.698709446 +0000 UTC m=+1099.629342933" watchObservedRunningTime="2026-09-17 00:48:06.700640091 +0000 UTC m=+1099.631273579" Sep 17 00:48:07.648095 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:07.648053 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7705-7905b0\" is forbidden: User \"system:node:ip-10-0-134-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-177.ec2.internal' and this object" podUID="fab35c9d-e845-47fe-8671-31e86483ceed" pod="opendatahub/test-curl-7705-7905b0" Sep 17 00:48:07.650509 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:07.650478 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fab35c9d-e845-47fe-8671-31e86483ceed" path="/var/lib/kubelet/pods/fab35c9d-e845-47fe-8671-31e86483ceed/volumes" Sep 17 00:48:07.851104 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:07.851081 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-694783" Sep 17 00:48:08.008099 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:08.008009 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmb88\" (UniqueName: \"kubernetes.io/projected/06981310-db4e-420a-9bc6-bcc9728f5b2b-kube-api-access-cmb88\") pod \"06981310-db4e-420a-9bc6-bcc9728f5b2b\" (UID: \"06981310-db4e-420a-9bc6-bcc9728f5b2b\") " Sep 17 00:48:08.010321 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:08.010294 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06981310-db4e-420a-9bc6-bcc9728f5b2b-kube-api-access-cmb88" (OuterVolumeSpecName: "kube-api-access-cmb88") pod "06981310-db4e-420a-9bc6-bcc9728f5b2b" (UID: "06981310-db4e-420a-9bc6-bcc9728f5b2b"). InnerVolumeSpecName "kube-api-access-cmb88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:48:08.109150 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:08.109109 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmb88\" (UniqueName: \"kubernetes.io/projected/06981310-db4e-420a-9bc6-bcc9728f5b2b-kube-api-access-cmb88\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:48:08.698852 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:08.698821 2582 generic.go:356] "Generic (PLEG): container finished" podID="06981310-db4e-420a-9bc6-bcc9728f5b2b" containerID="f924335c3a723c94cad5134051215ac282ddbfa647970fdbe9d2cde0c4e82b0f" exitCode=137 Sep 17 00:48:08.699038 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:08.698861 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7705-694783" Sep 17 00:48:08.699038 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:08.698924 2582 scope.go:117] "RemoveContainer" containerID="f924335c3a723c94cad5134051215ac282ddbfa647970fdbe9d2cde0c4e82b0f" Sep 17 00:48:08.713601 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:08.713566 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7705-694783\" is forbidden: User \"system:node:ip-10-0-134-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-177.ec2.internal' and this object" podUID="06981310-db4e-420a-9bc6-bcc9728f5b2b" pod="opendatahub/test-curl-7705-694783" Sep 17 00:48:08.716687 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:08.716578 2582 scope.go:117] "RemoveContainer" containerID="f924335c3a723c94cad5134051215ac282ddbfa647970fdbe9d2cde0c4e82b0f" Sep 17 00:48:08.717012 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:08.716991 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f924335c3a723c94cad5134051215ac282ddbfa647970fdbe9d2cde0c4e82b0f\": container with ID starting with f924335c3a723c94cad5134051215ac282ddbfa647970fdbe9d2cde0c4e82b0f not found: ID does not exist" containerID="f924335c3a723c94cad5134051215ac282ddbfa647970fdbe9d2cde0c4e82b0f" Sep 17 00:48:08.717071 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:08.717022 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f924335c3a723c94cad5134051215ac282ddbfa647970fdbe9d2cde0c4e82b0f"} err="failed to get container status \"f924335c3a723c94cad5134051215ac282ddbfa647970fdbe9d2cde0c4e82b0f\": rpc error: code = NotFound desc = could not find container \"f924335c3a723c94cad5134051215ac282ddbfa647970fdbe9d2cde0c4e82b0f\": container with ID starting with f924335c3a723c94cad5134051215ac282ddbfa647970fdbe9d2cde0c4e82b0f not found: ID does not exist" Sep 17 00:48:09.651447 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:09.651398 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06981310-db4e-420a-9bc6-bcc9728f5b2b" path="/var/lib/kubelet/pods/06981310-db4e-420a-9bc6-bcc9728f5b2b/volumes" Sep 17 00:48:10.058567 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.058529 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566"] Sep 17 00:48:10.059253 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.059220 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06981310-db4e-420a-9bc6-bcc9728f5b2b" containerName="test-curl-7705-694783" Sep 17 00:48:10.059306 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.059258 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="06981310-db4e-420a-9bc6-bcc9728f5b2b" containerName="test-curl-7705-694783" Sep 17 00:48:10.059411 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.059398 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="06981310-db4e-420a-9bc6-bcc9728f5b2b" containerName="test-curl-7705-694783" Sep 17 00:48:10.063341 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.063314 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" Sep 17 00:48:10.065450 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.065427 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8fd93c3d-serving-cert\"" Sep 17 00:48:10.073536 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.073487 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566"] Sep 17 00:48:10.127907 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.127872 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj56w\" (UniqueName: \"kubernetes.io/projected/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-kube-api-access-fj56w\") pod \"maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566\" (UID: \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" Sep 17 00:48:10.128107 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.127991 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-maas-api-tls\") pod \"maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566\" (UID: \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" Sep 17 00:48:10.128107 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.128074 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-metrics-certs\") pod \"maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566\" (UID: \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" Sep 17 00:48:10.128215 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.128111 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-openshift-service-ca\") pod \"maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566\" (UID: \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" Sep 17 00:48:10.229141 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.229109 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-maas-api-tls\") pod \"maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566\" (UID: \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" Sep 17 00:48:10.229141 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.229158 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-metrics-certs\") pod \"maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566\" (UID: \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" Sep 17 00:48:10.229393 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.229175 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-openshift-service-ca\") pod \"maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566\" (UID: \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" Sep 17 00:48:10.229393 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:10.229280 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-serving-cert: secret "maas-api-e2e-ait-8fd93c3d-serving-cert" not found Sep 17 00:48:10.229393 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.229303 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj56w\" (UniqueName: \"kubernetes.io/projected/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-kube-api-access-fj56w\") pod \"maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566\" (UID: \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" Sep 17 00:48:10.229393 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:10.229356 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-maas-api-tls podName:9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7 nodeName:}" failed. No retries permitted until 2026-09-17 00:48:10.729333478 +0000 UTC m=+1103.659966965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-maas-api-tls") pod "maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" (UID: "9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7") : secret "maas-api-e2e-ait-8fd93c3d-serving-cert" not found Sep 17 00:48:10.229845 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.229825 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-openshift-service-ca\") pod \"maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566\" (UID: \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" Sep 17 00:48:10.232012 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.231985 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-metrics-certs\") pod \"maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566\" (UID: \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" Sep 17 00:48:10.237541 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.237492 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj56w\" (UniqueName: \"kubernetes.io/projected/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-kube-api-access-fj56w\") pod \"maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566\" (UID: \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" Sep 17 00:48:10.735085 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.735033 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-maas-api-tls\") pod \"maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566\" (UID: \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" Sep 17 00:48:10.737623 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.737597 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-maas-api-tls\") pod \"maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566\" (UID: \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" Sep 17 00:48:10.978021 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:10.977984 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" Sep 17 00:48:11.237211 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.237128 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5"] Sep 17 00:48:11.242699 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.242674 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" Sep 17 00:48:11.244731 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.244706 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-6af38a-serving-cert\"" Sep 17 00:48:11.251871 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.251794 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5"] Sep 17 00:48:11.254835 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.254790 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566"] Sep 17 00:48:11.257072 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:48:11.257025 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fff58ab_6af9_4ef8_94d0_cb7ed626eaa7.slice/crio-8be1ef10fac09b101af79a9a49ae84d12399461c4e2b7f119d2fc7f7750b1020 WatchSource:0}: Error finding container 8be1ef10fac09b101af79a9a49ae84d12399461c4e2b7f119d2fc7f7750b1020: Status 404 returned error can't find the container with id 8be1ef10fac09b101af79a9a49ae84d12399461c4e2b7f119d2fc7f7750b1020 Sep 17 00:48:11.340915 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.340880 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc785bad-fc9b-4143-8f4a-e773b51153f8-maas-api-tls\") pod \"maas-api-e2e-autores-6af38a-5d748b648d-8n4g5\" (UID: \"dc785bad-fc9b-4143-8f4a-e773b51153f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" Sep 17 00:48:11.340915 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.340916 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc785bad-fc9b-4143-8f4a-e773b51153f8-metrics-certs\") pod \"maas-api-e2e-autores-6af38a-5d748b648d-8n4g5\" (UID: \"dc785bad-fc9b-4143-8f4a-e773b51153f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" Sep 17 00:48:11.341132 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.340945 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc785bad-fc9b-4143-8f4a-e773b51153f8-openshift-service-ca\") pod \"maas-api-e2e-autores-6af38a-5d748b648d-8n4g5\" (UID: \"dc785bad-fc9b-4143-8f4a-e773b51153f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" Sep 17 00:48:11.341170 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.341135 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk4rm\" (UniqueName: \"kubernetes.io/projected/dc785bad-fc9b-4143-8f4a-e773b51153f8-kube-api-access-jk4rm\") pod \"maas-api-e2e-autores-6af38a-5d748b648d-8n4g5\" (UID: \"dc785bad-fc9b-4143-8f4a-e773b51153f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" Sep 17 00:48:11.442440 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.442400 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk4rm\" (UniqueName: \"kubernetes.io/projected/dc785bad-fc9b-4143-8f4a-e773b51153f8-kube-api-access-jk4rm\") pod \"maas-api-e2e-autores-6af38a-5d748b648d-8n4g5\" (UID: \"dc785bad-fc9b-4143-8f4a-e773b51153f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" Sep 17 00:48:11.442634 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.442450 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc785bad-fc9b-4143-8f4a-e773b51153f8-maas-api-tls\") pod \"maas-api-e2e-autores-6af38a-5d748b648d-8n4g5\" (UID: \"dc785bad-fc9b-4143-8f4a-e773b51153f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" Sep 17 00:48:11.442634 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.442474 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc785bad-fc9b-4143-8f4a-e773b51153f8-metrics-certs\") pod \"maas-api-e2e-autores-6af38a-5d748b648d-8n4g5\" (UID: \"dc785bad-fc9b-4143-8f4a-e773b51153f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" Sep 17 00:48:11.442634 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.442533 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc785bad-fc9b-4143-8f4a-e773b51153f8-openshift-service-ca\") pod \"maas-api-e2e-autores-6af38a-5d748b648d-8n4g5\" (UID: \"dc785bad-fc9b-4143-8f4a-e773b51153f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" Sep 17 00:48:11.442634 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:11.442599 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-serving-cert: secret "maas-api-e2e-autores-6af38a-serving-cert" not found Sep 17 00:48:11.442786 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:11.442681 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc785bad-fc9b-4143-8f4a-e773b51153f8-maas-api-tls podName:dc785bad-fc9b-4143-8f4a-e773b51153f8 nodeName:}" failed. No retries permitted until 2026-09-17 00:48:11.942658438 +0000 UTC m=+1104.873291918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc785bad-fc9b-4143-8f4a-e773b51153f8-maas-api-tls") pod "maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" (UID: "dc785bad-fc9b-4143-8f4a-e773b51153f8") : secret "maas-api-e2e-autores-6af38a-serving-cert" not found Sep 17 00:48:11.443249 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.443232 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc785bad-fc9b-4143-8f4a-e773b51153f8-openshift-service-ca\") pod \"maas-api-e2e-autores-6af38a-5d748b648d-8n4g5\" (UID: \"dc785bad-fc9b-4143-8f4a-e773b51153f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" Sep 17 00:48:11.445135 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.445117 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc785bad-fc9b-4143-8f4a-e773b51153f8-metrics-certs\") pod \"maas-api-e2e-autores-6af38a-5d748b648d-8n4g5\" (UID: \"dc785bad-fc9b-4143-8f4a-e773b51153f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" Sep 17 00:48:11.451869 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.451846 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk4rm\" (UniqueName: \"kubernetes.io/projected/dc785bad-fc9b-4143-8f4a-e773b51153f8-kube-api-access-jk4rm\") pod \"maas-api-e2e-autores-6af38a-5d748b648d-8n4g5\" (UID: \"dc785bad-fc9b-4143-8f4a-e773b51153f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" Sep 17 00:48:11.718302 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.718255 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" event={"ID":"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7","Type":"ContainerStarted","Data":"861a11f080c6d6efc278cee84dae61994b73e03c89e527b04a914db3bddb481e"} Sep 17 00:48:11.718412 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.718329 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" Sep 17 00:48:11.718412 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.718351 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" event={"ID":"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7","Type":"ContainerStarted","Data":"8be1ef10fac09b101af79a9a49ae84d12399461c4e2b7f119d2fc7f7750b1020"} Sep 17 00:48:11.733810 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.733759 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" podStartSLOduration=1.347206247 podStartE2EDuration="1.733740808s" totalImagesPullingTime="386.534561ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:48:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:48:11.732226203 +0000 UTC m=+1104.662859724" watchObservedRunningTime="2026-09-17 00:48:11.733740808 +0000 UTC m=+1104.664374294" Sep 17 00:48:11.948320 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:11.948236 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc785bad-fc9b-4143-8f4a-e773b51153f8-maas-api-tls\") pod \"maas-api-e2e-autores-6af38a-5d748b648d-8n4g5\" (UID: \"dc785bad-fc9b-4143-8f4a-e773b51153f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" Sep 17 00:48:11.948784 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:11.948366 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-serving-cert: secret "maas-api-e2e-autores-6af38a-serving-cert" not found Sep 17 00:48:11.948784 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:11.948418 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc785bad-fc9b-4143-8f4a-e773b51153f8-maas-api-tls podName:dc785bad-fc9b-4143-8f4a-e773b51153f8 nodeName:}" failed. No retries permitted until 2026-09-17 00:48:12.94840365 +0000 UTC m=+1105.879037128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc785bad-fc9b-4143-8f4a-e773b51153f8-maas-api-tls") pod "maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" (UID: "dc785bad-fc9b-4143-8f4a-e773b51153f8") : secret "maas-api-e2e-autores-6af38a-serving-cert" not found Sep 17 00:48:12.698638 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:12.698604 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" Sep 17 00:48:12.958282 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:12.958172 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc785bad-fc9b-4143-8f4a-e773b51153f8-maas-api-tls\") pod \"maas-api-e2e-autores-6af38a-5d748b648d-8n4g5\" (UID: \"dc785bad-fc9b-4143-8f4a-e773b51153f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" Sep 17 00:48:12.960812 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:12.960784 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc785bad-fc9b-4143-8f4a-e773b51153f8-maas-api-tls\") pod \"maas-api-e2e-autores-6af38a-5d748b648d-8n4g5\" (UID: \"dc785bad-fc9b-4143-8f4a-e773b51153f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" Sep 17 00:48:13.059227 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:13.059191 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" Sep 17 00:48:13.329808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:13.329774 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5"] Sep 17 00:48:13.332874 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:48:13.332844 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc785bad_fc9b_4143_8f4a_e773b51153f8.slice/crio-e98c64f29b88aad26d3832362af1b29c76ffe401f25c1771575de854ee50b59c WatchSource:0}: Error finding container e98c64f29b88aad26d3832362af1b29c76ffe401f25c1771575de854ee50b59c: Status 404 returned error can't find the container with id e98c64f29b88aad26d3832362af1b29c76ffe401f25c1771575de854ee50b59c Sep 17 00:48:13.730864 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:13.730837 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" event={"ID":"dc785bad-fc9b-4143-8f4a-e773b51153f8","Type":"ContainerStarted","Data":"e98c64f29b88aad26d3832362af1b29c76ffe401f25c1771575de854ee50b59c"} Sep 17 00:48:14.741657 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:14.741622 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" event={"ID":"dc785bad-fc9b-4143-8f4a-e773b51153f8","Type":"ContainerStarted","Data":"b209cf764e8d60b6d1c1610f8beba7404c3b843b0fbc0e59279f8cb3ccf00785"} Sep 17 00:48:14.742160 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:14.741748 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" Sep 17 00:48:14.757971 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:14.757912 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" podStartSLOduration=3.432203078 podStartE2EDuration="3.757891971s" totalImagesPullingTime="325.688893ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:48:14.757278563 +0000 UTC m=+1107.687912053" watchObservedRunningTime="2026-09-17 00:48:14.757891971 +0000 UTC m=+1107.688525452" Sep 17 00:48:17.729826 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:17.729793 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" Sep 17 00:48:20.423383 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:20.423347 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:48:20.754778 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:20.754697 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" Sep 17 00:48:26.661164 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:26.661130 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz"] Sep 17 00:48:26.669184 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:26.669160 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" Sep 17 00:48:26.671101 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:26.671073 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-e421ba-serving-cert\"" Sep 17 00:48:26.673749 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:26.673276 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz"] Sep 17 00:48:26.800558 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:26.800489 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9m6p\" (UniqueName: \"kubernetes.io/projected/9b612ece-784b-4bff-942f-0c79d035e84c-kube-api-access-v9m6p\") pod \"maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz\" (UID: \"9b612ece-784b-4bff-942f-0c79d035e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" Sep 17 00:48:26.800733 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:26.800629 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b612ece-784b-4bff-942f-0c79d035e84c-metrics-certs\") pod \"maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz\" (UID: \"9b612ece-784b-4bff-942f-0c79d035e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" Sep 17 00:48:26.800733 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:26.800720 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b612ece-784b-4bff-942f-0c79d035e84c-maas-api-tls\") pod \"maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz\" (UID: \"9b612ece-784b-4bff-942f-0c79d035e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" Sep 17 00:48:26.800852 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:26.800760 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b612ece-784b-4bff-942f-0c79d035e84c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz\" (UID: \"9b612ece-784b-4bff-942f-0c79d035e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" Sep 17 00:48:26.901858 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:26.901812 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9m6p\" (UniqueName: \"kubernetes.io/projected/9b612ece-784b-4bff-942f-0c79d035e84c-kube-api-access-v9m6p\") pod \"maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz\" (UID: \"9b612ece-784b-4bff-942f-0c79d035e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" Sep 17 00:48:26.902066 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:26.901902 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b612ece-784b-4bff-942f-0c79d035e84c-metrics-certs\") pod \"maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz\" (UID: \"9b612ece-784b-4bff-942f-0c79d035e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" Sep 17 00:48:26.902066 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:26.901985 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b612ece-784b-4bff-942f-0c79d035e84c-maas-api-tls\") pod \"maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz\" (UID: \"9b612ece-784b-4bff-942f-0c79d035e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" Sep 17 00:48:26.902066 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:26.902034 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b612ece-784b-4bff-942f-0c79d035e84c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz\" (UID: \"9b612ece-784b-4bff-942f-0c79d035e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" Sep 17 00:48:26.902239 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:26.902122 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-serving-cert: secret "maas-api-e2e-inf-b-e421ba-serving-cert" not found Sep 17 00:48:26.902239 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:26.902199 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b612ece-784b-4bff-942f-0c79d035e84c-maas-api-tls podName:9b612ece-784b-4bff-942f-0c79d035e84c nodeName:}" failed. No retries permitted until 2026-09-17 00:48:27.40218332 +0000 UTC m=+1120.332816786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b612ece-784b-4bff-942f-0c79d035e84c-maas-api-tls") pod "maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" (UID: "9b612ece-784b-4bff-942f-0c79d035e84c") : secret "maas-api-e2e-inf-b-e421ba-serving-cert" not found Sep 17 00:48:26.902747 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:26.902724 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b612ece-784b-4bff-942f-0c79d035e84c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz\" (UID: \"9b612ece-784b-4bff-942f-0c79d035e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" Sep 17 00:48:26.904461 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:26.904438 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b612ece-784b-4bff-942f-0c79d035e84c-metrics-certs\") pod \"maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz\" (UID: \"9b612ece-784b-4bff-942f-0c79d035e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" Sep 17 00:48:26.909379 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:26.909351 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9m6p\" (UniqueName: \"kubernetes.io/projected/9b612ece-784b-4bff-942f-0c79d035e84c-kube-api-access-v9m6p\") pod \"maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz\" (UID: \"9b612ece-784b-4bff-942f-0c79d035e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" Sep 17 00:48:27.407695 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:27.407654 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b612ece-784b-4bff-942f-0c79d035e84c-maas-api-tls\") pod \"maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz\" (UID: \"9b612ece-784b-4bff-942f-0c79d035e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" Sep 17 00:48:27.407859 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:27.407807 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-serving-cert: secret "maas-api-e2e-inf-b-e421ba-serving-cert" not found Sep 17 00:48:27.407902 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:27.407882 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b612ece-784b-4bff-942f-0c79d035e84c-maas-api-tls podName:9b612ece-784b-4bff-942f-0c79d035e84c nodeName:}" failed. No retries permitted until 2026-09-17 00:48:28.407866084 +0000 UTC m=+1121.338499549 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b612ece-784b-4bff-942f-0c79d035e84c-maas-api-tls") pod "maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" (UID: "9b612ece-784b-4bff-942f-0c79d035e84c") : secret "maas-api-e2e-inf-b-e421ba-serving-cert" not found Sep 17 00:48:28.417019 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:28.416968 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b612ece-784b-4bff-942f-0c79d035e84c-maas-api-tls\") pod \"maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz\" (UID: \"9b612ece-784b-4bff-942f-0c79d035e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" Sep 17 00:48:28.419508 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:28.419482 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b612ece-784b-4bff-942f-0c79d035e84c-maas-api-tls\") pod \"maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz\" (UID: \"9b612ece-784b-4bff-942f-0c79d035e84c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" Sep 17 00:48:28.483352 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:28.483316 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" Sep 17 00:48:28.758627 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:28.758546 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz"] Sep 17 00:48:28.762206 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:48:28.762181 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b612ece_784b_4bff_942f_0c79d035e84c.slice/crio-6224151ca95e0aab7708c94dc26204512e88cfe6575a04657d290907d3ea6851 WatchSource:0}: Error finding container 6224151ca95e0aab7708c94dc26204512e88cfe6575a04657d290907d3ea6851: Status 404 returned error can't find the container with id 6224151ca95e0aab7708c94dc26204512e88cfe6575a04657d290907d3ea6851 Sep 17 00:48:28.824780 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:28.824742 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" event={"ID":"9b612ece-784b-4bff-942f-0c79d035e84c","Type":"ContainerStarted","Data":"6224151ca95e0aab7708c94dc26204512e88cfe6575a04657d290907d3ea6851"} Sep 17 00:48:29.379784 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.379747 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n"] Sep 17 00:48:29.385181 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.385159 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:29.387239 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.387218 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a79f56\"/\"kube-root-ca.crt\"" Sep 17 00:48:29.387866 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.387850 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a79f56\"/\"openshift-service-ca.crt\"" Sep 17 00:48:29.387960 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.387860 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a79f56\"/\"default-dockercfg-lqgbh\"" Sep 17 00:48:29.388069 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.387983 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a79f56\"/\"simulator-w0-a79f56-kserve-self-signed-certs\"" Sep 17 00:48:29.391745 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.391723 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n"] Sep 17 00:48:29.528366 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.528331 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-model-cache\") pod \"simulator-w0-a79f56-kserve-5cbc975465-svz5n\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:29.528366 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.528368 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-dshm\") pod \"simulator-w0-a79f56-kserve-5cbc975465-svz5n\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:29.528783 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.528407 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s9mg\" (UniqueName: \"kubernetes.io/projected/b2c18e33-b3f4-4197-9dab-9465b8327206-kube-api-access-8s9mg\") pod \"simulator-w0-a79f56-kserve-5cbc975465-svz5n\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:29.528783 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.528458 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-home\") pod \"simulator-w0-a79f56-kserve-5cbc975465-svz5n\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:29.528783 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.528528 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c18e33-b3f4-4197-9dab-9465b8327206-tls-certs\") pod \"simulator-w0-a79f56-kserve-5cbc975465-svz5n\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:29.528783 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.528561 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-tmp-dir\") pod \"simulator-w0-a79f56-kserve-5cbc975465-svz5n\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:29.629663 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.629565 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c18e33-b3f4-4197-9dab-9465b8327206-tls-certs\") pod \"simulator-w0-a79f56-kserve-5cbc975465-svz5n\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:29.629663 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.629646 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-tmp-dir\") pod \"simulator-w0-a79f56-kserve-5cbc975465-svz5n\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:29.629895 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.629703 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-model-cache\") pod \"simulator-w0-a79f56-kserve-5cbc975465-svz5n\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:29.629895 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.629727 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-dshm\") pod \"simulator-w0-a79f56-kserve-5cbc975465-svz5n\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:29.629895 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.629791 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s9mg\" (UniqueName: \"kubernetes.io/projected/b2c18e33-b3f4-4197-9dab-9465b8327206-kube-api-access-8s9mg\") pod \"simulator-w0-a79f56-kserve-5cbc975465-svz5n\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:29.629895 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.629847 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-home\") pod \"simulator-w0-a79f56-kserve-5cbc975465-svz5n\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:29.630105 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.630078 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-tmp-dir\") pod \"simulator-w0-a79f56-kserve-5cbc975465-svz5n\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:29.630178 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.630153 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-model-cache\") pod \"simulator-w0-a79f56-kserve-5cbc975465-svz5n\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:29.630408 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.630386 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-home\") pod \"simulator-w0-a79f56-kserve-5cbc975465-svz5n\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:29.632239 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.632209 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-dshm\") pod \"simulator-w0-a79f56-kserve-5cbc975465-svz5n\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:29.632614 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.632595 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c18e33-b3f4-4197-9dab-9465b8327206-tls-certs\") pod \"simulator-w0-a79f56-kserve-5cbc975465-svz5n\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:29.637276 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.637248 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s9mg\" (UniqueName: \"kubernetes.io/projected/b2c18e33-b3f4-4197-9dab-9465b8327206-kube-api-access-8s9mg\") pod \"simulator-w0-a79f56-kserve-5cbc975465-svz5n\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:29.696269 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.696232 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:29.832553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.832522 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" event={"ID":"9b612ece-784b-4bff-942f-0c79d035e84c","Type":"ContainerStarted","Data":"3eb5f561939e2ce675e2d5ac558bd542d4ae5c173363630c1122e026ac9f319a"} Sep 17 00:48:29.832725 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.832615 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" Sep 17 00:48:29.848163 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.848094 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" podStartSLOduration=3.322842826 podStartE2EDuration="3.848080412s" totalImagesPullingTime="525.237586ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:48:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:48:29.847142134 +0000 UTC m=+1122.777775620" watchObservedRunningTime="2026-09-17 00:48:29.848080412 +0000 UTC m=+1122.778713904" Sep 17 00:48:29.975178 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:29.975147 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n"] Sep 17 00:48:29.977203 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:48:29.977171 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2c18e33_b3f4_4197_9dab_9465b8327206.slice/crio-7fad9f99a3d441acd75750c7cfbe3413cb973431b1843a2219f57ebfd6e6e9f9 WatchSource:0}: Error finding container 7fad9f99a3d441acd75750c7cfbe3413cb973431b1843a2219f57ebfd6e6e9f9: Status 404 returned error can't find the container with id 7fad9f99a3d441acd75750c7cfbe3413cb973431b1843a2219f57ebfd6e6e9f9 Sep 17 00:48:30.839915 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:30.839875 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" event={"ID":"b2c18e33-b3f4-4197-9dab-9465b8327206","Type":"ContainerStarted","Data":"69ef9b6d0ab87a822969a25b9cc05c23ea6e802db5bb7c8ae63cf1d30b41f97e"} Sep 17 00:48:30.839915 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:30.839924 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" event={"ID":"b2c18e33-b3f4-4197-9dab-9465b8327206","Type":"ContainerStarted","Data":"7fad9f99a3d441acd75750c7cfbe3413cb973431b1843a2219f57ebfd6e6e9f9"} Sep 17 00:48:30.856675 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:30.856625 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" podStartSLOduration=1.856609331 podStartE2EDuration="1.856609331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:48:30.854285881 +0000 UTC m=+1123.784919367" watchObservedRunningTime="2026-09-17 00:48:30.856609331 +0000 UTC m=+1123.787242818" Sep 17 00:48:32.776114 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:32.776070 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566"] Sep 17 00:48:32.776642 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:32.776265 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" podUID="9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7" containerName="maas-api" containerID="cri-o://861a11f080c6d6efc278cee84dae61994b73e03c89e527b04a914db3bddb481e" gracePeriod=30 Sep 17 00:48:33.035729 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.035653 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" Sep 17 00:48:33.164094 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.164062 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj56w\" (UniqueName: \"kubernetes.io/projected/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-kube-api-access-fj56w\") pod \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\" (UID: \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\") " Sep 17 00:48:33.164238 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.164111 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-maas-api-tls\") pod \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\" (UID: \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\") " Sep 17 00:48:33.164238 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.164162 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-metrics-certs\") pod \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\" (UID: \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\") " Sep 17 00:48:33.164384 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.164240 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-openshift-service-ca\") pod \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\" (UID: \"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7\") " Sep 17 00:48:33.164645 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.164616 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7" (UID: "9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 00:48:33.166523 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.166469 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-kube-api-access-fj56w" (OuterVolumeSpecName: "kube-api-access-fj56w") pod "9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7" (UID: "9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7"). InnerVolumeSpecName "kube-api-access-fj56w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:48:33.166523 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.166475 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7" (UID: "9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:48:33.166686 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.166536 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7" (UID: "9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:48:33.264982 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.264950 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-openshift-service-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:48:33.264982 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.264976 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj56w\" (UniqueName: \"kubernetes.io/projected/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-kube-api-access-fj56w\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:48:33.264982 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.264986 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-maas-api-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:48:33.265201 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.264994 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7-metrics-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:48:33.860998 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.860966 2582 generic.go:356] "Generic (PLEG): container finished" podID="9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7" containerID="861a11f080c6d6efc278cee84dae61994b73e03c89e527b04a914db3bddb481e" exitCode=0 Sep 17 00:48:33.861467 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.861017 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" Sep 17 00:48:33.861467 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.861050 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" event={"ID":"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7","Type":"ContainerDied","Data":"861a11f080c6d6efc278cee84dae61994b73e03c89e527b04a914db3bddb481e"} Sep 17 00:48:33.861467 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.861107 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566" event={"ID":"9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7","Type":"ContainerDied","Data":"8be1ef10fac09b101af79a9a49ae84d12399461c4e2b7f119d2fc7f7750b1020"} Sep 17 00:48:33.861467 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.861131 2582 scope.go:117] "RemoveContainer" containerID="861a11f080c6d6efc278cee84dae61994b73e03c89e527b04a914db3bddb481e" Sep 17 00:48:33.870304 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.870285 2582 scope.go:117] "RemoveContainer" containerID="861a11f080c6d6efc278cee84dae61994b73e03c89e527b04a914db3bddb481e" Sep 17 00:48:33.870618 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:33.870599 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"861a11f080c6d6efc278cee84dae61994b73e03c89e527b04a914db3bddb481e\": container with ID starting with 861a11f080c6d6efc278cee84dae61994b73e03c89e527b04a914db3bddb481e not found: ID does not exist" containerID="861a11f080c6d6efc278cee84dae61994b73e03c89e527b04a914db3bddb481e" Sep 17 00:48:33.870688 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.870627 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"861a11f080c6d6efc278cee84dae61994b73e03c89e527b04a914db3bddb481e"} err="failed to get container status \"861a11f080c6d6efc278cee84dae61994b73e03c89e527b04a914db3bddb481e\": rpc error: code = NotFound desc = could not find container \"861a11f080c6d6efc278cee84dae61994b73e03c89e527b04a914db3bddb481e\": container with ID starting with 861a11f080c6d6efc278cee84dae61994b73e03c89e527b04a914db3bddb481e not found: ID does not exist" Sep 17 00:48:33.876772 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.876747 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566"] Sep 17 00:48:33.884062 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:33.884040 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8fd93c3d-6d678748f8-qj566"] Sep 17 00:48:35.650778 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:35.650749 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7" path="/var/lib/kubelet/pods/9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7/volumes" Sep 17 00:48:35.845416 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:35.845380 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" Sep 17 00:48:38.835163 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:38.835129 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj"] Sep 17 00:48:38.835881 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:38.835860 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7" containerName="maas-api" Sep 17 00:48:38.835951 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:38.835885 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7" containerName="maas-api" Sep 17 00:48:38.836007 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:38.835995 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fff58ab-6af9-4ef8-94d0-cb7ed626eaa7" containerName="maas-api" Sep 17 00:48:38.840800 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:38.840772 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" Sep 17 00:48:38.842875 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:38.842852 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-ef36871b-serving-cert\"" Sep 17 00:48:38.846916 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:38.846882 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj"] Sep 17 00:48:38.921691 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:38.921659 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4acc9329-a231-4f8e-bcdf-25580f5390ee-openshift-service-ca\") pod \"maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj\" (UID: \"4acc9329-a231-4f8e-bcdf-25580f5390ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" Sep 17 00:48:38.921893 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:38.921709 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f87w6\" (UniqueName: \"kubernetes.io/projected/4acc9329-a231-4f8e-bcdf-25580f5390ee-kube-api-access-f87w6\") pod \"maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj\" (UID: \"4acc9329-a231-4f8e-bcdf-25580f5390ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" Sep 17 00:48:38.921893 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:38.921809 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4acc9329-a231-4f8e-bcdf-25580f5390ee-maas-api-tls\") pod \"maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj\" (UID: \"4acc9329-a231-4f8e-bcdf-25580f5390ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" Sep 17 00:48:38.921893 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:38.921857 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4acc9329-a231-4f8e-bcdf-25580f5390ee-metrics-certs\") pod \"maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj\" (UID: \"4acc9329-a231-4f8e-bcdf-25580f5390ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" Sep 17 00:48:39.023244 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:39.023191 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4acc9329-a231-4f8e-bcdf-25580f5390ee-openshift-service-ca\") pod \"maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj\" (UID: \"4acc9329-a231-4f8e-bcdf-25580f5390ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" Sep 17 00:48:39.023455 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:39.023252 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87w6\" (UniqueName: \"kubernetes.io/projected/4acc9329-a231-4f8e-bcdf-25580f5390ee-kube-api-access-f87w6\") pod \"maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj\" (UID: \"4acc9329-a231-4f8e-bcdf-25580f5390ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" Sep 17 00:48:39.023455 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:39.023320 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4acc9329-a231-4f8e-bcdf-25580f5390ee-maas-api-tls\") pod \"maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj\" (UID: \"4acc9329-a231-4f8e-bcdf-25580f5390ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" Sep 17 00:48:39.023455 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:39.023360 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4acc9329-a231-4f8e-bcdf-25580f5390ee-metrics-certs\") pod \"maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj\" (UID: \"4acc9329-a231-4f8e-bcdf-25580f5390ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" Sep 17 00:48:39.023702 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:39.023475 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-serving-cert: secret "maas-api-e2e-migrate-ef36871b-serving-cert" not found Sep 17 00:48:39.023702 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:48:39.023587 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4acc9329-a231-4f8e-bcdf-25580f5390ee-maas-api-tls podName:4acc9329-a231-4f8e-bcdf-25580f5390ee nodeName:}" failed. No retries permitted until 2026-09-17 00:48:39.523569382 +0000 UTC m=+1132.454202847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4acc9329-a231-4f8e-bcdf-25580f5390ee-maas-api-tls") pod "maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" (UID: "4acc9329-a231-4f8e-bcdf-25580f5390ee") : secret "maas-api-e2e-migrate-ef36871b-serving-cert" not found Sep 17 00:48:39.023969 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:39.023950 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4acc9329-a231-4f8e-bcdf-25580f5390ee-openshift-service-ca\") pod \"maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj\" (UID: \"4acc9329-a231-4f8e-bcdf-25580f5390ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" Sep 17 00:48:39.025948 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:39.025925 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4acc9329-a231-4f8e-bcdf-25580f5390ee-metrics-certs\") pod \"maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj\" (UID: \"4acc9329-a231-4f8e-bcdf-25580f5390ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" Sep 17 00:48:39.033143 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:39.033118 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f87w6\" (UniqueName: \"kubernetes.io/projected/4acc9329-a231-4f8e-bcdf-25580f5390ee-kube-api-access-f87w6\") pod \"maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj\" (UID: \"4acc9329-a231-4f8e-bcdf-25580f5390ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" Sep 17 00:48:39.527764 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:39.527719 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4acc9329-a231-4f8e-bcdf-25580f5390ee-maas-api-tls\") pod \"maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj\" (UID: \"4acc9329-a231-4f8e-bcdf-25580f5390ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" Sep 17 00:48:39.530354 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:39.530327 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4acc9329-a231-4f8e-bcdf-25580f5390ee-maas-api-tls\") pod \"maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj\" (UID: \"4acc9329-a231-4f8e-bcdf-25580f5390ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" Sep 17 00:48:39.696447 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:39.696412 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:39.696676 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:39.696565 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:39.708971 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:39.708949 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:39.756285 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:39.756243 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" Sep 17 00:48:39.913857 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:39.913821 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:48:40.029142 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:40.029113 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj"] Sep 17 00:48:40.031162 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:48:40.031136 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4acc9329_a231_4f8e_bcdf_25580f5390ee.slice/crio-610e194863ac20c5b9f3dc35a8b868bd9e5a74e5d397ce804e1ec6c731eb63a4 WatchSource:0}: Error finding container 610e194863ac20c5b9f3dc35a8b868bd9e5a74e5d397ce804e1ec6c731eb63a4: Status 404 returned error can't find the container with id 610e194863ac20c5b9f3dc35a8b868bd9e5a74e5d397ce804e1ec6c731eb63a4 Sep 17 00:48:40.909010 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:40.908964 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" event={"ID":"4acc9329-a231-4f8e-bcdf-25580f5390ee","Type":"ContainerStarted","Data":"f9e1585a8e0d0a129c2eaba4b906d285f713bfb07431694d5ba0c6f5157d69f9"} Sep 17 00:48:40.909494 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:40.909202 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" Sep 17 00:48:40.909494 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:40.909238 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" event={"ID":"4acc9329-a231-4f8e-bcdf-25580f5390ee","Type":"ContainerStarted","Data":"610e194863ac20c5b9f3dc35a8b868bd9e5a74e5d397ce804e1ec6c731eb63a4"} Sep 17 00:48:40.924666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:40.924621 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" podStartSLOduration=2.585042979 podStartE2EDuration="2.924606214s" totalImagesPullingTime="339.563235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:48:40.923009292 +0000 UTC m=+1133.853642778" watchObservedRunningTime="2026-09-17 00:48:40.924606214 +0000 UTC m=+1133.855239701" Sep 17 00:48:46.920033 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:46.920001 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" Sep 17 00:48:50.494236 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:48:50.494202 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:49:08.675653 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:08.675562 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj"] Sep 17 00:49:08.676221 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:08.675858 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" podUID="4acc9329-a231-4f8e-bcdf-25580f5390ee" containerName="maas-api" containerID="cri-o://f9e1585a8e0d0a129c2eaba4b906d285f713bfb07431694d5ba0c6f5157d69f9" gracePeriod=30 Sep 17 00:49:08.915923 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:08.915902 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" Sep 17 00:49:09.011572 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.010808 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4acc9329-a231-4f8e-bcdf-25580f5390ee-metrics-certs\") pod \"4acc9329-a231-4f8e-bcdf-25580f5390ee\" (UID: \"4acc9329-a231-4f8e-bcdf-25580f5390ee\") " Sep 17 00:49:09.011572 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.010902 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4acc9329-a231-4f8e-bcdf-25580f5390ee-maas-api-tls\") pod \"4acc9329-a231-4f8e-bcdf-25580f5390ee\" (UID: \"4acc9329-a231-4f8e-bcdf-25580f5390ee\") " Sep 17 00:49:09.011572 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.011000 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f87w6\" (UniqueName: \"kubernetes.io/projected/4acc9329-a231-4f8e-bcdf-25580f5390ee-kube-api-access-f87w6\") pod \"4acc9329-a231-4f8e-bcdf-25580f5390ee\" (UID: \"4acc9329-a231-4f8e-bcdf-25580f5390ee\") " Sep 17 00:49:09.011572 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.011037 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4acc9329-a231-4f8e-bcdf-25580f5390ee-openshift-service-ca\") pod \"4acc9329-a231-4f8e-bcdf-25580f5390ee\" (UID: \"4acc9329-a231-4f8e-bcdf-25580f5390ee\") " Sep 17 00:49:09.011958 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.011805 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4acc9329-a231-4f8e-bcdf-25580f5390ee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4acc9329-a231-4f8e-bcdf-25580f5390ee" (UID: "4acc9329-a231-4f8e-bcdf-25580f5390ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 00:49:09.019762 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.015860 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4acc9329-a231-4f8e-bcdf-25580f5390ee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4acc9329-a231-4f8e-bcdf-25580f5390ee" (UID: "4acc9329-a231-4f8e-bcdf-25580f5390ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:49:09.023694 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.023652 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4acc9329-a231-4f8e-bcdf-25580f5390ee-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4acc9329-a231-4f8e-bcdf-25580f5390ee" (UID: "4acc9329-a231-4f8e-bcdf-25580f5390ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:49:09.023987 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.023958 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4acc9329-a231-4f8e-bcdf-25580f5390ee-kube-api-access-f87w6" (OuterVolumeSpecName: "kube-api-access-f87w6") pod "4acc9329-a231-4f8e-bcdf-25580f5390ee" (UID: "4acc9329-a231-4f8e-bcdf-25580f5390ee"). InnerVolumeSpecName "kube-api-access-f87w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:49:09.057100 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.057072 2582 generic.go:356] "Generic (PLEG): container finished" podID="4acc9329-a231-4f8e-bcdf-25580f5390ee" containerID="f9e1585a8e0d0a129c2eaba4b906d285f713bfb07431694d5ba0c6f5157d69f9" exitCode=0 Sep 17 00:49:09.057239 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.057110 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" event={"ID":"4acc9329-a231-4f8e-bcdf-25580f5390ee","Type":"ContainerDied","Data":"f9e1585a8e0d0a129c2eaba4b906d285f713bfb07431694d5ba0c6f5157d69f9"} Sep 17 00:49:09.057239 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.057127 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" Sep 17 00:49:09.057239 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.057180 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj" event={"ID":"4acc9329-a231-4f8e-bcdf-25580f5390ee","Type":"ContainerDied","Data":"610e194863ac20c5b9f3dc35a8b868bd9e5a74e5d397ce804e1ec6c731eb63a4"} Sep 17 00:49:09.057239 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.057217 2582 scope.go:117] "RemoveContainer" containerID="f9e1585a8e0d0a129c2eaba4b906d285f713bfb07431694d5ba0c6f5157d69f9" Sep 17 00:49:09.066655 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.066639 2582 scope.go:117] "RemoveContainer" containerID="f9e1585a8e0d0a129c2eaba4b906d285f713bfb07431694d5ba0c6f5157d69f9" Sep 17 00:49:09.066931 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:49:09.066913 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e1585a8e0d0a129c2eaba4b906d285f713bfb07431694d5ba0c6f5157d69f9\": container with ID starting with f9e1585a8e0d0a129c2eaba4b906d285f713bfb07431694d5ba0c6f5157d69f9 not found: ID does not exist" containerID="f9e1585a8e0d0a129c2eaba4b906d285f713bfb07431694d5ba0c6f5157d69f9" Sep 17 00:49:09.066978 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.066941 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e1585a8e0d0a129c2eaba4b906d285f713bfb07431694d5ba0c6f5157d69f9"} err="failed to get container status \"f9e1585a8e0d0a129c2eaba4b906d285f713bfb07431694d5ba0c6f5157d69f9\": rpc error: code = NotFound desc = could not find container \"f9e1585a8e0d0a129c2eaba4b906d285f713bfb07431694d5ba0c6f5157d69f9\": container with ID starting with f9e1585a8e0d0a129c2eaba4b906d285f713bfb07431694d5ba0c6f5157d69f9 not found: ID does not exist" Sep 17 00:49:09.077427 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.077408 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj"] Sep 17 00:49:09.081191 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.081173 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-ef36871b-7fc49d67b7-rnzlj"] Sep 17 00:49:09.112181 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.112156 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f87w6\" (UniqueName: \"kubernetes.io/projected/4acc9329-a231-4f8e-bcdf-25580f5390ee-kube-api-access-f87w6\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:49:09.112181 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.112180 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4acc9329-a231-4f8e-bcdf-25580f5390ee-openshift-service-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:49:09.112305 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.112191 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4acc9329-a231-4f8e-bcdf-25580f5390ee-metrics-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:49:09.112305 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.112220 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4acc9329-a231-4f8e-bcdf-25580f5390ee-maas-api-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:49:09.651442 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:09.651405 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4acc9329-a231-4f8e-bcdf-25580f5390ee" path="/var/lib/kubelet/pods/4acc9329-a231-4f8e-bcdf-25580f5390ee/volumes" Sep 17 00:49:13.976609 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:13.976576 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt"] Sep 17 00:49:13.977271 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:13.977251 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4acc9329-a231-4f8e-bcdf-25580f5390ee" containerName="maas-api" Sep 17 00:49:13.977322 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:13.977276 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4acc9329-a231-4f8e-bcdf-25580f5390ee" containerName="maas-api" Sep 17 00:49:13.977406 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:13.977394 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4acc9329-a231-4f8e-bcdf-25580f5390ee" containerName="maas-api" Sep 17 00:49:13.982432 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:13.982410 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" Sep 17 00:49:13.984694 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:13.984566 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1035c6c0-serving-cert\"" Sep 17 00:49:13.987909 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:13.987886 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt"] Sep 17 00:49:14.055183 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:14.055155 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-openshift-service-ca\") pod \"maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt\" (UID: \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" Sep 17 00:49:14.055322 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:14.055212 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-metrics-certs\") pod \"maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt\" (UID: \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" Sep 17 00:49:14.055322 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:14.055293 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4wst\" (UniqueName: \"kubernetes.io/projected/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-kube-api-access-v4wst\") pod \"maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt\" (UID: \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" Sep 17 00:49:14.055399 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:14.055323 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-maas-api-tls\") pod \"maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt\" (UID: \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" Sep 17 00:49:14.156464 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:14.156432 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-metrics-certs\") pod \"maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt\" (UID: \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" Sep 17 00:49:14.156638 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:14.156488 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4wst\" (UniqueName: \"kubernetes.io/projected/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-kube-api-access-v4wst\") pod \"maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt\" (UID: \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" Sep 17 00:49:14.156638 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:14.156537 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-maas-api-tls\") pod \"maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt\" (UID: \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" Sep 17 00:49:14.156638 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:14.156586 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-openshift-service-ca\") pod \"maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt\" (UID: \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" Sep 17 00:49:14.156744 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:49:14.156707 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-serving-cert: secret "maas-api-e2e-ait-1035c6c0-serving-cert" not found Sep 17 00:49:14.156811 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:49:14.156798 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-maas-api-tls podName:a87de2bc-13bd-4dbd-91a2-3bdd5333746b nodeName:}" failed. No retries permitted until 2026-09-17 00:49:14.656776067 +0000 UTC m=+1167.587409549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-maas-api-tls") pod "maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" (UID: "a87de2bc-13bd-4dbd-91a2-3bdd5333746b") : secret "maas-api-e2e-ait-1035c6c0-serving-cert" not found Sep 17 00:49:14.157212 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:14.157187 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-openshift-service-ca\") pod \"maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt\" (UID: \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" Sep 17 00:49:14.159195 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:14.159173 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-metrics-certs\") pod \"maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt\" (UID: \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" Sep 17 00:49:14.164142 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:14.164116 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4wst\" (UniqueName: \"kubernetes.io/projected/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-kube-api-access-v4wst\") pod \"maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt\" (UID: \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" Sep 17 00:49:14.661087 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:14.661055 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-maas-api-tls\") pod \"maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt\" (UID: \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" Sep 17 00:49:14.663825 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:14.663794 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-maas-api-tls\") pod \"maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt\" (UID: \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" Sep 17 00:49:14.895261 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:14.895230 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" Sep 17 00:49:15.166076 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:15.166049 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt"] Sep 17 00:49:15.168691 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:49:15.168652 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87de2bc_13bd_4dbd_91a2_3bdd5333746b.slice/crio-979b0f87a97dc73649a0ef6462dde962d4c9532c19ecfe247b092acacbd5d728 WatchSource:0}: Error finding container 979b0f87a97dc73649a0ef6462dde962d4c9532c19ecfe247b092acacbd5d728: Status 404 returned error can't find the container with id 979b0f87a97dc73649a0ef6462dde962d4c9532c19ecfe247b092acacbd5d728 Sep 17 00:49:15.292771 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:15.292746 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:49:16.096432 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:16.096396 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" event={"ID":"a87de2bc-13bd-4dbd-91a2-3bdd5333746b","Type":"ContainerStarted","Data":"a4bfcea8b14bc2be4095516b0c29d6e2f2470e61dfdb43e0b1a0310a9009b81f"} Sep 17 00:49:16.096638 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:16.096459 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" Sep 17 00:49:16.096638 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:16.096476 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" event={"ID":"a87de2bc-13bd-4dbd-91a2-3bdd5333746b","Type":"ContainerStarted","Data":"979b0f87a97dc73649a0ef6462dde962d4c9532c19ecfe247b092acacbd5d728"} Sep 17 00:49:16.113008 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:16.112964 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" podStartSLOduration=2.753601933 podStartE2EDuration="3.112950185s" totalImagesPullingTime="359.348252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:49:16.110905101 +0000 UTC m=+1169.041538591" watchObservedRunningTime="2026-09-17 00:49:16.112950185 +0000 UTC m=+1169.043583672" Sep 17 00:49:22.107146 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:22.107116 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" Sep 17 00:49:33.089179 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:33.089145 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:49:43.995637 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:43.995604 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt"] Sep 17 00:49:43.996114 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:43.995849 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" podUID="a87de2bc-13bd-4dbd-91a2-3bdd5333746b" containerName="maas-api" containerID="cri-o://a4bfcea8b14bc2be4095516b0c29d6e2f2470e61dfdb43e0b1a0310a9009b81f" gracePeriod=30 Sep 17 00:49:44.239098 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.239074 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" Sep 17 00:49:44.251880 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.251814 2582 generic.go:356] "Generic (PLEG): container finished" podID="a87de2bc-13bd-4dbd-91a2-3bdd5333746b" containerID="a4bfcea8b14bc2be4095516b0c29d6e2f2470e61dfdb43e0b1a0310a9009b81f" exitCode=0 Sep 17 00:49:44.251880 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.251855 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" Sep 17 00:49:44.252022 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.251903 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" event={"ID":"a87de2bc-13bd-4dbd-91a2-3bdd5333746b","Type":"ContainerDied","Data":"a4bfcea8b14bc2be4095516b0c29d6e2f2470e61dfdb43e0b1a0310a9009b81f"} Sep 17 00:49:44.252022 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.251967 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt" event={"ID":"a87de2bc-13bd-4dbd-91a2-3bdd5333746b","Type":"ContainerDied","Data":"979b0f87a97dc73649a0ef6462dde962d4c9532c19ecfe247b092acacbd5d728"} Sep 17 00:49:44.252022 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.251995 2582 scope.go:117] "RemoveContainer" containerID="a4bfcea8b14bc2be4095516b0c29d6e2f2470e61dfdb43e0b1a0310a9009b81f" Sep 17 00:49:44.262672 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.262620 2582 scope.go:117] "RemoveContainer" containerID="a4bfcea8b14bc2be4095516b0c29d6e2f2470e61dfdb43e0b1a0310a9009b81f" Sep 17 00:49:44.262988 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:49:44.262968 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4bfcea8b14bc2be4095516b0c29d6e2f2470e61dfdb43e0b1a0310a9009b81f\": container with ID starting with a4bfcea8b14bc2be4095516b0c29d6e2f2470e61dfdb43e0b1a0310a9009b81f not found: ID does not exist" containerID="a4bfcea8b14bc2be4095516b0c29d6e2f2470e61dfdb43e0b1a0310a9009b81f" Sep 17 00:49:44.263058 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.263003 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4bfcea8b14bc2be4095516b0c29d6e2f2470e61dfdb43e0b1a0310a9009b81f"} err="failed to get container status \"a4bfcea8b14bc2be4095516b0c29d6e2f2470e61dfdb43e0b1a0310a9009b81f\": rpc error: code = NotFound desc = could not find container \"a4bfcea8b14bc2be4095516b0c29d6e2f2470e61dfdb43e0b1a0310a9009b81f\": container with ID starting with a4bfcea8b14bc2be4095516b0c29d6e2f2470e61dfdb43e0b1a0310a9009b81f not found: ID does not exist" Sep 17 00:49:44.333462 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.333436 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-maas-api-tls\") pod \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\" (UID: \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\") " Sep 17 00:49:44.333596 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.333468 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4wst\" (UniqueName: \"kubernetes.io/projected/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-kube-api-access-v4wst\") pod \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\" (UID: \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\") " Sep 17 00:49:44.333596 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.333554 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-metrics-certs\") pod \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\" (UID: \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\") " Sep 17 00:49:44.333689 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.333602 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-openshift-service-ca\") pod \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\" (UID: \"a87de2bc-13bd-4dbd-91a2-3bdd5333746b\") " Sep 17 00:49:44.333999 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.333977 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a87de2bc-13bd-4dbd-91a2-3bdd5333746b" (UID: "a87de2bc-13bd-4dbd-91a2-3bdd5333746b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 00:49:44.335756 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.335730 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a87de2bc-13bd-4dbd-91a2-3bdd5333746b" (UID: "a87de2bc-13bd-4dbd-91a2-3bdd5333746b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:49:44.335863 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.335843 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a87de2bc-13bd-4dbd-91a2-3bdd5333746b" (UID: "a87de2bc-13bd-4dbd-91a2-3bdd5333746b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:49:44.335863 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.335849 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-kube-api-access-v4wst" (OuterVolumeSpecName: "kube-api-access-v4wst") pod "a87de2bc-13bd-4dbd-91a2-3bdd5333746b" (UID: "a87de2bc-13bd-4dbd-91a2-3bdd5333746b"). InnerVolumeSpecName "kube-api-access-v4wst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:49:44.434864 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.434838 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-metrics-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:49:44.434864 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.434864 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-openshift-service-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:49:44.435026 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.434873 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-maas-api-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:49:44.435026 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.434882 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4wst\" (UniqueName: \"kubernetes.io/projected/a87de2bc-13bd-4dbd-91a2-3bdd5333746b-kube-api-access-v4wst\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:49:44.579519 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.579469 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt"] Sep 17 00:49:44.584887 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:44.584866 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1035c6c0-656c68fd7f-p9gjt"] Sep 17 00:49:45.650794 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:45.650763 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a87de2bc-13bd-4dbd-91a2-3bdd5333746b" path="/var/lib/kubelet/pods/a87de2bc-13bd-4dbd-91a2-3bdd5333746b/volumes" Sep 17 00:49:47.671979 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:47.671872 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/1.log" Sep 17 00:49:47.677644 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:47.677038 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/1.log" Sep 17 00:49:49.891015 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:49.890983 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:49:51.628837 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:51.628804 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64"] Sep 17 00:49:51.629459 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:51.629442 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87de2bc-13bd-4dbd-91a2-3bdd5333746b" containerName="maas-api" Sep 17 00:49:51.629538 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:51.629462 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87de2bc-13bd-4dbd-91a2-3bdd5333746b" containerName="maas-api" Sep 17 00:49:51.629632 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:51.629619 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87de2bc-13bd-4dbd-91a2-3bdd5333746b" containerName="maas-api" Sep 17 00:49:51.638817 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:51.638793 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" Sep 17 00:49:51.641073 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:51.640995 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-39a5d066-serving-cert\"" Sep 17 00:49:51.641384 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:51.641358 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64"] Sep 17 00:49:51.700833 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:51.700802 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwgjx\" (UniqueName: \"kubernetes.io/projected/e11b1d71-591d-44e4-b926-8b09841a3075-kube-api-access-mwgjx\") pod \"maas-api-e2e-derive-39a5d066-694984fb67-twk64\" (UID: \"e11b1d71-591d-44e4-b926-8b09841a3075\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" Sep 17 00:49:51.701026 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:51.700859 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e11b1d71-591d-44e4-b926-8b09841a3075-openshift-service-ca\") pod \"maas-api-e2e-derive-39a5d066-694984fb67-twk64\" (UID: \"e11b1d71-591d-44e4-b926-8b09841a3075\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" Sep 17 00:49:51.701026 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:51.700904 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e11b1d71-591d-44e4-b926-8b09841a3075-metrics-certs\") pod \"maas-api-e2e-derive-39a5d066-694984fb67-twk64\" (UID: \"e11b1d71-591d-44e4-b926-8b09841a3075\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" Sep 17 00:49:51.701026 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:51.700942 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e11b1d71-591d-44e4-b926-8b09841a3075-maas-api-tls\") pod \"maas-api-e2e-derive-39a5d066-694984fb67-twk64\" (UID: \"e11b1d71-591d-44e4-b926-8b09841a3075\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" Sep 17 00:49:51.801667 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:51.801632 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwgjx\" (UniqueName: \"kubernetes.io/projected/e11b1d71-591d-44e4-b926-8b09841a3075-kube-api-access-mwgjx\") pod \"maas-api-e2e-derive-39a5d066-694984fb67-twk64\" (UID: \"e11b1d71-591d-44e4-b926-8b09841a3075\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" Sep 17 00:49:51.801856 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:51.801706 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e11b1d71-591d-44e4-b926-8b09841a3075-openshift-service-ca\") pod \"maas-api-e2e-derive-39a5d066-694984fb67-twk64\" (UID: \"e11b1d71-591d-44e4-b926-8b09841a3075\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" Sep 17 00:49:51.801856 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:51.801730 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e11b1d71-591d-44e4-b926-8b09841a3075-metrics-certs\") pod \"maas-api-e2e-derive-39a5d066-694984fb67-twk64\" (UID: \"e11b1d71-591d-44e4-b926-8b09841a3075\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" Sep 17 00:49:51.801856 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:51.801752 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e11b1d71-591d-44e4-b926-8b09841a3075-maas-api-tls\") pod \"maas-api-e2e-derive-39a5d066-694984fb67-twk64\" (UID: \"e11b1d71-591d-44e4-b926-8b09841a3075\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" Sep 17 00:49:51.802020 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:49:51.801940 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-serving-cert: secret "maas-api-e2e-derive-39a5d066-serving-cert" not found Sep 17 00:49:51.802020 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:49:51.802015 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e11b1d71-591d-44e4-b926-8b09841a3075-maas-api-tls podName:e11b1d71-591d-44e4-b926-8b09841a3075 nodeName:}" failed. No retries permitted until 2026-09-17 00:49:52.301996953 +0000 UTC m=+1205.232630435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e11b1d71-591d-44e4-b926-8b09841a3075-maas-api-tls") pod "maas-api-e2e-derive-39a5d066-694984fb67-twk64" (UID: "e11b1d71-591d-44e4-b926-8b09841a3075") : secret "maas-api-e2e-derive-39a5d066-serving-cert" not found Sep 17 00:49:51.802374 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:51.802348 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e11b1d71-591d-44e4-b926-8b09841a3075-openshift-service-ca\") pod \"maas-api-e2e-derive-39a5d066-694984fb67-twk64\" (UID: \"e11b1d71-591d-44e4-b926-8b09841a3075\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" Sep 17 00:49:51.804247 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:51.804228 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e11b1d71-591d-44e4-b926-8b09841a3075-metrics-certs\") pod \"maas-api-e2e-derive-39a5d066-694984fb67-twk64\" (UID: \"e11b1d71-591d-44e4-b926-8b09841a3075\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" Sep 17 00:49:51.810282 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:51.810260 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwgjx\" (UniqueName: \"kubernetes.io/projected/e11b1d71-591d-44e4-b926-8b09841a3075-kube-api-access-mwgjx\") pod \"maas-api-e2e-derive-39a5d066-694984fb67-twk64\" (UID: \"e11b1d71-591d-44e4-b926-8b09841a3075\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" Sep 17 00:49:52.308639 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:52.308608 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e11b1d71-591d-44e4-b926-8b09841a3075-maas-api-tls\") pod \"maas-api-e2e-derive-39a5d066-694984fb67-twk64\" (UID: \"e11b1d71-591d-44e4-b926-8b09841a3075\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" Sep 17 00:49:52.311355 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:52.311326 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e11b1d71-591d-44e4-b926-8b09841a3075-maas-api-tls\") pod \"maas-api-e2e-derive-39a5d066-694984fb67-twk64\" (UID: \"e11b1d71-591d-44e4-b926-8b09841a3075\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" Sep 17 00:49:52.552850 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:52.552817 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" Sep 17 00:49:52.824293 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:52.824265 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64"] Sep 17 00:49:52.826349 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:49:52.826325 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11b1d71_591d_44e4_b926_8b09841a3075.slice/crio-e34e37fa7a67e33fbe45ddfd6661c8f0ff7463074aae96a13231648a6083adc9 WatchSource:0}: Error finding container e34e37fa7a67e33fbe45ddfd6661c8f0ff7463074aae96a13231648a6083adc9: Status 404 returned error can't find the container with id e34e37fa7a67e33fbe45ddfd6661c8f0ff7463074aae96a13231648a6083adc9 Sep 17 00:49:53.305808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:53.305776 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" event={"ID":"e11b1d71-591d-44e4-b926-8b09841a3075","Type":"ContainerStarted","Data":"ed50972bdeb1f17d894893d3eb5d954f3a23c46ee49dacdfeee61f8b0fdb57a5"} Sep 17 00:49:53.305961 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:53.305938 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" Sep 17 00:49:53.306032 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:53.305985 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" event={"ID":"e11b1d71-591d-44e4-b926-8b09841a3075","Type":"ContainerStarted","Data":"e34e37fa7a67e33fbe45ddfd6661c8f0ff7463074aae96a13231648a6083adc9"} Sep 17 00:49:53.320835 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:53.320795 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" podStartSLOduration=1.924363052 podStartE2EDuration="2.32078103s" totalImagesPullingTime="396.417978ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:49:53.319239569 +0000 UTC m=+1206.249873067" watchObservedRunningTime="2026-09-17 00:49:53.32078103 +0000 UTC m=+1206.251414516" Sep 17 00:49:59.318782 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:49:59.318739 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" Sep 17 00:50:06.574527 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.574473 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv"] Sep 17 00:50:06.578569 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.578549 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:06.581319 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.581291 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f40754\"/\"default-dockercfg-f8kxk\"" Sep 17 00:50:06.581462 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.581291 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f40754\"/\"simulator-w4-f40754-kserve-self-signed-certs\"" Sep 17 00:50:06.581462 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.581341 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f40754\"/\"openshift-service-ca.crt\"" Sep 17 00:50:06.581462 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.581322 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f40754\"/\"kube-root-ca.crt\"" Sep 17 00:50:06.587416 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.587395 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv"] Sep 17 00:50:06.741582 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.741553 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-tmp-dir\") pod \"simulator-w4-f40754-kserve-85f5594c8c-9sszv\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:06.741759 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.741603 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-home\") pod \"simulator-w4-f40754-kserve-85f5594c8c-9sszv\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:06.741759 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.741683 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08638c74-f335-4aa5-8290-659b4f15496f-tls-certs\") pod \"simulator-w4-f40754-kserve-85f5594c8c-9sszv\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:06.741759 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.741731 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-model-cache\") pod \"simulator-w4-f40754-kserve-85f5594c8c-9sszv\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:06.741888 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.741784 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-dshm\") pod \"simulator-w4-f40754-kserve-85f5594c8c-9sszv\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:06.741888 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.741841 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5v4f\" (UniqueName: \"kubernetes.io/projected/08638c74-f335-4aa5-8290-659b4f15496f-kube-api-access-t5v4f\") pod \"simulator-w4-f40754-kserve-85f5594c8c-9sszv\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:06.842355 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.842263 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-model-cache\") pod \"simulator-w4-f40754-kserve-85f5594c8c-9sszv\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:06.842355 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.842313 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-dshm\") pod \"simulator-w4-f40754-kserve-85f5594c8c-9sszv\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:06.842355 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.842352 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5v4f\" (UniqueName: \"kubernetes.io/projected/08638c74-f335-4aa5-8290-659b4f15496f-kube-api-access-t5v4f\") pod \"simulator-w4-f40754-kserve-85f5594c8c-9sszv\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:06.842631 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.842375 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-tmp-dir\") pod \"simulator-w4-f40754-kserve-85f5594c8c-9sszv\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:06.842631 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.842410 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-home\") pod \"simulator-w4-f40754-kserve-85f5594c8c-9sszv\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:06.842631 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.842554 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08638c74-f335-4aa5-8290-659b4f15496f-tls-certs\") pod \"simulator-w4-f40754-kserve-85f5594c8c-9sszv\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:06.842804 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.842668 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-model-cache\") pod \"simulator-w4-f40754-kserve-85f5594c8c-9sszv\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:06.842863 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.842811 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-home\") pod \"simulator-w4-f40754-kserve-85f5594c8c-9sszv\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:06.842863 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.842844 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-tmp-dir\") pod \"simulator-w4-f40754-kserve-85f5594c8c-9sszv\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:06.844833 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.844809 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-dshm\") pod \"simulator-w4-f40754-kserve-85f5594c8c-9sszv\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:06.845065 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.845045 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08638c74-f335-4aa5-8290-659b4f15496f-tls-certs\") pod \"simulator-w4-f40754-kserve-85f5594c8c-9sszv\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:06.849748 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.849727 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5v4f\" (UniqueName: \"kubernetes.io/projected/08638c74-f335-4aa5-8290-659b4f15496f-kube-api-access-t5v4f\") pod \"simulator-w4-f40754-kserve-85f5594c8c-9sszv\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:06.888764 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:06.888743 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:07.160023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:07.159846 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv"] Sep 17 00:50:07.163086 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:50:07.163056 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08638c74_f335_4aa5_8290_659b4f15496f.slice/crio-23a9f64173d378462fb314ce6ff07ab79793086a95fd1fb6b36a2eda7809b439 WatchSource:0}: Error finding container 23a9f64173d378462fb314ce6ff07ab79793086a95fd1fb6b36a2eda7809b439: Status 404 returned error can't find the container with id 23a9f64173d378462fb314ce6ff07ab79793086a95fd1fb6b36a2eda7809b439 Sep 17 00:50:07.386863 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:07.386828 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" event={"ID":"08638c74-f335-4aa5-8290-659b4f15496f","Type":"ContainerStarted","Data":"bce698b0de488d4c1c7a7c287553c45889c7def3cc66e54cde09f6d95b797797"} Sep 17 00:50:07.386863 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:07.386869 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" event={"ID":"08638c74-f335-4aa5-8290-659b4f15496f","Type":"ContainerStarted","Data":"23a9f64173d378462fb314ce6ff07ab79793086a95fd1fb6b36a2eda7809b439"} Sep 17 00:50:07.402195 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:07.402142 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" podStartSLOduration=1.402127781 podStartE2EDuration="1.402127781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:50:07.400561712 +0000 UTC m=+1220.331195198" watchObservedRunningTime="2026-09-17 00:50:07.402127781 +0000 UTC m=+1220.332761267" Sep 17 00:50:07.795815 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:07.795782 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:50:16.888999 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:16.888955 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:16.889445 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:16.889024 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:16.902778 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:16.902749 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:17.455171 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:17.455137 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:50:23.896823 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:23.896786 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:50:29.826711 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:29.826674 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64"] Sep 17 00:50:29.827154 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:29.826874 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" podUID="e11b1d71-591d-44e4-b926-8b09841a3075" containerName="maas-api" containerID="cri-o://ed50972bdeb1f17d894893d3eb5d954f3a23c46ee49dacdfeee61f8b0fdb57a5" gracePeriod=30 Sep 17 00:50:30.091319 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.091255 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" Sep 17 00:50:30.165930 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.165902 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e11b1d71-591d-44e4-b926-8b09841a3075-metrics-certs\") pod \"e11b1d71-591d-44e4-b926-8b09841a3075\" (UID: \"e11b1d71-591d-44e4-b926-8b09841a3075\") " Sep 17 00:50:30.165930 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.165945 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e11b1d71-591d-44e4-b926-8b09841a3075-maas-api-tls\") pod \"e11b1d71-591d-44e4-b926-8b09841a3075\" (UID: \"e11b1d71-591d-44e4-b926-8b09841a3075\") " Sep 17 00:50:30.166179 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.165975 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwgjx\" (UniqueName: \"kubernetes.io/projected/e11b1d71-591d-44e4-b926-8b09841a3075-kube-api-access-mwgjx\") pod \"e11b1d71-591d-44e4-b926-8b09841a3075\" (UID: \"e11b1d71-591d-44e4-b926-8b09841a3075\") " Sep 17 00:50:30.166179 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.166055 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e11b1d71-591d-44e4-b926-8b09841a3075-openshift-service-ca\") pod \"e11b1d71-591d-44e4-b926-8b09841a3075\" (UID: \"e11b1d71-591d-44e4-b926-8b09841a3075\") " Sep 17 00:50:30.166759 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.166702 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e11b1d71-591d-44e4-b926-8b09841a3075-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e11b1d71-591d-44e4-b926-8b09841a3075" (UID: "e11b1d71-591d-44e4-b926-8b09841a3075"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 00:50:30.169445 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.169405 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e11b1d71-591d-44e4-b926-8b09841a3075-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e11b1d71-591d-44e4-b926-8b09841a3075" (UID: "e11b1d71-591d-44e4-b926-8b09841a3075"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:50:30.171294 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.171267 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11b1d71-591d-44e4-b926-8b09841a3075-kube-api-access-mwgjx" (OuterVolumeSpecName: "kube-api-access-mwgjx") pod "e11b1d71-591d-44e4-b926-8b09841a3075" (UID: "e11b1d71-591d-44e4-b926-8b09841a3075"). InnerVolumeSpecName "kube-api-access-mwgjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:50:30.171393 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.171357 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e11b1d71-591d-44e4-b926-8b09841a3075-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e11b1d71-591d-44e4-b926-8b09841a3075" (UID: "e11b1d71-591d-44e4-b926-8b09841a3075"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:50:30.267967 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.267921 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e11b1d71-591d-44e4-b926-8b09841a3075-maas-api-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:50:30.267967 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.267962 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwgjx\" (UniqueName: \"kubernetes.io/projected/e11b1d71-591d-44e4-b926-8b09841a3075-kube-api-access-mwgjx\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:50:30.267967 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.267971 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e11b1d71-591d-44e4-b926-8b09841a3075-openshift-service-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:50:30.268203 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.267983 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e11b1d71-591d-44e4-b926-8b09841a3075-metrics-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:50:30.512130 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.512045 2582 generic.go:356] "Generic (PLEG): container finished" podID="e11b1d71-591d-44e4-b926-8b09841a3075" containerID="ed50972bdeb1f17d894893d3eb5d954f3a23c46ee49dacdfeee61f8b0fdb57a5" exitCode=0 Sep 17 00:50:30.512130 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.512091 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" Sep 17 00:50:30.512337 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.512130 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" event={"ID":"e11b1d71-591d-44e4-b926-8b09841a3075","Type":"ContainerDied","Data":"ed50972bdeb1f17d894893d3eb5d954f3a23c46ee49dacdfeee61f8b0fdb57a5"} Sep 17 00:50:30.512337 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.512182 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64" event={"ID":"e11b1d71-591d-44e4-b926-8b09841a3075","Type":"ContainerDied","Data":"e34e37fa7a67e33fbe45ddfd6661c8f0ff7463074aae96a13231648a6083adc9"} Sep 17 00:50:30.512337 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.512203 2582 scope.go:117] "RemoveContainer" containerID="ed50972bdeb1f17d894893d3eb5d954f3a23c46ee49dacdfeee61f8b0fdb57a5" Sep 17 00:50:30.522026 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.522010 2582 scope.go:117] "RemoveContainer" containerID="ed50972bdeb1f17d894893d3eb5d954f3a23c46ee49dacdfeee61f8b0fdb57a5" Sep 17 00:50:30.522321 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:50:30.522300 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed50972bdeb1f17d894893d3eb5d954f3a23c46ee49dacdfeee61f8b0fdb57a5\": container with ID starting with ed50972bdeb1f17d894893d3eb5d954f3a23c46ee49dacdfeee61f8b0fdb57a5 not found: ID does not exist" containerID="ed50972bdeb1f17d894893d3eb5d954f3a23c46ee49dacdfeee61f8b0fdb57a5" Sep 17 00:50:30.522407 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.522327 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed50972bdeb1f17d894893d3eb5d954f3a23c46ee49dacdfeee61f8b0fdb57a5"} err="failed to get container status \"ed50972bdeb1f17d894893d3eb5d954f3a23c46ee49dacdfeee61f8b0fdb57a5\": rpc error: code = NotFound desc = could not find container \"ed50972bdeb1f17d894893d3eb5d954f3a23c46ee49dacdfeee61f8b0fdb57a5\": container with ID starting with ed50972bdeb1f17d894893d3eb5d954f3a23c46ee49dacdfeee61f8b0fdb57a5 not found: ID does not exist" Sep 17 00:50:30.534019 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.533997 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64"] Sep 17 00:50:30.539385 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:30.539364 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-39a5d066-694984fb67-twk64"] Sep 17 00:50:31.651268 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:31.651233 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e11b1d71-591d-44e4-b926-8b09841a3075" path="/var/lib/kubelet/pods/e11b1d71-591d-44e4-b926-8b09841a3075/volumes" Sep 17 00:50:38.002177 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:50:38.002140 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:51:07.492785 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:07.492755 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:51:12.433663 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.433629 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5"] Sep 17 00:51:12.434091 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.433852 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" podUID="dc785bad-fc9b-4143-8f4a-e773b51153f8" containerName="maas-api" containerID="cri-o://b209cf764e8d60b6d1c1610f8beba7404c3b843b0fbc0e59279f8cb3ccf00785" gracePeriod=30 Sep 17 00:51:12.688122 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.688060 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" Sep 17 00:51:12.740864 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.740829 2582 generic.go:356] "Generic (PLEG): container finished" podID="dc785bad-fc9b-4143-8f4a-e773b51153f8" containerID="b209cf764e8d60b6d1c1610f8beba7404c3b843b0fbc0e59279f8cb3ccf00785" exitCode=0 Sep 17 00:51:12.741026 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.740871 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" Sep 17 00:51:12.741026 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.740875 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" event={"ID":"dc785bad-fc9b-4143-8f4a-e773b51153f8","Type":"ContainerDied","Data":"b209cf764e8d60b6d1c1610f8beba7404c3b843b0fbc0e59279f8cb3ccf00785"} Sep 17 00:51:12.741026 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.740924 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5" event={"ID":"dc785bad-fc9b-4143-8f4a-e773b51153f8","Type":"ContainerDied","Data":"e98c64f29b88aad26d3832362af1b29c76ffe401f25c1771575de854ee50b59c"} Sep 17 00:51:12.741026 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.740953 2582 scope.go:117] "RemoveContainer" containerID="b209cf764e8d60b6d1c1610f8beba7404c3b843b0fbc0e59279f8cb3ccf00785" Sep 17 00:51:12.750620 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.750598 2582 scope.go:117] "RemoveContainer" containerID="b209cf764e8d60b6d1c1610f8beba7404c3b843b0fbc0e59279f8cb3ccf00785" Sep 17 00:51:12.750990 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:51:12.750963 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b209cf764e8d60b6d1c1610f8beba7404c3b843b0fbc0e59279f8cb3ccf00785\": container with ID starting with b209cf764e8d60b6d1c1610f8beba7404c3b843b0fbc0e59279f8cb3ccf00785 not found: ID does not exist" containerID="b209cf764e8d60b6d1c1610f8beba7404c3b843b0fbc0e59279f8cb3ccf00785" Sep 17 00:51:12.751095 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.751003 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b209cf764e8d60b6d1c1610f8beba7404c3b843b0fbc0e59279f8cb3ccf00785"} err="failed to get container status \"b209cf764e8d60b6d1c1610f8beba7404c3b843b0fbc0e59279f8cb3ccf00785\": rpc error: code = NotFound desc = could not find container \"b209cf764e8d60b6d1c1610f8beba7404c3b843b0fbc0e59279f8cb3ccf00785\": container with ID starting with b209cf764e8d60b6d1c1610f8beba7404c3b843b0fbc0e59279f8cb3ccf00785 not found: ID does not exist" Sep 17 00:51:12.778354 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.778327 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc785bad-fc9b-4143-8f4a-e773b51153f8-maas-api-tls\") pod \"dc785bad-fc9b-4143-8f4a-e773b51153f8\" (UID: \"dc785bad-fc9b-4143-8f4a-e773b51153f8\") " Sep 17 00:51:12.778527 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.778391 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc785bad-fc9b-4143-8f4a-e773b51153f8-metrics-certs\") pod \"dc785bad-fc9b-4143-8f4a-e773b51153f8\" (UID: \"dc785bad-fc9b-4143-8f4a-e773b51153f8\") " Sep 17 00:51:12.778527 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.778410 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jk4rm\" (UniqueName: \"kubernetes.io/projected/dc785bad-fc9b-4143-8f4a-e773b51153f8-kube-api-access-jk4rm\") pod \"dc785bad-fc9b-4143-8f4a-e773b51153f8\" (UID: \"dc785bad-fc9b-4143-8f4a-e773b51153f8\") " Sep 17 00:51:12.778527 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.778465 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc785bad-fc9b-4143-8f4a-e773b51153f8-openshift-service-ca\") pod \"dc785bad-fc9b-4143-8f4a-e773b51153f8\" (UID: \"dc785bad-fc9b-4143-8f4a-e773b51153f8\") " Sep 17 00:51:12.779432 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.778896 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc785bad-fc9b-4143-8f4a-e773b51153f8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dc785bad-fc9b-4143-8f4a-e773b51153f8" (UID: "dc785bad-fc9b-4143-8f4a-e773b51153f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 00:51:12.779432 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.779425 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc785bad-fc9b-4143-8f4a-e773b51153f8-openshift-service-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:51:12.780853 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.780824 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc785bad-fc9b-4143-8f4a-e773b51153f8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dc785bad-fc9b-4143-8f4a-e773b51153f8" (UID: "dc785bad-fc9b-4143-8f4a-e773b51153f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:51:12.780958 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.780916 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc785bad-fc9b-4143-8f4a-e773b51153f8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dc785bad-fc9b-4143-8f4a-e773b51153f8" (UID: "dc785bad-fc9b-4143-8f4a-e773b51153f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:51:12.781299 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.781272 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc785bad-fc9b-4143-8f4a-e773b51153f8-kube-api-access-jk4rm" (OuterVolumeSpecName: "kube-api-access-jk4rm") pod "dc785bad-fc9b-4143-8f4a-e773b51153f8" (UID: "dc785bad-fc9b-4143-8f4a-e773b51153f8"). InnerVolumeSpecName "kube-api-access-jk4rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:51:12.881039 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.881002 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc785bad-fc9b-4143-8f4a-e773b51153f8-maas-api-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:51:12.881039 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.881032 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc785bad-fc9b-4143-8f4a-e773b51153f8-metrics-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:51:12.881039 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:12.881042 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk4rm\" (UniqueName: \"kubernetes.io/projected/dc785bad-fc9b-4143-8f4a-e773b51153f8-kube-api-access-jk4rm\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:51:13.072391 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:13.072356 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5"] Sep 17 00:51:13.081224 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:13.081196 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6af38a-5d748b648d-8n4g5"] Sep 17 00:51:13.651592 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:13.651562 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc785bad-fc9b-4143-8f4a-e773b51153f8" path="/var/lib/kubelet/pods/dc785bad-fc9b-4143-8f4a-e773b51153f8/volumes" Sep 17 00:51:27.686678 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:27.686632 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:51:30.238812 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:30.238770 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv"] Sep 17 00:51:35.848967 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:35.848937 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49"] Sep 17 00:51:35.849523 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:35.849141 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" podUID="221c5b4f-3870-4aa3-8fa4-77a46596ffc2" containerName="maas-api" containerID="cri-o://b902e5204abe5ac092eddf465baa225448285edf3d190790c77822f95360096c" gracePeriod=30 Sep 17 00:51:36.101199 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.101148 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" Sep 17 00:51:36.205188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.205152 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr47g\" (UniqueName: \"kubernetes.io/projected/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-kube-api-access-zr47g\") pod \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\" (UID: \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\") " Sep 17 00:51:36.205352 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.205210 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-openshift-service-ca\") pod \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\" (UID: \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\") " Sep 17 00:51:36.205352 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.205293 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-metrics-certs\") pod \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\" (UID: \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\") " Sep 17 00:51:36.205352 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.205327 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-maas-api-tls\") pod \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\" (UID: \"221c5b4f-3870-4aa3-8fa4-77a46596ffc2\") " Sep 17 00:51:36.205616 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.205592 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "221c5b4f-3870-4aa3-8fa4-77a46596ffc2" (UID: "221c5b4f-3870-4aa3-8fa4-77a46596ffc2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 00:51:36.207567 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.207538 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-kube-api-access-zr47g" (OuterVolumeSpecName: "kube-api-access-zr47g") pod "221c5b4f-3870-4aa3-8fa4-77a46596ffc2" (UID: "221c5b4f-3870-4aa3-8fa4-77a46596ffc2"). InnerVolumeSpecName "kube-api-access-zr47g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:51:36.207671 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.207592 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "221c5b4f-3870-4aa3-8fa4-77a46596ffc2" (UID: "221c5b4f-3870-4aa3-8fa4-77a46596ffc2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:51:36.207671 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.207624 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "221c5b4f-3870-4aa3-8fa4-77a46596ffc2" (UID: "221c5b4f-3870-4aa3-8fa4-77a46596ffc2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:51:36.306963 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.306942 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-openshift-service-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:51:36.306963 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.306965 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-metrics-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:51:36.307109 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.306974 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-maas-api-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:51:36.307109 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.306982 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr47g\" (UniqueName: \"kubernetes.io/projected/221c5b4f-3870-4aa3-8fa4-77a46596ffc2-kube-api-access-zr47g\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:51:36.869136 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.869099 2582 generic.go:356] "Generic (PLEG): container finished" podID="221c5b4f-3870-4aa3-8fa4-77a46596ffc2" containerID="b902e5204abe5ac092eddf465baa225448285edf3d190790c77822f95360096c" exitCode=0 Sep 17 00:51:36.869627 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.869144 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" Sep 17 00:51:36.869627 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.869187 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" event={"ID":"221c5b4f-3870-4aa3-8fa4-77a46596ffc2","Type":"ContainerDied","Data":"b902e5204abe5ac092eddf465baa225448285edf3d190790c77822f95360096c"} Sep 17 00:51:36.869627 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.869248 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49" event={"ID":"221c5b4f-3870-4aa3-8fa4-77a46596ffc2","Type":"ContainerDied","Data":"a424bec7fc5fea802fdab511c7ea7517cc4c334ae8d54dc663be4e61068b350a"} Sep 17 00:51:36.869627 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.869283 2582 scope.go:117] "RemoveContainer" containerID="b902e5204abe5ac092eddf465baa225448285edf3d190790c77822f95360096c" Sep 17 00:51:36.879124 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.879102 2582 scope.go:117] "RemoveContainer" containerID="b902e5204abe5ac092eddf465baa225448285edf3d190790c77822f95360096c" Sep 17 00:51:36.879478 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:51:36.879453 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b902e5204abe5ac092eddf465baa225448285edf3d190790c77822f95360096c\": container with ID starting with b902e5204abe5ac092eddf465baa225448285edf3d190790c77822f95360096c not found: ID does not exist" containerID="b902e5204abe5ac092eddf465baa225448285edf3d190790c77822f95360096c" Sep 17 00:51:36.879590 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.879487 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b902e5204abe5ac092eddf465baa225448285edf3d190790c77822f95360096c"} err="failed to get container status \"b902e5204abe5ac092eddf465baa225448285edf3d190790c77822f95360096c\": rpc error: code = NotFound desc = could not find container \"b902e5204abe5ac092eddf465baa225448285edf3d190790c77822f95360096c\": container with ID starting with b902e5204abe5ac092eddf465baa225448285edf3d190790c77822f95360096c not found: ID does not exist" Sep 17 00:51:36.891355 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.891331 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49"] Sep 17 00:51:36.894625 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:36.894607 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f40754-5bfbfbffdd-cgc49"] Sep 17 00:51:37.657259 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:37.657221 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="221c5b4f-3870-4aa3-8fa4-77a46596ffc2" path="/var/lib/kubelet/pods/221c5b4f-3870-4aa3-8fa4-77a46596ffc2/volumes" Sep 17 00:51:45.259549 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.259483 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" podUID="08638c74-f335-4aa5-8290-659b4f15496f" containerName="main" containerID="cri-o://bce698b0de488d4c1c7a7c287553c45889c7def3cc66e54cde09f6d95b797797" gracePeriod=45 Sep 17 00:51:45.514869 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.514808 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:51:45.594981 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.594949 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-home\") pod \"08638c74-f335-4aa5-8290-659b4f15496f\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " Sep 17 00:51:45.595168 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.595025 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-tmp-dir\") pod \"08638c74-f335-4aa5-8290-659b4f15496f\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " Sep 17 00:51:45.595168 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.595069 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-dshm\") pod \"08638c74-f335-4aa5-8290-659b4f15496f\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " Sep 17 00:51:45.595168 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.595109 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5v4f\" (UniqueName: \"kubernetes.io/projected/08638c74-f335-4aa5-8290-659b4f15496f-kube-api-access-t5v4f\") pod \"08638c74-f335-4aa5-8290-659b4f15496f\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " Sep 17 00:51:45.595329 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.595169 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-model-cache\") pod \"08638c74-f335-4aa5-8290-659b4f15496f\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " Sep 17 00:51:45.595329 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.595239 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-home" (OuterVolumeSpecName: "home") pod "08638c74-f335-4aa5-8290-659b4f15496f" (UID: "08638c74-f335-4aa5-8290-659b4f15496f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:51:45.595329 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.595263 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08638c74-f335-4aa5-8290-659b4f15496f-tls-certs\") pod \"08638c74-f335-4aa5-8290-659b4f15496f\" (UID: \"08638c74-f335-4aa5-8290-659b4f15496f\") " Sep 17 00:51:45.595329 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.595262 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "08638c74-f335-4aa5-8290-659b4f15496f" (UID: "08638c74-f335-4aa5-8290-659b4f15496f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:51:45.595542 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.595457 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-model-cache" (OuterVolumeSpecName: "model-cache") pod "08638c74-f335-4aa5-8290-659b4f15496f" (UID: "08638c74-f335-4aa5-8290-659b4f15496f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:51:45.595611 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.595593 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-model-cache\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:51:45.595686 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.595615 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-home\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:51:45.595686 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.595623 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-tmp-dir\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:51:45.597443 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.597414 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-dshm" (OuterVolumeSpecName: "dshm") pod "08638c74-f335-4aa5-8290-659b4f15496f" (UID: "08638c74-f335-4aa5-8290-659b4f15496f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:51:45.597443 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.597433 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08638c74-f335-4aa5-8290-659b4f15496f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08638c74-f335-4aa5-8290-659b4f15496f" (UID: "08638c74-f335-4aa5-8290-659b4f15496f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:51:45.597583 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.597557 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08638c74-f335-4aa5-8290-659b4f15496f-kube-api-access-t5v4f" (OuterVolumeSpecName: "kube-api-access-t5v4f") pod "08638c74-f335-4aa5-8290-659b4f15496f" (UID: "08638c74-f335-4aa5-8290-659b4f15496f"). InnerVolumeSpecName "kube-api-access-t5v4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:51:45.697066 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.697026 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08638c74-f335-4aa5-8290-659b4f15496f-dshm\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:51:45.697066 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.697058 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5v4f\" (UniqueName: \"kubernetes.io/projected/08638c74-f335-4aa5-8290-659b4f15496f-kube-api-access-t5v4f\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:51:45.697066 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.697073 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08638c74-f335-4aa5-8290-659b4f15496f-tls-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:51:45.920647 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.920613 2582 generic.go:356] "Generic (PLEG): container finished" podID="08638c74-f335-4aa5-8290-659b4f15496f" containerID="bce698b0de488d4c1c7a7c287553c45889c7def3cc66e54cde09f6d95b797797" exitCode=0 Sep 17 00:51:45.920816 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.920702 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" Sep 17 00:51:45.920816 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.920727 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" event={"ID":"08638c74-f335-4aa5-8290-659b4f15496f","Type":"ContainerDied","Data":"bce698b0de488d4c1c7a7c287553c45889c7def3cc66e54cde09f6d95b797797"} Sep 17 00:51:45.920816 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.920760 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv" event={"ID":"08638c74-f335-4aa5-8290-659b4f15496f","Type":"ContainerDied","Data":"23a9f64173d378462fb314ce6ff07ab79793086a95fd1fb6b36a2eda7809b439"} Sep 17 00:51:45.920816 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.920776 2582 scope.go:117] "RemoveContainer" containerID="bce698b0de488d4c1c7a7c287553c45889c7def3cc66e54cde09f6d95b797797" Sep 17 00:51:45.929729 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.929709 2582 scope.go:117] "RemoveContainer" containerID="bce698b0de488d4c1c7a7c287553c45889c7def3cc66e54cde09f6d95b797797" Sep 17 00:51:45.930001 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:51:45.929980 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce698b0de488d4c1c7a7c287553c45889c7def3cc66e54cde09f6d95b797797\": container with ID starting with bce698b0de488d4c1c7a7c287553c45889c7def3cc66e54cde09f6d95b797797 not found: ID does not exist" containerID="bce698b0de488d4c1c7a7c287553c45889c7def3cc66e54cde09f6d95b797797" Sep 17 00:51:45.930044 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.930010 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce698b0de488d4c1c7a7c287553c45889c7def3cc66e54cde09f6d95b797797"} err="failed to get container status \"bce698b0de488d4c1c7a7c287553c45889c7def3cc66e54cde09f6d95b797797\": rpc error: code = NotFound desc = could not find container \"bce698b0de488d4c1c7a7c287553c45889c7def3cc66e54cde09f6d95b797797\": container with ID starting with bce698b0de488d4c1c7a7c287553c45889c7def3cc66e54cde09f6d95b797797 not found: ID does not exist" Sep 17 00:51:45.934933 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.934910 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv"] Sep 17 00:51:45.940549 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:45.940525 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f40754/simulator-w4-f40754-kserve-85f5594c8c-9sszv"] Sep 17 00:51:46.896249 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:46.896213 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:51:47.655948 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:51:47.655914 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08638c74-f335-4aa5-8290-659b4f15496f" path="/var/lib/kubelet/pods/08638c74-f335-4aa5-8290-659b4f15496f/volumes" Sep 17 00:52:01.170195 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.170157 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt"] Sep 17 00:52:01.170659 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.170644 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc785bad-fc9b-4143-8f4a-e773b51153f8" containerName="maas-api" Sep 17 00:52:01.170704 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.170661 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc785bad-fc9b-4143-8f4a-e773b51153f8" containerName="maas-api" Sep 17 00:52:01.170704 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.170685 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e11b1d71-591d-44e4-b926-8b09841a3075" containerName="maas-api" Sep 17 00:52:01.170704 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.170690 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e11b1d71-591d-44e4-b926-8b09841a3075" containerName="maas-api" Sep 17 00:52:01.170704 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.170697 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="221c5b4f-3870-4aa3-8fa4-77a46596ffc2" containerName="maas-api" Sep 17 00:52:01.170704 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.170702 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="221c5b4f-3870-4aa3-8fa4-77a46596ffc2" containerName="maas-api" Sep 17 00:52:01.170857 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.170709 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08638c74-f335-4aa5-8290-659b4f15496f" containerName="main" Sep 17 00:52:01.170857 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.170714 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="08638c74-f335-4aa5-8290-659b4f15496f" containerName="main" Sep 17 00:52:01.170857 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.170780 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc785bad-fc9b-4143-8f4a-e773b51153f8" containerName="maas-api" Sep 17 00:52:01.170857 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.170790 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e11b1d71-591d-44e4-b926-8b09841a3075" containerName="maas-api" Sep 17 00:52:01.170857 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.170796 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="08638c74-f335-4aa5-8290-659b4f15496f" containerName="main" Sep 17 00:52:01.170857 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.170803 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="221c5b4f-3870-4aa3-8fa4-77a46596ffc2" containerName="maas-api" Sep 17 00:52:01.175459 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.175431 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" Sep 17 00:52:01.177328 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.177271 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-c98a88-serving-cert\"" Sep 17 00:52:01.182430 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.182403 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt"] Sep 17 00:52:01.245884 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.245859 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nfwz\" (UniqueName: \"kubernetes.io/projected/9af9636a-7161-485f-9bce-103f53dedc69-kube-api-access-4nfwz\") pod \"maas-api-e2e-explicit-c98a88-85944c575f-8zplt\" (UID: \"9af9636a-7161-485f-9bce-103f53dedc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" Sep 17 00:52:01.246018 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.245896 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9af9636a-7161-485f-9bce-103f53dedc69-maas-api-tls\") pod \"maas-api-e2e-explicit-c98a88-85944c575f-8zplt\" (UID: \"9af9636a-7161-485f-9bce-103f53dedc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" Sep 17 00:52:01.246063 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.246018 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af9636a-7161-485f-9bce-103f53dedc69-openshift-service-ca\") pod \"maas-api-e2e-explicit-c98a88-85944c575f-8zplt\" (UID: \"9af9636a-7161-485f-9bce-103f53dedc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" Sep 17 00:52:01.246063 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.246035 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af9636a-7161-485f-9bce-103f53dedc69-metrics-certs\") pod \"maas-api-e2e-explicit-c98a88-85944c575f-8zplt\" (UID: \"9af9636a-7161-485f-9bce-103f53dedc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" Sep 17 00:52:01.346571 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.346545 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nfwz\" (UniqueName: \"kubernetes.io/projected/9af9636a-7161-485f-9bce-103f53dedc69-kube-api-access-4nfwz\") pod \"maas-api-e2e-explicit-c98a88-85944c575f-8zplt\" (UID: \"9af9636a-7161-485f-9bce-103f53dedc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" Sep 17 00:52:01.346676 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.346576 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9af9636a-7161-485f-9bce-103f53dedc69-maas-api-tls\") pod \"maas-api-e2e-explicit-c98a88-85944c575f-8zplt\" (UID: \"9af9636a-7161-485f-9bce-103f53dedc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" Sep 17 00:52:01.346676 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.346649 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af9636a-7161-485f-9bce-103f53dedc69-openshift-service-ca\") pod \"maas-api-e2e-explicit-c98a88-85944c575f-8zplt\" (UID: \"9af9636a-7161-485f-9bce-103f53dedc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" Sep 17 00:52:01.346676 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.346666 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af9636a-7161-485f-9bce-103f53dedc69-metrics-certs\") pod \"maas-api-e2e-explicit-c98a88-85944c575f-8zplt\" (UID: \"9af9636a-7161-485f-9bce-103f53dedc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" Sep 17 00:52:01.347347 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.347317 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af9636a-7161-485f-9bce-103f53dedc69-openshift-service-ca\") pod \"maas-api-e2e-explicit-c98a88-85944c575f-8zplt\" (UID: \"9af9636a-7161-485f-9bce-103f53dedc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" Sep 17 00:52:01.349489 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.349468 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9af9636a-7161-485f-9bce-103f53dedc69-maas-api-tls\") pod \"maas-api-e2e-explicit-c98a88-85944c575f-8zplt\" (UID: \"9af9636a-7161-485f-9bce-103f53dedc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" Sep 17 00:52:01.349585 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.349468 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af9636a-7161-485f-9bce-103f53dedc69-metrics-certs\") pod \"maas-api-e2e-explicit-c98a88-85944c575f-8zplt\" (UID: \"9af9636a-7161-485f-9bce-103f53dedc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" Sep 17 00:52:01.353243 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.353219 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nfwz\" (UniqueName: \"kubernetes.io/projected/9af9636a-7161-485f-9bce-103f53dedc69-kube-api-access-4nfwz\") pod \"maas-api-e2e-explicit-c98a88-85944c575f-8zplt\" (UID: \"9af9636a-7161-485f-9bce-103f53dedc69\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" Sep 17 00:52:01.487313 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.487251 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" Sep 17 00:52:01.763697 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:01.763578 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt"] Sep 17 00:52:01.766679 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:52:01.766651 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af9636a_7161_485f_9bce_103f53dedc69.slice/crio-7108062809aabb2ce4a0e3d769f4be67e15546fe7db279148882846751d68825 WatchSource:0}: Error finding container 7108062809aabb2ce4a0e3d769f4be67e15546fe7db279148882846751d68825: Status 404 returned error can't find the container with id 7108062809aabb2ce4a0e3d769f4be67e15546fe7db279148882846751d68825 Sep 17 00:52:02.016447 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:02.016362 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" event={"ID":"9af9636a-7161-485f-9bce-103f53dedc69","Type":"ContainerStarted","Data":"7108062809aabb2ce4a0e3d769f4be67e15546fe7db279148882846751d68825"} Sep 17 00:52:03.024268 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:03.024230 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" event={"ID":"9af9636a-7161-485f-9bce-103f53dedc69","Type":"ContainerStarted","Data":"c653231d23b6b4d7d8764a83039af88de4585d6393f99becd4fa1b27df3042da"} Sep 17 00:52:03.024684 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:03.024296 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" Sep 17 00:52:03.039384 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:03.039339 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" podStartSLOduration=1.671538472 podStartE2EDuration="2.039322481s" totalImagesPullingTime="367.784009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:52:03.038295007 +0000 UTC m=+1335.968928495" watchObservedRunningTime="2026-09-17 00:52:03.039322481 +0000 UTC m=+1335.969955968" Sep 17 00:52:06.998540 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:06.998483 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:52:09.035284 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:09.035217 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" Sep 17 00:52:14.428633 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:14.428593 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57"] Sep 17 00:52:14.432666 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:14.432643 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" Sep 17 00:52:14.434617 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:14.434488 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-69d1f859-serving-cert\"" Sep 17 00:52:14.441156 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:14.441130 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57"] Sep 17 00:52:14.569937 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:14.569890 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97lr4\" (UniqueName: \"kubernetes.io/projected/674c1bec-ba65-4d4c-9062-af018b751463-kube-api-access-97lr4\") pod \"maas-api-e2e-mt-69d1f859-548f8f49d5-spd57\" (UID: \"674c1bec-ba65-4d4c-9062-af018b751463\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" Sep 17 00:52:14.570126 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:14.569981 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/674c1bec-ba65-4d4c-9062-af018b751463-metrics-certs\") pod \"maas-api-e2e-mt-69d1f859-548f8f49d5-spd57\" (UID: \"674c1bec-ba65-4d4c-9062-af018b751463\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" Sep 17 00:52:14.570126 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:14.570021 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/674c1bec-ba65-4d4c-9062-af018b751463-maas-api-tls\") pod \"maas-api-e2e-mt-69d1f859-548f8f49d5-spd57\" (UID: \"674c1bec-ba65-4d4c-9062-af018b751463\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" Sep 17 00:52:14.570126 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:14.570041 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/674c1bec-ba65-4d4c-9062-af018b751463-openshift-service-ca\") pod \"maas-api-e2e-mt-69d1f859-548f8f49d5-spd57\" (UID: \"674c1bec-ba65-4d4c-9062-af018b751463\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" Sep 17 00:52:14.670560 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:14.670523 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/674c1bec-ba65-4d4c-9062-af018b751463-metrics-certs\") pod \"maas-api-e2e-mt-69d1f859-548f8f49d5-spd57\" (UID: \"674c1bec-ba65-4d4c-9062-af018b751463\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" Sep 17 00:52:14.670560 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:14.670567 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/674c1bec-ba65-4d4c-9062-af018b751463-maas-api-tls\") pod \"maas-api-e2e-mt-69d1f859-548f8f49d5-spd57\" (UID: \"674c1bec-ba65-4d4c-9062-af018b751463\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" Sep 17 00:52:14.670792 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:14.670587 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/674c1bec-ba65-4d4c-9062-af018b751463-openshift-service-ca\") pod \"maas-api-e2e-mt-69d1f859-548f8f49d5-spd57\" (UID: \"674c1bec-ba65-4d4c-9062-af018b751463\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" Sep 17 00:52:14.670792 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:14.670672 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97lr4\" (UniqueName: \"kubernetes.io/projected/674c1bec-ba65-4d4c-9062-af018b751463-kube-api-access-97lr4\") pod \"maas-api-e2e-mt-69d1f859-548f8f49d5-spd57\" (UID: \"674c1bec-ba65-4d4c-9062-af018b751463\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" Sep 17 00:52:14.670792 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:52:14.670699 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-serving-cert: secret "maas-api-e2e-mt-69d1f859-serving-cert" not found Sep 17 00:52:14.670792 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:52:14.670793 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/674c1bec-ba65-4d4c-9062-af018b751463-maas-api-tls podName:674c1bec-ba65-4d4c-9062-af018b751463 nodeName:}" failed. No retries permitted until 2026-09-17 00:52:15.170770854 +0000 UTC m=+1348.101404333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/674c1bec-ba65-4d4c-9062-af018b751463-maas-api-tls") pod "maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" (UID: "674c1bec-ba65-4d4c-9062-af018b751463") : secret "maas-api-e2e-mt-69d1f859-serving-cert" not found Sep 17 00:52:14.671321 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:14.671297 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/674c1bec-ba65-4d4c-9062-af018b751463-openshift-service-ca\") pod \"maas-api-e2e-mt-69d1f859-548f8f49d5-spd57\" (UID: \"674c1bec-ba65-4d4c-9062-af018b751463\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" Sep 17 00:52:14.673100 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:14.673080 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/674c1bec-ba65-4d4c-9062-af018b751463-metrics-certs\") pod \"maas-api-e2e-mt-69d1f859-548f8f49d5-spd57\" (UID: \"674c1bec-ba65-4d4c-9062-af018b751463\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" Sep 17 00:52:14.678327 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:14.678302 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97lr4\" (UniqueName: \"kubernetes.io/projected/674c1bec-ba65-4d4c-9062-af018b751463-kube-api-access-97lr4\") pod \"maas-api-e2e-mt-69d1f859-548f8f49d5-spd57\" (UID: \"674c1bec-ba65-4d4c-9062-af018b751463\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" Sep 17 00:52:15.177157 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:15.177116 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/674c1bec-ba65-4d4c-9062-af018b751463-maas-api-tls\") pod \"maas-api-e2e-mt-69d1f859-548f8f49d5-spd57\" (UID: \"674c1bec-ba65-4d4c-9062-af018b751463\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" Sep 17 00:52:15.179779 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:15.179746 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/674c1bec-ba65-4d4c-9062-af018b751463-maas-api-tls\") pod \"maas-api-e2e-mt-69d1f859-548f8f49d5-spd57\" (UID: \"674c1bec-ba65-4d4c-9062-af018b751463\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" Sep 17 00:52:15.347461 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:15.347420 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" Sep 17 00:52:15.622764 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:15.622732 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57"] Sep 17 00:52:15.625532 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:52:15.625484 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod674c1bec_ba65_4d4c_9062_af018b751463.slice/crio-ec01978bae030afb7d31929a91b136b4ab5d010cccb2ef42d929a91b981c7cfc WatchSource:0}: Error finding container ec01978bae030afb7d31929a91b136b4ab5d010cccb2ef42d929a91b981c7cfc: Status 404 returned error can't find the container with id ec01978bae030afb7d31929a91b136b4ab5d010cccb2ef42d929a91b981c7cfc Sep 17 00:52:16.093815 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:16.093779 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" event={"ID":"674c1bec-ba65-4d4c-9062-af018b751463","Type":"ContainerStarted","Data":"ec01978bae030afb7d31929a91b136b4ab5d010cccb2ef42d929a91b981c7cfc"} Sep 17 00:52:17.100751 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:17.100716 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" event={"ID":"674c1bec-ba65-4d4c-9062-af018b751463","Type":"ContainerStarted","Data":"c3094a9d9710bc42ae5db4b11b47d0d0543570fc92e72995bba7fba753de81c2"} Sep 17 00:52:17.101139 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:17.100783 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" Sep 17 00:52:17.123406 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:17.123355 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" podStartSLOduration=2.5627437520000003 podStartE2EDuration="3.12333883s" totalImagesPullingTime="560.595078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:52:17.122605886 +0000 UTC m=+1350.053239376" watchObservedRunningTime="2026-09-17 00:52:17.12333883 +0000 UTC m=+1350.053972317" Sep 17 00:52:23.110967 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:23.110927 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" Sep 17 00:52:26.882468 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:26.882422 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5"] Sep 17 00:52:26.886105 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:26.886088 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:26.888822 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:26.888800 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e421ba\"/\"test-model-e421ba-kserve-self-signed-certs\"" Sep 17 00:52:26.888822 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:26.888816 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e421ba\"/\"kube-root-ca.crt\"" Sep 17 00:52:26.888982 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:26.888836 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-e421ba\"/\"openshift-service-ca.crt\"" Sep 17 00:52:26.888982 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:26.888822 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-e421ba\"/\"default-dockercfg-6tt8d\"" Sep 17 00:52:26.896116 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:26.896093 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5"] Sep 17 00:52:26.974134 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:26.974102 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-dshm\") pod \"test-model-e421ba-kserve-854c5d7b4c-8xmx5\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:26.974316 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:26.974150 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-tmp-dir\") pod \"test-model-e421ba-kserve-854c5d7b4c-8xmx5\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:26.974316 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:26.974229 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-tls-certs\") pod \"test-model-e421ba-kserve-854c5d7b4c-8xmx5\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:26.974316 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:26.974250 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-model-cache\") pod \"test-model-e421ba-kserve-854c5d7b4c-8xmx5\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:26.974489 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:26.974359 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-home\") pod \"test-model-e421ba-kserve-854c5d7b4c-8xmx5\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:26.974489 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:26.974390 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5w68\" (UniqueName: \"kubernetes.io/projected/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-kube-api-access-x5w68\") pod \"test-model-e421ba-kserve-854c5d7b4c-8xmx5\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:27.075059 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:27.075014 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-tls-certs\") pod \"test-model-e421ba-kserve-854c5d7b4c-8xmx5\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:27.075214 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:27.075064 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-model-cache\") pod \"test-model-e421ba-kserve-854c5d7b4c-8xmx5\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:27.075214 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:27.075137 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-home\") pod \"test-model-e421ba-kserve-854c5d7b4c-8xmx5\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:27.075214 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:27.075171 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5w68\" (UniqueName: \"kubernetes.io/projected/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-kube-api-access-x5w68\") pod \"test-model-e421ba-kserve-854c5d7b4c-8xmx5\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:27.075368 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:27.075221 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-dshm\") pod \"test-model-e421ba-kserve-854c5d7b4c-8xmx5\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:27.075368 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:27.075262 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-tmp-dir\") pod \"test-model-e421ba-kserve-854c5d7b4c-8xmx5\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:27.075733 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:27.075677 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-home\") pod \"test-model-e421ba-kserve-854c5d7b4c-8xmx5\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:27.075733 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:27.075717 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-tmp-dir\") pod \"test-model-e421ba-kserve-854c5d7b4c-8xmx5\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:27.075972 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:27.075951 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-model-cache\") pod \"test-model-e421ba-kserve-854c5d7b4c-8xmx5\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:27.077661 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:27.077640 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-dshm\") pod \"test-model-e421ba-kserve-854c5d7b4c-8xmx5\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:27.077898 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:27.077883 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-tls-certs\") pod \"test-model-e421ba-kserve-854c5d7b4c-8xmx5\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:27.082125 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:27.082107 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5w68\" (UniqueName: \"kubernetes.io/projected/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-kube-api-access-x5w68\") pod \"test-model-e421ba-kserve-854c5d7b4c-8xmx5\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:27.196194 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:27.196065 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:27.478511 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:27.478473 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5"] Sep 17 00:52:27.480979 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:52:27.480950 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd27af4e7_e46d_4489_bba4_60ef41cbcdd3.slice/crio-b861cd8681f5b6342d3796fb72927284bcf060c7381df3013aefeaa34faeff55 WatchSource:0}: Error finding container b861cd8681f5b6342d3796fb72927284bcf060c7381df3013aefeaa34faeff55: Status 404 returned error can't find the container with id b861cd8681f5b6342d3796fb72927284bcf060c7381df3013aefeaa34faeff55 Sep 17 00:52:28.161817 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:28.161779 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" event={"ID":"d27af4e7-e46d-4489-bba4-60ef41cbcdd3","Type":"ContainerStarted","Data":"7b454b145c5fe1254eca593e442f6fd5a7024dc3a088993fba232143c7d310d3"} Sep 17 00:52:28.162260 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:28.161826 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" event={"ID":"d27af4e7-e46d-4489-bba4-60ef41cbcdd3","Type":"ContainerStarted","Data":"b861cd8681f5b6342d3796fb72927284bcf060c7381df3013aefeaa34faeff55"} Sep 17 00:52:28.177871 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:28.177818 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" podStartSLOduration=2.17780067 podStartE2EDuration="2.17780067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:52:28.175946688 +0000 UTC m=+1361.106580175" watchObservedRunningTime="2026-09-17 00:52:28.17780067 +0000 UTC m=+1361.108434158" Sep 17 00:52:32.913077 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:32.913029 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:52:37.196454 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:37.196413 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:37.196929 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:37.196473 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:37.209604 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:37.209575 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:37.221540 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:37.221510 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:52:41.198480 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.198442 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7"] Sep 17 00:52:41.203893 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.203873 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:41.206517 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.206474 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c98a88\"/\"openshift-service-ca.crt\"" Sep 17 00:52:41.206517 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.206476 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c98a88\"/\"explicit-model-c98a88-kserve-self-signed-certs\"" Sep 17 00:52:41.206699 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.206476 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c98a88\"/\"default-dockercfg-5bdxb\"" Sep 17 00:52:41.206699 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.206530 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c98a88\"/\"kube-root-ca.crt\"" Sep 17 00:52:41.209774 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.209753 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7"] Sep 17 00:52:41.313190 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.313152 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-model-cache\") pod \"explicit-model-c98a88-kserve-6844b44787-qsnr7\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:41.313334 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.313205 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-home\") pod \"explicit-model-c98a88-kserve-6844b44787-qsnr7\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:41.313334 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.313231 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-tmp-dir\") pod \"explicit-model-c98a88-kserve-6844b44787-qsnr7\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:41.313334 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.313324 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqhw6\" (UniqueName: \"kubernetes.io/projected/fbfc271f-a808-4f3c-a703-66e4f1ce8250-kube-api-access-wqhw6\") pod \"explicit-model-c98a88-kserve-6844b44787-qsnr7\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:41.313458 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.313365 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbfc271f-a808-4f3c-a703-66e4f1ce8250-tls-certs\") pod \"explicit-model-c98a88-kserve-6844b44787-qsnr7\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:41.313458 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.313446 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-dshm\") pod \"explicit-model-c98a88-kserve-6844b44787-qsnr7\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:41.414543 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.414485 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbfc271f-a808-4f3c-a703-66e4f1ce8250-tls-certs\") pod \"explicit-model-c98a88-kserve-6844b44787-qsnr7\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:41.414712 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.414575 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-dshm\") pod \"explicit-model-c98a88-kserve-6844b44787-qsnr7\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:41.414712 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.414625 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-model-cache\") pod \"explicit-model-c98a88-kserve-6844b44787-qsnr7\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:41.414712 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.414659 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-home\") pod \"explicit-model-c98a88-kserve-6844b44787-qsnr7\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:41.414712 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.414681 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-tmp-dir\") pod \"explicit-model-c98a88-kserve-6844b44787-qsnr7\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:41.414884 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.414724 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqhw6\" (UniqueName: \"kubernetes.io/projected/fbfc271f-a808-4f3c-a703-66e4f1ce8250-kube-api-access-wqhw6\") pod \"explicit-model-c98a88-kserve-6844b44787-qsnr7\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:41.415132 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.415093 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-tmp-dir\") pod \"explicit-model-c98a88-kserve-6844b44787-qsnr7\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:41.415132 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.415111 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-home\") pod \"explicit-model-c98a88-kserve-6844b44787-qsnr7\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:41.415335 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.415132 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-model-cache\") pod \"explicit-model-c98a88-kserve-6844b44787-qsnr7\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:41.417187 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.417161 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-dshm\") pod \"explicit-model-c98a88-kserve-6844b44787-qsnr7\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:41.417274 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.417214 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbfc271f-a808-4f3c-a703-66e4f1ce8250-tls-certs\") pod \"explicit-model-c98a88-kserve-6844b44787-qsnr7\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:41.422388 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.422365 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqhw6\" (UniqueName: \"kubernetes.io/projected/fbfc271f-a808-4f3c-a703-66e4f1ce8250-kube-api-access-wqhw6\") pod \"explicit-model-c98a88-kserve-6844b44787-qsnr7\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:41.516108 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.516022 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:41.794250 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:41.794211 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7"] Sep 17 00:52:41.796751 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:52:41.796725 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbfc271f_a808_4f3c_a703_66e4f1ce8250.slice/crio-655fded4a38d67b5047cca9d8087bf67210fd118cd7c3d1ec23257256d5b0428 WatchSource:0}: Error finding container 655fded4a38d67b5047cca9d8087bf67210fd118cd7c3d1ec23257256d5b0428: Status 404 returned error can't find the container with id 655fded4a38d67b5047cca9d8087bf67210fd118cd7c3d1ec23257256d5b0428 Sep 17 00:52:42.237063 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:42.236973 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" event={"ID":"fbfc271f-a808-4f3c-a703-66e4f1ce8250","Type":"ContainerStarted","Data":"973f3c1cbfe238ca1f0deb35bd07f226b8aef5a950f3202577493eda0c20def9"} Sep 17 00:52:42.237063 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:42.237016 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" event={"ID":"fbfc271f-a808-4f3c-a703-66e4f1ce8250","Type":"ContainerStarted","Data":"655fded4a38d67b5047cca9d8087bf67210fd118cd7c3d1ec23257256d5b0428"} Sep 17 00:52:42.254287 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:42.254230 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" podStartSLOduration=1.254215706 podStartE2EDuration="1.254215706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:52:42.25173555 +0000 UTC m=+1375.182369037" watchObservedRunningTime="2026-09-17 00:52:42.254215706 +0000 UTC m=+1375.184849193" Sep 17 00:52:51.517202 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:51.517157 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:51.517722 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:51.517223 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:51.529904 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:51.529875 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:52.305074 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:52.305038 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:52:58.517053 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:58.517015 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57"] Sep 17 00:52:58.517416 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:58.517250 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" podUID="674c1bec-ba65-4d4c-9062-af018b751463" containerName="maas-api" containerID="cri-o://c3094a9d9710bc42ae5db4b11b47d0d0543570fc92e72995bba7fba753de81c2" gracePeriod=30 Sep 17 00:52:58.769410 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:58.769349 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" Sep 17 00:52:58.892381 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:58.892277 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/674c1bec-ba65-4d4c-9062-af018b751463-openshift-service-ca\") pod \"674c1bec-ba65-4d4c-9062-af018b751463\" (UID: \"674c1bec-ba65-4d4c-9062-af018b751463\") " Sep 17 00:52:58.892607 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:58.892394 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97lr4\" (UniqueName: \"kubernetes.io/projected/674c1bec-ba65-4d4c-9062-af018b751463-kube-api-access-97lr4\") pod \"674c1bec-ba65-4d4c-9062-af018b751463\" (UID: \"674c1bec-ba65-4d4c-9062-af018b751463\") " Sep 17 00:52:58.892607 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:58.892432 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/674c1bec-ba65-4d4c-9062-af018b751463-metrics-certs\") pod \"674c1bec-ba65-4d4c-9062-af018b751463\" (UID: \"674c1bec-ba65-4d4c-9062-af018b751463\") " Sep 17 00:52:58.892757 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:58.892663 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/674c1bec-ba65-4d4c-9062-af018b751463-maas-api-tls\") pod \"674c1bec-ba65-4d4c-9062-af018b751463\" (UID: \"674c1bec-ba65-4d4c-9062-af018b751463\") " Sep 17 00:52:58.893083 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:58.893035 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/674c1bec-ba65-4d4c-9062-af018b751463-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "674c1bec-ba65-4d4c-9062-af018b751463" (UID: "674c1bec-ba65-4d4c-9062-af018b751463"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 00:52:58.893197 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:58.893159 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/674c1bec-ba65-4d4c-9062-af018b751463-openshift-service-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:52:58.895126 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:58.895101 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/674c1bec-ba65-4d4c-9062-af018b751463-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "674c1bec-ba65-4d4c-9062-af018b751463" (UID: "674c1bec-ba65-4d4c-9062-af018b751463"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:52:58.895323 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:58.895301 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/674c1bec-ba65-4d4c-9062-af018b751463-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "674c1bec-ba65-4d4c-9062-af018b751463" (UID: "674c1bec-ba65-4d4c-9062-af018b751463"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:52:58.895323 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:58.895314 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674c1bec-ba65-4d4c-9062-af018b751463-kube-api-access-97lr4" (OuterVolumeSpecName: "kube-api-access-97lr4") pod "674c1bec-ba65-4d4c-9062-af018b751463" (UID: "674c1bec-ba65-4d4c-9062-af018b751463"). InnerVolumeSpecName "kube-api-access-97lr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:52:58.993914 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:58.993881 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97lr4\" (UniqueName: \"kubernetes.io/projected/674c1bec-ba65-4d4c-9062-af018b751463-kube-api-access-97lr4\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:52:58.993914 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:58.993908 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/674c1bec-ba65-4d4c-9062-af018b751463-metrics-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:52:58.993914 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:58.993917 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/674c1bec-ba65-4d4c-9062-af018b751463-maas-api-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:52:59.331572 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:59.331533 2582 generic.go:356] "Generic (PLEG): container finished" podID="674c1bec-ba65-4d4c-9062-af018b751463" containerID="c3094a9d9710bc42ae5db4b11b47d0d0543570fc92e72995bba7fba753de81c2" exitCode=0 Sep 17 00:52:59.331748 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:59.331579 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" Sep 17 00:52:59.331748 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:59.331589 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" event={"ID":"674c1bec-ba65-4d4c-9062-af018b751463","Type":"ContainerDied","Data":"c3094a9d9710bc42ae5db4b11b47d0d0543570fc92e72995bba7fba753de81c2"} Sep 17 00:52:59.331748 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:59.331672 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57" event={"ID":"674c1bec-ba65-4d4c-9062-af018b751463","Type":"ContainerDied","Data":"ec01978bae030afb7d31929a91b136b4ab5d010cccb2ef42d929a91b981c7cfc"} Sep 17 00:52:59.331748 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:59.331695 2582 scope.go:117] "RemoveContainer" containerID="c3094a9d9710bc42ae5db4b11b47d0d0543570fc92e72995bba7fba753de81c2" Sep 17 00:52:59.341596 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:59.341577 2582 scope.go:117] "RemoveContainer" containerID="c3094a9d9710bc42ae5db4b11b47d0d0543570fc92e72995bba7fba753de81c2" Sep 17 00:52:59.341863 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:52:59.341847 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3094a9d9710bc42ae5db4b11b47d0d0543570fc92e72995bba7fba753de81c2\": container with ID starting with c3094a9d9710bc42ae5db4b11b47d0d0543570fc92e72995bba7fba753de81c2 not found: ID does not exist" containerID="c3094a9d9710bc42ae5db4b11b47d0d0543570fc92e72995bba7fba753de81c2" Sep 17 00:52:59.341907 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:59.341871 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3094a9d9710bc42ae5db4b11b47d0d0543570fc92e72995bba7fba753de81c2"} err="failed to get container status \"c3094a9d9710bc42ae5db4b11b47d0d0543570fc92e72995bba7fba753de81c2\": rpc error: code = NotFound desc = could not find container \"c3094a9d9710bc42ae5db4b11b47d0d0543570fc92e72995bba7fba753de81c2\": container with ID starting with c3094a9d9710bc42ae5db4b11b47d0d0543570fc92e72995bba7fba753de81c2 not found: ID does not exist" Sep 17 00:52:59.355510 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:59.355474 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57"] Sep 17 00:52:59.359181 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:59.359161 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69d1f859-548f8f49d5-spd57"] Sep 17 00:52:59.650525 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:52:59.650425 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="674c1bec-ba65-4d4c-9062-af018b751463" path="/var/lib/kubelet/pods/674c1bec-ba65-4d4c-9062-af018b751463/volumes" Sep 17 00:53:01.090635 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:01.090601 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:53:02.295392 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:02.295358 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7"] Sep 17 00:53:12.897107 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:12.897067 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt"] Sep 17 00:53:12.897618 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:12.897284 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" podUID="9af9636a-7161-485f-9bce-103f53dedc69" containerName="maas-api" containerID="cri-o://c653231d23b6b4d7d8764a83039af88de4585d6393f99becd4fa1b27df3042da" gracePeriod=30 Sep 17 00:53:13.147885 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.147827 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" Sep 17 00:53:13.239250 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.239228 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9af9636a-7161-485f-9bce-103f53dedc69-maas-api-tls\") pod \"9af9636a-7161-485f-9bce-103f53dedc69\" (UID: \"9af9636a-7161-485f-9bce-103f53dedc69\") " Sep 17 00:53:13.239403 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.239309 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nfwz\" (UniqueName: \"kubernetes.io/projected/9af9636a-7161-485f-9bce-103f53dedc69-kube-api-access-4nfwz\") pod \"9af9636a-7161-485f-9bce-103f53dedc69\" (UID: \"9af9636a-7161-485f-9bce-103f53dedc69\") " Sep 17 00:53:13.239403 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.239356 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af9636a-7161-485f-9bce-103f53dedc69-openshift-service-ca\") pod \"9af9636a-7161-485f-9bce-103f53dedc69\" (UID: \"9af9636a-7161-485f-9bce-103f53dedc69\") " Sep 17 00:53:13.239403 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.239396 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af9636a-7161-485f-9bce-103f53dedc69-metrics-certs\") pod \"9af9636a-7161-485f-9bce-103f53dedc69\" (UID: \"9af9636a-7161-485f-9bce-103f53dedc69\") " Sep 17 00:53:13.239806 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.239779 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af9636a-7161-485f-9bce-103f53dedc69-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9af9636a-7161-485f-9bce-103f53dedc69" (UID: "9af9636a-7161-485f-9bce-103f53dedc69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 00:53:13.241544 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.241492 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9af9636a-7161-485f-9bce-103f53dedc69-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9af9636a-7161-485f-9bce-103f53dedc69" (UID: "9af9636a-7161-485f-9bce-103f53dedc69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:53:13.241638 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.241555 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af9636a-7161-485f-9bce-103f53dedc69-kube-api-access-4nfwz" (OuterVolumeSpecName: "kube-api-access-4nfwz") pod "9af9636a-7161-485f-9bce-103f53dedc69" (UID: "9af9636a-7161-485f-9bce-103f53dedc69"). InnerVolumeSpecName "kube-api-access-4nfwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:53:13.241638 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.241560 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9af9636a-7161-485f-9bce-103f53dedc69-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9af9636a-7161-485f-9bce-103f53dedc69" (UID: "9af9636a-7161-485f-9bce-103f53dedc69"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:53:13.341053 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.341028 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af9636a-7161-485f-9bce-103f53dedc69-metrics-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:53:13.341053 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.341051 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9af9636a-7161-485f-9bce-103f53dedc69-maas-api-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:53:13.341209 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.341061 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nfwz\" (UniqueName: \"kubernetes.io/projected/9af9636a-7161-485f-9bce-103f53dedc69-kube-api-access-4nfwz\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:53:13.341209 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.341069 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9af9636a-7161-485f-9bce-103f53dedc69-openshift-service-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:53:13.415642 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.415558 2582 generic.go:356] "Generic (PLEG): container finished" podID="9af9636a-7161-485f-9bce-103f53dedc69" containerID="c653231d23b6b4d7d8764a83039af88de4585d6393f99becd4fa1b27df3042da" exitCode=0 Sep 17 00:53:13.415642 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.415602 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" Sep 17 00:53:13.415794 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.415646 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" event={"ID":"9af9636a-7161-485f-9bce-103f53dedc69","Type":"ContainerDied","Data":"c653231d23b6b4d7d8764a83039af88de4585d6393f99becd4fa1b27df3042da"} Sep 17 00:53:13.415794 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.415696 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt" event={"ID":"9af9636a-7161-485f-9bce-103f53dedc69","Type":"ContainerDied","Data":"7108062809aabb2ce4a0e3d769f4be67e15546fe7db279148882846751d68825"} Sep 17 00:53:13.415794 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.415718 2582 scope.go:117] "RemoveContainer" containerID="c653231d23b6b4d7d8764a83039af88de4585d6393f99becd4fa1b27df3042da" Sep 17 00:53:13.425221 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.425205 2582 scope.go:117] "RemoveContainer" containerID="c653231d23b6b4d7d8764a83039af88de4585d6393f99becd4fa1b27df3042da" Sep 17 00:53:13.425552 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:53:13.425527 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c653231d23b6b4d7d8764a83039af88de4585d6393f99becd4fa1b27df3042da\": container with ID starting with c653231d23b6b4d7d8764a83039af88de4585d6393f99becd4fa1b27df3042da not found: ID does not exist" containerID="c653231d23b6b4d7d8764a83039af88de4585d6393f99becd4fa1b27df3042da" Sep 17 00:53:13.425645 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.425558 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c653231d23b6b4d7d8764a83039af88de4585d6393f99becd4fa1b27df3042da"} err="failed to get container status \"c653231d23b6b4d7d8764a83039af88de4585d6393f99becd4fa1b27df3042da\": rpc error: code = NotFound desc = could not find container \"c653231d23b6b4d7d8764a83039af88de4585d6393f99becd4fa1b27df3042da\": container with ID starting with c653231d23b6b4d7d8764a83039af88de4585d6393f99becd4fa1b27df3042da not found: ID does not exist" Sep 17 00:53:13.435041 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.435021 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt"] Sep 17 00:53:13.436824 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.436805 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c98a88-85944c575f-8zplt"] Sep 17 00:53:13.650610 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:13.650579 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9af9636a-7161-485f-9bce-103f53dedc69" path="/var/lib/kubelet/pods/9af9636a-7161-485f-9bce-103f53dedc69/volumes" Sep 17 00:53:17.316234 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.316182 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" podUID="fbfc271f-a808-4f3c-a703-66e4f1ce8250" containerName="main" containerID="cri-o://973f3c1cbfe238ca1f0deb35bd07f226b8aef5a950f3202577493eda0c20def9" gracePeriod=45 Sep 17 00:53:17.439448 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.439414 2582 generic.go:356] "Generic (PLEG): container finished" podID="fbfc271f-a808-4f3c-a703-66e4f1ce8250" containerID="973f3c1cbfe238ca1f0deb35bd07f226b8aef5a950f3202577493eda0c20def9" exitCode=0 Sep 17 00:53:17.439577 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.439463 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" event={"ID":"fbfc271f-a808-4f3c-a703-66e4f1ce8250","Type":"ContainerDied","Data":"973f3c1cbfe238ca1f0deb35bd07f226b8aef5a950f3202577493eda0c20def9"} Sep 17 00:53:17.569641 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.569585 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:53:17.678911 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.678887 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-model-cache\") pod \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " Sep 17 00:53:17.679054 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.678955 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-tmp-dir\") pod \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " Sep 17 00:53:17.679054 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.678977 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-dshm\") pod \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " Sep 17 00:53:17.679054 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.679045 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbfc271f-a808-4f3c-a703-66e4f1ce8250-tls-certs\") pod \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " Sep 17 00:53:17.679165 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.679069 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-home\") pod \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " Sep 17 00:53:17.679165 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.679097 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqhw6\" (UniqueName: \"kubernetes.io/projected/fbfc271f-a808-4f3c-a703-66e4f1ce8250-kube-api-access-wqhw6\") pod \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\" (UID: \"fbfc271f-a808-4f3c-a703-66e4f1ce8250\") " Sep 17 00:53:17.679248 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.679170 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-model-cache" (OuterVolumeSpecName: "model-cache") pod "fbfc271f-a808-4f3c-a703-66e4f1ce8250" (UID: "fbfc271f-a808-4f3c-a703-66e4f1ce8250"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:53:17.679248 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.679200 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fbfc271f-a808-4f3c-a703-66e4f1ce8250" (UID: "fbfc271f-a808-4f3c-a703-66e4f1ce8250"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:53:17.679414 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.679389 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-home" (OuterVolumeSpecName: "home") pod "fbfc271f-a808-4f3c-a703-66e4f1ce8250" (UID: "fbfc271f-a808-4f3c-a703-66e4f1ce8250"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:53:17.679483 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.679417 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-tmp-dir\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:53:17.679483 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.679435 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-model-cache\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:53:17.681485 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.681459 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbfc271f-a808-4f3c-a703-66e4f1ce8250-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fbfc271f-a808-4f3c-a703-66e4f1ce8250" (UID: "fbfc271f-a808-4f3c-a703-66e4f1ce8250"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:53:17.681614 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.681517 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-dshm" (OuterVolumeSpecName: "dshm") pod "fbfc271f-a808-4f3c-a703-66e4f1ce8250" (UID: "fbfc271f-a808-4f3c-a703-66e4f1ce8250"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:53:17.681614 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.681547 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbfc271f-a808-4f3c-a703-66e4f1ce8250-kube-api-access-wqhw6" (OuterVolumeSpecName: "kube-api-access-wqhw6") pod "fbfc271f-a808-4f3c-a703-66e4f1ce8250" (UID: "fbfc271f-a808-4f3c-a703-66e4f1ce8250"). InnerVolumeSpecName "kube-api-access-wqhw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:53:17.780187 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.780152 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-dshm\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:53:17.780187 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.780178 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbfc271f-a808-4f3c-a703-66e4f1ce8250-tls-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:53:17.780187 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.780193 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbfc271f-a808-4f3c-a703-66e4f1ce8250-home\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:53:17.780389 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:17.780205 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqhw6\" (UniqueName: \"kubernetes.io/projected/fbfc271f-a808-4f3c-a703-66e4f1ce8250-kube-api-access-wqhw6\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:53:18.446171 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:18.446143 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" Sep 17 00:53:18.446608 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:18.446177 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7" event={"ID":"fbfc271f-a808-4f3c-a703-66e4f1ce8250","Type":"ContainerDied","Data":"655fded4a38d67b5047cca9d8087bf67210fd118cd7c3d1ec23257256d5b0428"} Sep 17 00:53:18.446608 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:18.446226 2582 scope.go:117] "RemoveContainer" containerID="973f3c1cbfe238ca1f0deb35bd07f226b8aef5a950f3202577493eda0c20def9" Sep 17 00:53:18.466338 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:18.466316 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7"] Sep 17 00:53:18.470101 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:18.470070 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-c98a88/explicit-model-c98a88-kserve-6844b44787-qsnr7"] Sep 17 00:53:19.650930 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:19.650888 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbfc271f-a808-4f3c-a703-66e4f1ce8250" path="/var/lib/kubelet/pods/fbfc271f-a808-4f3c-a703-66e4f1ce8250/volumes" Sep 17 00:53:24.738870 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.738832 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm"] Sep 17 00:53:24.739521 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.739269 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9af9636a-7161-485f-9bce-103f53dedc69" containerName="maas-api" Sep 17 00:53:24.739521 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.739284 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9af9636a-7161-485f-9bce-103f53dedc69" containerName="maas-api" Sep 17 00:53:24.739521 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.739296 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="674c1bec-ba65-4d4c-9062-af018b751463" containerName="maas-api" Sep 17 00:53:24.739521 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.739301 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="674c1bec-ba65-4d4c-9062-af018b751463" containerName="maas-api" Sep 17 00:53:24.739521 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.739315 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbfc271f-a808-4f3c-a703-66e4f1ce8250" containerName="main" Sep 17 00:53:24.739521 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.739320 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbfc271f-a808-4f3c-a703-66e4f1ce8250" containerName="main" Sep 17 00:53:24.739521 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.739396 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbfc271f-a808-4f3c-a703-66e4f1ce8250" containerName="main" Sep 17 00:53:24.739521 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.739404 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9af9636a-7161-485f-9bce-103f53dedc69" containerName="maas-api" Sep 17 00:53:24.739521 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.739415 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="674c1bec-ba65-4d4c-9062-af018b751463" containerName="maas-api" Sep 17 00:53:24.742770 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.742754 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" Sep 17 00:53:24.744578 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.744560 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-929f3a-serving-cert\"" Sep 17 00:53:24.754839 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.754802 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm"] Sep 17 00:53:24.800927 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.800893 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:53:24.849575 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.849530 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a325eb70-884a-4177-82c6-7e20a20822d8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm\" (UID: \"a325eb70-884a-4177-82c6-7e20a20822d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" Sep 17 00:53:24.849779 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.849591 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a325eb70-884a-4177-82c6-7e20a20822d8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm\" (UID: \"a325eb70-884a-4177-82c6-7e20a20822d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" Sep 17 00:53:24.849779 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.849614 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a325eb70-884a-4177-82c6-7e20a20822d8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm\" (UID: \"a325eb70-884a-4177-82c6-7e20a20822d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" Sep 17 00:53:24.849779 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.849716 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gj79\" (UniqueName: \"kubernetes.io/projected/a325eb70-884a-4177-82c6-7e20a20822d8-kube-api-access-4gj79\") pod \"maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm\" (UID: \"a325eb70-884a-4177-82c6-7e20a20822d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" Sep 17 00:53:24.950515 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.950464 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gj79\" (UniqueName: \"kubernetes.io/projected/a325eb70-884a-4177-82c6-7e20a20822d8-kube-api-access-4gj79\") pod \"maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm\" (UID: \"a325eb70-884a-4177-82c6-7e20a20822d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" Sep 17 00:53:24.950695 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.950638 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a325eb70-884a-4177-82c6-7e20a20822d8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm\" (UID: \"a325eb70-884a-4177-82c6-7e20a20822d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" Sep 17 00:53:24.950695 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.950686 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a325eb70-884a-4177-82c6-7e20a20822d8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm\" (UID: \"a325eb70-884a-4177-82c6-7e20a20822d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" Sep 17 00:53:24.950801 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.950719 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a325eb70-884a-4177-82c6-7e20a20822d8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm\" (UID: \"a325eb70-884a-4177-82c6-7e20a20822d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" Sep 17 00:53:24.951469 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.951442 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a325eb70-884a-4177-82c6-7e20a20822d8-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm\" (UID: \"a325eb70-884a-4177-82c6-7e20a20822d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" Sep 17 00:53:24.953878 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.953853 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a325eb70-884a-4177-82c6-7e20a20822d8-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm\" (UID: \"a325eb70-884a-4177-82c6-7e20a20822d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" Sep 17 00:53:24.953982 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.953903 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a325eb70-884a-4177-82c6-7e20a20822d8-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm\" (UID: \"a325eb70-884a-4177-82c6-7e20a20822d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" Sep 17 00:53:24.960452 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:24.960425 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gj79\" (UniqueName: \"kubernetes.io/projected/a325eb70-884a-4177-82c6-7e20a20822d8-kube-api-access-4gj79\") pod \"maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm\" (UID: \"a325eb70-884a-4177-82c6-7e20a20822d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" Sep 17 00:53:25.054072 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:25.053742 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" Sep 17 00:53:25.372439 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:25.372408 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm"] Sep 17 00:53:25.375618 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:53:25.375584 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda325eb70_884a_4177_82c6_7e20a20822d8.slice/crio-dc538696c89d76dde5a8ed3270cd2688d5cd5416fe1aacc8407f2f735567adbc WatchSource:0}: Error finding container dc538696c89d76dde5a8ed3270cd2688d5cd5416fe1aacc8407f2f735567adbc: Status 404 returned error can't find the container with id dc538696c89d76dde5a8ed3270cd2688d5cd5416fe1aacc8407f2f735567adbc Sep 17 00:53:25.377617 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:25.377598 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 00:53:25.486875 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:25.486835 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" event={"ID":"a325eb70-884a-4177-82c6-7e20a20822d8","Type":"ContainerStarted","Data":"dc538696c89d76dde5a8ed3270cd2688d5cd5416fe1aacc8407f2f735567adbc"} Sep 17 00:53:26.495087 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:26.495054 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" event={"ID":"a325eb70-884a-4177-82c6-7e20a20822d8","Type":"ContainerStarted","Data":"7caed02a01944480101077cbb163da010c0dbb6c0f93e20fd630ccf730212192"} Sep 17 00:53:26.495465 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:26.495187 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" Sep 17 00:53:26.509512 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:26.509450 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" podStartSLOduration=2.04267915 podStartE2EDuration="2.509431566s" totalImagesPullingTime="466.752416ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:53:26.508334332 +0000 UTC m=+1419.438967820" watchObservedRunningTime="2026-09-17 00:53:26.509431566 +0000 UTC m=+1419.440065054" Sep 17 00:53:32.506785 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:32.506757 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" Sep 17 00:53:45.090293 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:45.090207 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:53:54.432812 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:54.432768 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz"] Sep 17 00:53:54.437974 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:54.437948 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" Sep 17 00:53:54.439808 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:54.439786 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-93017d-serving-cert\"" Sep 17 00:53:54.444751 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:54.444724 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz"] Sep 17 00:53:54.536290 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:54.536257 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78d09de2-5e5d-4aa8-a557-61df130c7e3d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz\" (UID: \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" Sep 17 00:53:54.536473 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:54.536323 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx52q\" (UniqueName: \"kubernetes.io/projected/78d09de2-5e5d-4aa8-a557-61df130c7e3d-kube-api-access-zx52q\") pod \"maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz\" (UID: \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" Sep 17 00:53:54.536473 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:54.536398 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78d09de2-5e5d-4aa8-a557-61df130c7e3d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz\" (UID: \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" Sep 17 00:53:54.536473 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:54.536438 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78d09de2-5e5d-4aa8-a557-61df130c7e3d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz\" (UID: \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" Sep 17 00:53:54.637338 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:54.637293 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx52q\" (UniqueName: \"kubernetes.io/projected/78d09de2-5e5d-4aa8-a557-61df130c7e3d-kube-api-access-zx52q\") pod \"maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz\" (UID: \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" Sep 17 00:53:54.637338 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:54.637348 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78d09de2-5e5d-4aa8-a557-61df130c7e3d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz\" (UID: \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" Sep 17 00:53:54.637574 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:54.637384 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78d09de2-5e5d-4aa8-a557-61df130c7e3d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz\" (UID: \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" Sep 17 00:53:54.637574 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:54.637476 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78d09de2-5e5d-4aa8-a557-61df130c7e3d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz\" (UID: \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" Sep 17 00:53:54.637654 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:53:54.637640 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-serving-cert: secret "maas-api-e2e-shared-b-w2-93017d-serving-cert" not found Sep 17 00:53:54.637733 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:53:54.637722 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78d09de2-5e5d-4aa8-a557-61df130c7e3d-maas-api-tls podName:78d09de2-5e5d-4aa8-a557-61df130c7e3d nodeName:}" failed. No retries permitted until 2026-09-17 00:53:55.137700236 +0000 UTC m=+1448.068333705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/78d09de2-5e5d-4aa8-a557-61df130c7e3d-maas-api-tls") pod "maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" (UID: "78d09de2-5e5d-4aa8-a557-61df130c7e3d") : secret "maas-api-e2e-shared-b-w2-93017d-serving-cert" not found Sep 17 00:53:54.638085 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:54.638064 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78d09de2-5e5d-4aa8-a557-61df130c7e3d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz\" (UID: \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" Sep 17 00:53:54.640153 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:54.640121 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78d09de2-5e5d-4aa8-a557-61df130c7e3d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz\" (UID: \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" Sep 17 00:53:54.647150 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:54.647132 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx52q\" (UniqueName: \"kubernetes.io/projected/78d09de2-5e5d-4aa8-a557-61df130c7e3d-kube-api-access-zx52q\") pod \"maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz\" (UID: \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" Sep 17 00:53:55.143151 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:55.143100 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78d09de2-5e5d-4aa8-a557-61df130c7e3d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz\" (UID: \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" Sep 17 00:53:55.145823 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:55.145798 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78d09de2-5e5d-4aa8-a557-61df130c7e3d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz\" (UID: \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" Sep 17 00:53:55.352315 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:55.352280 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" Sep 17 00:53:55.627049 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:55.627022 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz"] Sep 17 00:53:55.629457 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:53:55.629433 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d09de2_5e5d_4aa8_a557_61df130c7e3d.slice/crio-93c844197f59b4606366bf6f0bdfc00454a3d0e0117c4ab6e591249fb1f8657b WatchSource:0}: Error finding container 93c844197f59b4606366bf6f0bdfc00454a3d0e0117c4ab6e591249fb1f8657b: Status 404 returned error can't find the container with id 93c844197f59b4606366bf6f0bdfc00454a3d0e0117c4ab6e591249fb1f8657b Sep 17 00:53:55.657102 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:55.657023 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" event={"ID":"78d09de2-5e5d-4aa8-a557-61df130c7e3d","Type":"ContainerStarted","Data":"93c844197f59b4606366bf6f0bdfc00454a3d0e0117c4ab6e591249fb1f8657b"} Sep 17 00:53:56.663689 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:56.663654 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" event={"ID":"78d09de2-5e5d-4aa8-a557-61df130c7e3d","Type":"ContainerStarted","Data":"4c88f4e64feef7fd5403565676ae35359845e78147668219364aafe6a4c0c9cf"} Sep 17 00:53:56.664068 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:56.663846 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" Sep 17 00:53:56.681418 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:53:56.681361 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" podStartSLOduration=2.369774643 podStartE2EDuration="2.681345359s" totalImagesPullingTime="311.570716ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:53:56.680362411 +0000 UTC m=+1449.610995923" watchObservedRunningTime="2026-09-17 00:53:56.681345359 +0000 UTC m=+1449.611978846" Sep 17 00:54:02.674219 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:02.674190 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" Sep 17 00:54:06.990727 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:06.990680 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:54:10.673976 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.673939 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt"] Sep 17 00:54:10.678019 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.678000 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:10.680623 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.680601 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-5a9565\"/\"default-dockercfg-m8pjw\"" Sep 17 00:54:10.680727 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.680604 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-5a9565\"/\"noten-model-5a9565-kserve-self-signed-certs\"" Sep 17 00:54:10.680727 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.680604 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-5a9565\"/\"openshift-service-ca.crt\"" Sep 17 00:54:10.680727 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.680612 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-5a9565\"/\"kube-root-ca.crt\"" Sep 17 00:54:10.686860 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.686834 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt"] Sep 17 00:54:10.806799 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.806751 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-model-cache\") pod \"noten-model-5a9565-kserve-fb659677f-958jt\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:10.806799 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.806806 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-home\") pod \"noten-model-5a9565-kserve-fb659677f-958jt\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:10.807024 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.806907 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz92k\" (UniqueName: \"kubernetes.io/projected/f5d06746-0303-44bf-9bdd-26bf456b0779-kube-api-access-gz92k\") pod \"noten-model-5a9565-kserve-fb659677f-958jt\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:10.807024 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.806957 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-dshm\") pod \"noten-model-5a9565-kserve-fb659677f-958jt\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:10.807024 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.807020 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5d06746-0303-44bf-9bdd-26bf456b0779-tls-certs\") pod \"noten-model-5a9565-kserve-fb659677f-958jt\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:10.807126 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.807095 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-tmp-dir\") pod \"noten-model-5a9565-kserve-fb659677f-958jt\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:10.908071 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.908037 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-tmp-dir\") pod \"noten-model-5a9565-kserve-fb659677f-958jt\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:10.908269 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.908086 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-model-cache\") pod \"noten-model-5a9565-kserve-fb659677f-958jt\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:10.908269 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.908204 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-home\") pod \"noten-model-5a9565-kserve-fb659677f-958jt\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:10.908393 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.908294 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz92k\" (UniqueName: \"kubernetes.io/projected/f5d06746-0303-44bf-9bdd-26bf456b0779-kube-api-access-gz92k\") pod \"noten-model-5a9565-kserve-fb659677f-958jt\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:10.908393 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.908352 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-dshm\") pod \"noten-model-5a9565-kserve-fb659677f-958jt\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:10.908493 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.908421 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5d06746-0303-44bf-9bdd-26bf456b0779-tls-certs\") pod \"noten-model-5a9565-kserve-fb659677f-958jt\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:10.908493 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.908453 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-model-cache\") pod \"noten-model-5a9565-kserve-fb659677f-958jt\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:10.908493 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.908482 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-tmp-dir\") pod \"noten-model-5a9565-kserve-fb659677f-958jt\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:10.908667 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.908597 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-home\") pod \"noten-model-5a9565-kserve-fb659677f-958jt\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:10.910719 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.910699 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-dshm\") pod \"noten-model-5a9565-kserve-fb659677f-958jt\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:10.910995 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.910977 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5d06746-0303-44bf-9bdd-26bf456b0779-tls-certs\") pod \"noten-model-5a9565-kserve-fb659677f-958jt\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:10.916333 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.916310 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz92k\" (UniqueName: \"kubernetes.io/projected/f5d06746-0303-44bf-9bdd-26bf456b0779-kube-api-access-gz92k\") pod \"noten-model-5a9565-kserve-fb659677f-958jt\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:10.988249 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:10.988161 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:11.260936 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:11.260891 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt"] Sep 17 00:54:11.263251 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:54:11.263205 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5d06746_0303_44bf_9bdd_26bf456b0779.slice/crio-45e290ae42c9294f09a1ac725a9a311a08f811cb88294bc696795720ded693cd WatchSource:0}: Error finding container 45e290ae42c9294f09a1ac725a9a311a08f811cb88294bc696795720ded693cd: Status 404 returned error can't find the container with id 45e290ae42c9294f09a1ac725a9a311a08f811cb88294bc696795720ded693cd Sep 17 00:54:11.741915 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:11.741876 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" event={"ID":"f5d06746-0303-44bf-9bdd-26bf456b0779","Type":"ContainerStarted","Data":"6a3caee63da0bb7236fe4478ab738a6b3fe9a90fa1af04ee68a6bf46847278df"} Sep 17 00:54:11.742299 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:11.741930 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" event={"ID":"f5d06746-0303-44bf-9bdd-26bf456b0779","Type":"ContainerStarted","Data":"45e290ae42c9294f09a1ac725a9a311a08f811cb88294bc696795720ded693cd"} Sep 17 00:54:11.757453 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:11.757398 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" podStartSLOduration=1.757384243 podStartE2EDuration="1.757384243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:54:11.755108984 +0000 UTC m=+1464.685742471" watchObservedRunningTime="2026-09-17 00:54:11.757384243 +0000 UTC m=+1464.688017729" Sep 17 00:54:20.988600 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:20.988565 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:20.989007 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:20.988619 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:21.001044 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:21.001019 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:21.439663 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:21.439629 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7693-748860"] Sep 17 00:54:21.443318 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:21.443303 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-748860" Sep 17 00:54:21.445181 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:21.445160 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bd8qq\"" Sep 17 00:54:21.448984 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:21.448959 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-748860"] Sep 17 00:54:21.512726 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:21.512690 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjcl7\" (UniqueName: \"kubernetes.io/projected/73f1398e-caa5-4993-9d9d-a8023b4ef855-kube-api-access-pjcl7\") pod \"test-curl-7693-748860\" (UID: \"73f1398e-caa5-4993-9d9d-a8023b4ef855\") " pod="opendatahub/test-curl-7693-748860" Sep 17 00:54:21.614386 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:21.614344 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjcl7\" (UniqueName: \"kubernetes.io/projected/73f1398e-caa5-4993-9d9d-a8023b4ef855-kube-api-access-pjcl7\") pod \"test-curl-7693-748860\" (UID: \"73f1398e-caa5-4993-9d9d-a8023b4ef855\") " pod="opendatahub/test-curl-7693-748860" Sep 17 00:54:21.623619 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:21.623595 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjcl7\" (UniqueName: \"kubernetes.io/projected/73f1398e-caa5-4993-9d9d-a8023b4ef855-kube-api-access-pjcl7\") pod \"test-curl-7693-748860\" (UID: \"73f1398e-caa5-4993-9d9d-a8023b4ef855\") " pod="opendatahub/test-curl-7693-748860" Sep 17 00:54:21.755944 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:21.755866 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-748860" Sep 17 00:54:21.813216 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:21.813185 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:22.026936 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:22.026909 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-748860"] Sep 17 00:54:22.029610 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:54:22.029584 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73f1398e_caa5_4993_9d9d_a8023b4ef855.slice/crio-27fd7c683c340f30e86f5404d6c8c76800c4be4f7dea9cf27aa85b6442054a7e WatchSource:0}: Error finding container 27fd7c683c340f30e86f5404d6c8c76800c4be4f7dea9cf27aa85b6442054a7e: Status 404 returned error can't find the container with id 27fd7c683c340f30e86f5404d6c8c76800c4be4f7dea9cf27aa85b6442054a7e Sep 17 00:54:22.805941 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:22.805907 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-748860" event={"ID":"73f1398e-caa5-4993-9d9d-a8023b4ef855","Type":"ContainerStarted","Data":"530caae6f3fd0bc048d03ad57f437026c7c0aaea2f058531b8a6a5fd7c599dc7"} Sep 17 00:54:22.805941 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:22.805946 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-748860" event={"ID":"73f1398e-caa5-4993-9d9d-a8023b4ef855","Type":"ContainerStarted","Data":"27fd7c683c340f30e86f5404d6c8c76800c4be4f7dea9cf27aa85b6442054a7e"} Sep 17 00:54:22.818609 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:22.818559 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7693-748860" podStartSLOduration=1.818539441 podStartE2EDuration="1.818539441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:54:22.81801086 +0000 UTC m=+1475.748644347" watchObservedRunningTime="2026-09-17 00:54:22.818539441 +0000 UTC m=+1475.749172929" Sep 17 00:54:23.413229 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:23.413194 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7693-748860"] Sep 17 00:54:23.417366 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:23.417331 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7693-748860"] Sep 17 00:54:23.520515 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:23.520416 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7693-c1b93b"] Sep 17 00:54:23.520854 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:23.520840 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73f1398e-caa5-4993-9d9d-a8023b4ef855" containerName="test-curl-7693-748860" Sep 17 00:54:23.520897 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:23.520856 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="73f1398e-caa5-4993-9d9d-a8023b4ef855" containerName="test-curl-7693-748860" Sep 17 00:54:23.520951 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:23.520942 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="73f1398e-caa5-4993-9d9d-a8023b4ef855" containerName="test-curl-7693-748860" Sep 17 00:54:23.524185 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:23.524169 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-c1b93b" Sep 17 00:54:23.530184 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:23.530162 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-c1b93b"] Sep 17 00:54:23.636205 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:23.636173 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps6mv\" (UniqueName: \"kubernetes.io/projected/4ac9b70f-f9e9-400b-8ba5-04327a153cc9-kube-api-access-ps6mv\") pod \"test-curl-7693-c1b93b\" (UID: \"4ac9b70f-f9e9-400b-8ba5-04327a153cc9\") " pod="opendatahub/test-curl-7693-c1b93b" Sep 17 00:54:23.737617 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:23.737582 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps6mv\" (UniqueName: \"kubernetes.io/projected/4ac9b70f-f9e9-400b-8ba5-04327a153cc9-kube-api-access-ps6mv\") pod \"test-curl-7693-c1b93b\" (UID: \"4ac9b70f-f9e9-400b-8ba5-04327a153cc9\") " pod="opendatahub/test-curl-7693-c1b93b" Sep 17 00:54:23.745984 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:23.745961 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps6mv\" (UniqueName: \"kubernetes.io/projected/4ac9b70f-f9e9-400b-8ba5-04327a153cc9-kube-api-access-ps6mv\") pod \"test-curl-7693-c1b93b\" (UID: \"4ac9b70f-f9e9-400b-8ba5-04327a153cc9\") " pod="opendatahub/test-curl-7693-c1b93b" Sep 17 00:54:23.811479 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:54:23.811454 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7693-748860" container="test-curl-7693-748860" Sep 17 00:54:23.835182 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:23.835158 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-c1b93b" Sep 17 00:54:24.105416 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:24.105392 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-c1b93b"] Sep 17 00:54:24.106945 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:54:24.106917 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ac9b70f_f9e9_400b_8ba5_04327a153cc9.slice/crio-a7fdb27d459489864c8213042ba830420933fc4d757fb1d2e9459558a3653a61 WatchSource:0}: Error finding container a7fdb27d459489864c8213042ba830420933fc4d757fb1d2e9459558a3653a61: Status 404 returned error can't find the container with id a7fdb27d459489864c8213042ba830420933fc4d757fb1d2e9459558a3653a61 Sep 17 00:54:24.818567 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:24.818531 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-c1b93b" event={"ID":"4ac9b70f-f9e9-400b-8ba5-04327a153cc9","Type":"ContainerStarted","Data":"379f0db55394b7b806d86119c64f281782550883dbcf3329928cc1d76c2f19e7"} Sep 17 00:54:24.818567 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:24.818575 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-c1b93b" event={"ID":"4ac9b70f-f9e9-400b-8ba5-04327a153cc9","Type":"ContainerStarted","Data":"a7fdb27d459489864c8213042ba830420933fc4d757fb1d2e9459558a3653a61"} Sep 17 00:54:24.819071 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:24.818665 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7693-748860" podUID="73f1398e-caa5-4993-9d9d-a8023b4ef855" containerName="test-curl-7693-748860" containerID="cri-o://530caae6f3fd0bc048d03ad57f437026c7c0aaea2f058531b8a6a5fd7c599dc7" gracePeriod=2 Sep 17 00:54:24.834684 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:24.834631 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7693-c1b93b" podStartSLOduration=1.834615873 podStartE2EDuration="1.834615873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:54:24.832922397 +0000 UTC m=+1477.763555886" watchObservedRunningTime="2026-09-17 00:54:24.834615873 +0000 UTC m=+1477.765249357" Sep 17 00:54:25.381025 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:25.380994 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt"] Sep 17 00:54:25.432813 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:25.432780 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7693-c1b93b"] Sep 17 00:54:25.437837 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:25.437806 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7693-c1b93b"] Sep 17 00:54:25.546801 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:25.546761 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7693-6a9bd5"] Sep 17 00:54:25.547292 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:25.547270 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ac9b70f-f9e9-400b-8ba5-04327a153cc9" containerName="test-curl-7693-c1b93b" Sep 17 00:54:25.547392 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:25.547295 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ac9b70f-f9e9-400b-8ba5-04327a153cc9" containerName="test-curl-7693-c1b93b" Sep 17 00:54:25.547449 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:25.547438 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ac9b70f-f9e9-400b-8ba5-04327a153cc9" containerName="test-curl-7693-c1b93b" Sep 17 00:54:25.550698 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:25.550678 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-6a9bd5" Sep 17 00:54:25.555519 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:25.555483 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-6a9bd5"] Sep 17 00:54:25.656698 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:25.656670 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwq4p\" (UniqueName: \"kubernetes.io/projected/f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf-kube-api-access-wwq4p\") pod \"test-curl-7693-6a9bd5\" (UID: \"f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf\") " pod="opendatahub/test-curl-7693-6a9bd5" Sep 17 00:54:25.758032 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:25.757985 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwq4p\" (UniqueName: \"kubernetes.io/projected/f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf-kube-api-access-wwq4p\") pod \"test-curl-7693-6a9bd5\" (UID: \"f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf\") " pod="opendatahub/test-curl-7693-6a9bd5" Sep 17 00:54:25.765357 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:25.765335 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwq4p\" (UniqueName: \"kubernetes.io/projected/f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf-kube-api-access-wwq4p\") pod \"test-curl-7693-6a9bd5\" (UID: \"f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf\") " pod="opendatahub/test-curl-7693-6a9bd5" Sep 17 00:54:25.824608 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:54:25.824532 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7693-c1b93b" container="test-curl-7693-c1b93b" Sep 17 00:54:25.861606 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:25.861566 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-6a9bd5" Sep 17 00:54:26.133946 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:26.133917 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-6a9bd5"] Sep 17 00:54:26.136737 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:54:26.136706 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3d56b05_d8aa_4f76_9b01_485bcb2d7bcf.slice/crio-610434414f936a405ffeaa95d374af6f55bc0e6acd7e25c9701c20b56ee5e204 WatchSource:0}: Error finding container 610434414f936a405ffeaa95d374af6f55bc0e6acd7e25c9701c20b56ee5e204: Status 404 returned error can't find the container with id 610434414f936a405ffeaa95d374af6f55bc0e6acd7e25c9701c20b56ee5e204 Sep 17 00:54:26.588611 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:26.588578 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5"] Sep 17 00:54:26.832694 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:26.832658 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-6a9bd5" event={"ID":"f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf","Type":"ContainerStarted","Data":"2374461bf8f6a572d12b4b2dad6c05404fd3050680dcb6025b80082ebbf687dd"} Sep 17 00:54:26.833039 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:26.832712 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-6a9bd5" event={"ID":"f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf","Type":"ContainerStarted","Data":"610434414f936a405ffeaa95d374af6f55bc0e6acd7e25c9701c20b56ee5e204"} Sep 17 00:54:26.833039 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:26.832937 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7693-c1b93b" podUID="4ac9b70f-f9e9-400b-8ba5-04327a153cc9" containerName="test-curl-7693-c1b93b" containerID="cri-o://379f0db55394b7b806d86119c64f281782550883dbcf3329928cc1d76c2f19e7" gracePeriod=2 Sep 17 00:54:26.846967 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:26.846920 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7693-6a9bd5" podStartSLOduration=1.846902965 podStartE2EDuration="1.846902965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:54:26.844370087 +0000 UTC m=+1479.775003575" watchObservedRunningTime="2026-09-17 00:54:26.846902965 +0000 UTC m=+1479.777536452" Sep 17 00:54:26.965886 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:26.965860 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-748860" Sep 17 00:54:27.073071 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.073044 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjcl7\" (UniqueName: \"kubernetes.io/projected/73f1398e-caa5-4993-9d9d-a8023b4ef855-kube-api-access-pjcl7\") pod \"73f1398e-caa5-4993-9d9d-a8023b4ef855\" (UID: \"73f1398e-caa5-4993-9d9d-a8023b4ef855\") " Sep 17 00:54:27.075072 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.075047 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f1398e-caa5-4993-9d9d-a8023b4ef855-kube-api-access-pjcl7" (OuterVolumeSpecName: "kube-api-access-pjcl7") pod "73f1398e-caa5-4993-9d9d-a8023b4ef855" (UID: "73f1398e-caa5-4993-9d9d-a8023b4ef855"). InnerVolumeSpecName "kube-api-access-pjcl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:54:27.174289 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.174213 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjcl7\" (UniqueName: \"kubernetes.io/projected/73f1398e-caa5-4993-9d9d-a8023b4ef855-kube-api-access-pjcl7\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:54:27.422234 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.422189 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7693-6a9bd5"] Sep 17 00:54:27.425954 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.425879 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7693-6a9bd5"] Sep 17 00:54:27.534239 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.534203 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7693-04f7c3"] Sep 17 00:54:27.534653 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.534640 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf" containerName="test-curl-7693-6a9bd5" Sep 17 00:54:27.534710 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.534655 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf" containerName="test-curl-7693-6a9bd5" Sep 17 00:54:27.534745 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.534730 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf" containerName="test-curl-7693-6a9bd5" Sep 17 00:54:27.537796 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.537781 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-04f7c3" Sep 17 00:54:27.543150 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.543123 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-04f7c3"] Sep 17 00:54:27.650929 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.650900 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73f1398e-caa5-4993-9d9d-a8023b4ef855" path="/var/lib/kubelet/pods/73f1398e-caa5-4993-9d9d-a8023b4ef855/volumes" Sep 17 00:54:27.679986 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.679902 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grgrn\" (UniqueName: \"kubernetes.io/projected/a2e68ce4-81aa-4faa-bafa-9499102413b2-kube-api-access-grgrn\") pod \"test-curl-7693-04f7c3\" (UID: \"a2e68ce4-81aa-4faa-bafa-9499102413b2\") " pod="opendatahub/test-curl-7693-04f7c3" Sep 17 00:54:27.781587 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.781544 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grgrn\" (UniqueName: \"kubernetes.io/projected/a2e68ce4-81aa-4faa-bafa-9499102413b2-kube-api-access-grgrn\") pod \"test-curl-7693-04f7c3\" (UID: \"a2e68ce4-81aa-4faa-bafa-9499102413b2\") " pod="opendatahub/test-curl-7693-04f7c3" Sep 17 00:54:27.788806 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.788777 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grgrn\" (UniqueName: \"kubernetes.io/projected/a2e68ce4-81aa-4faa-bafa-9499102413b2-kube-api-access-grgrn\") pod \"test-curl-7693-04f7c3\" (UID: \"a2e68ce4-81aa-4faa-bafa-9499102413b2\") " pod="opendatahub/test-curl-7693-04f7c3" Sep 17 00:54:27.840101 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.840067 2582 generic.go:356] "Generic (PLEG): container finished" podID="73f1398e-caa5-4993-9d9d-a8023b4ef855" containerID="530caae6f3fd0bc048d03ad57f437026c7c0aaea2f058531b8a6a5fd7c599dc7" exitCode=137 Sep 17 00:54:27.840588 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.840116 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-748860" Sep 17 00:54:27.840588 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.840167 2582 scope.go:117] "RemoveContainer" containerID="530caae6f3fd0bc048d03ad57f437026c7c0aaea2f058531b8a6a5fd7c599dc7" Sep 17 00:54:27.840588 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:54:27.840299 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7693-6a9bd5" container="test-curl-7693-6a9bd5" Sep 17 00:54:27.848774 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.848753 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-04f7c3" Sep 17 00:54:27.849432 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.849414 2582 scope.go:117] "RemoveContainer" containerID="530caae6f3fd0bc048d03ad57f437026c7c0aaea2f058531b8a6a5fd7c599dc7" Sep 17 00:54:27.849760 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:54:27.849742 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"530caae6f3fd0bc048d03ad57f437026c7c0aaea2f058531b8a6a5fd7c599dc7\": container with ID starting with 530caae6f3fd0bc048d03ad57f437026c7c0aaea2f058531b8a6a5fd7c599dc7 not found: ID does not exist" containerID="530caae6f3fd0bc048d03ad57f437026c7c0aaea2f058531b8a6a5fd7c599dc7" Sep 17 00:54:27.849813 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:27.849768 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"530caae6f3fd0bc048d03ad57f437026c7c0aaea2f058531b8a6a5fd7c599dc7"} err="failed to get container status \"530caae6f3fd0bc048d03ad57f437026c7c0aaea2f058531b8a6a5fd7c599dc7\": rpc error: code = NotFound desc = could not find container \"530caae6f3fd0bc048d03ad57f437026c7c0aaea2f058531b8a6a5fd7c599dc7\": container with ID starting with 530caae6f3fd0bc048d03ad57f437026c7c0aaea2f058531b8a6a5fd7c599dc7 not found: ID does not exist" Sep 17 00:54:28.119057 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:28.119033 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7693-04f7c3"] Sep 17 00:54:28.121977 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:54:28.121943 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2e68ce4_81aa_4faa_bafa_9499102413b2.slice/crio-f9a8ffb2f8a65adcb02541068b4bf7d24316b173b4bf95fe0e646213f073717b WatchSource:0}: Error finding container f9a8ffb2f8a65adcb02541068b4bf7d24316b173b4bf95fe0e646213f073717b: Status 404 returned error can't find the container with id f9a8ffb2f8a65adcb02541068b4bf7d24316b173b4bf95fe0e646213f073717b Sep 17 00:54:28.848176 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:28.848143 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-04f7c3" event={"ID":"a2e68ce4-81aa-4faa-bafa-9499102413b2","Type":"ContainerStarted","Data":"665f45c4d5b6f07fe519af3066d95b717f7a4b30364ec964e30ccfa6d777cbf3"} Sep 17 00:54:28.848472 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:28.848196 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7693-04f7c3" event={"ID":"a2e68ce4-81aa-4faa-bafa-9499102413b2","Type":"ContainerStarted","Data":"f9a8ffb2f8a65adcb02541068b4bf7d24316b173b4bf95fe0e646213f073717b"} Sep 17 00:54:28.848472 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:28.848328 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7693-6a9bd5" podUID="f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf" containerName="test-curl-7693-6a9bd5" containerID="cri-o://2374461bf8f6a572d12b4b2dad6c05404fd3050680dcb6025b80082ebbf687dd" gracePeriod=2 Sep 17 00:54:28.862490 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:28.862441 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7693-04f7c3" podStartSLOduration=1.862427746 podStartE2EDuration="1.862427746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:54:28.860396792 +0000 UTC m=+1481.791030278" watchObservedRunningTime="2026-09-17 00:54:28.862427746 +0000 UTC m=+1481.793061233" Sep 17 00:54:29.010185 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:29.010159 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-c1b93b" Sep 17 00:54:29.093700 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:29.093635 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps6mv\" (UniqueName: \"kubernetes.io/projected/4ac9b70f-f9e9-400b-8ba5-04327a153cc9-kube-api-access-ps6mv\") pod \"4ac9b70f-f9e9-400b-8ba5-04327a153cc9\" (UID: \"4ac9b70f-f9e9-400b-8ba5-04327a153cc9\") " Sep 17 00:54:29.095668 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:29.095647 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac9b70f-f9e9-400b-8ba5-04327a153cc9-kube-api-access-ps6mv" (OuterVolumeSpecName: "kube-api-access-ps6mv") pod "4ac9b70f-f9e9-400b-8ba5-04327a153cc9" (UID: "4ac9b70f-f9e9-400b-8ba5-04327a153cc9"). InnerVolumeSpecName "kube-api-access-ps6mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:54:29.194544 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:29.194491 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps6mv\" (UniqueName: \"kubernetes.io/projected/4ac9b70f-f9e9-400b-8ba5-04327a153cc9-kube-api-access-ps6mv\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:54:29.436642 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:29.436552 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7693-04f7c3"] Sep 17 00:54:29.439652 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:29.439621 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7693-04f7c3"] Sep 17 00:54:29.650444 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:29.650410 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ac9b70f-f9e9-400b-8ba5-04327a153cc9" path="/var/lib/kubelet/pods/4ac9b70f-f9e9-400b-8ba5-04327a153cc9/volumes" Sep 17 00:54:29.854976 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:29.854940 2582 generic.go:356] "Generic (PLEG): container finished" podID="4ac9b70f-f9e9-400b-8ba5-04327a153cc9" containerID="379f0db55394b7b806d86119c64f281782550883dbcf3329928cc1d76c2f19e7" exitCode=137 Sep 17 00:54:29.855376 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:29.854980 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-c1b93b" Sep 17 00:54:29.855376 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:29.855040 2582 scope.go:117] "RemoveContainer" containerID="379f0db55394b7b806d86119c64f281782550883dbcf3329928cc1d76c2f19e7" Sep 17 00:54:29.855376 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:54:29.855244 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7693-04f7c3" container="test-curl-7693-04f7c3" Sep 17 00:54:29.863869 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:29.863850 2582 scope.go:117] "RemoveContainer" containerID="379f0db55394b7b806d86119c64f281782550883dbcf3329928cc1d76c2f19e7" Sep 17 00:54:29.864125 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:54:29.864108 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"379f0db55394b7b806d86119c64f281782550883dbcf3329928cc1d76c2f19e7\": container with ID starting with 379f0db55394b7b806d86119c64f281782550883dbcf3329928cc1d76c2f19e7 not found: ID does not exist" containerID="379f0db55394b7b806d86119c64f281782550883dbcf3329928cc1d76c2f19e7" Sep 17 00:54:29.864176 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:29.864142 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379f0db55394b7b806d86119c64f281782550883dbcf3329928cc1d76c2f19e7"} err="failed to get container status \"379f0db55394b7b806d86119c64f281782550883dbcf3329928cc1d76c2f19e7\": rpc error: code = NotFound desc = could not find container \"379f0db55394b7b806d86119c64f281782550883dbcf3329928cc1d76c2f19e7\": container with ID starting with 379f0db55394b7b806d86119c64f281782550883dbcf3329928cc1d76c2f19e7 not found: ID does not exist" Sep 17 00:54:30.863067 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:30.863034 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7693-04f7c3" podUID="a2e68ce4-81aa-4faa-bafa-9499102413b2" containerName="test-curl-7693-04f7c3" containerID="cri-o://665f45c4d5b6f07fe519af3066d95b717f7a4b30364ec964e30ccfa6d777cbf3" gracePeriod=2 Sep 17 00:54:30.993455 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:30.993429 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-6a9bd5" Sep 17 00:54:31.114515 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:31.114403 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwq4p\" (UniqueName: \"kubernetes.io/projected/f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf-kube-api-access-wwq4p\") pod \"f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf\" (UID: \"f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf\") " Sep 17 00:54:31.116638 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:31.116607 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf-kube-api-access-wwq4p" (OuterVolumeSpecName: "kube-api-access-wwq4p") pod "f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf" (UID: "f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf"). InnerVolumeSpecName "kube-api-access-wwq4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:54:31.215301 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:31.215270 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwq4p\" (UniqueName: \"kubernetes.io/projected/f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf-kube-api-access-wwq4p\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:54:31.651260 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:31.651229 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf" path="/var/lib/kubelet/pods/f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf/volumes" Sep 17 00:54:31.869426 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:31.869390 2582 generic.go:356] "Generic (PLEG): container finished" podID="f3d56b05-d8aa-4f76-9b01-485bcb2d7bcf" containerID="2374461bf8f6a572d12b4b2dad6c05404fd3050680dcb6025b80082ebbf687dd" exitCode=137 Sep 17 00:54:31.869861 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:31.869428 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-6a9bd5" Sep 17 00:54:31.869861 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:31.869458 2582 scope.go:117] "RemoveContainer" containerID="2374461bf8f6a572d12b4b2dad6c05404fd3050680dcb6025b80082ebbf687dd" Sep 17 00:54:31.878302 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:31.878283 2582 scope.go:117] "RemoveContainer" containerID="2374461bf8f6a572d12b4b2dad6c05404fd3050680dcb6025b80082ebbf687dd" Sep 17 00:54:31.878592 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:54:31.878574 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2374461bf8f6a572d12b4b2dad6c05404fd3050680dcb6025b80082ebbf687dd\": container with ID starting with 2374461bf8f6a572d12b4b2dad6c05404fd3050680dcb6025b80082ebbf687dd not found: ID does not exist" containerID="2374461bf8f6a572d12b4b2dad6c05404fd3050680dcb6025b80082ebbf687dd" Sep 17 00:54:31.878642 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:31.878599 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2374461bf8f6a572d12b4b2dad6c05404fd3050680dcb6025b80082ebbf687dd"} err="failed to get container status \"2374461bf8f6a572d12b4b2dad6c05404fd3050680dcb6025b80082ebbf687dd\": rpc error: code = NotFound desc = could not find container \"2374461bf8f6a572d12b4b2dad6c05404fd3050680dcb6025b80082ebbf687dd\": container with ID starting with 2374461bf8f6a572d12b4b2dad6c05404fd3050680dcb6025b80082ebbf687dd not found: ID does not exist" Sep 17 00:54:33.058905 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:33.058878 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-04f7c3" Sep 17 00:54:33.136327 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:33.136250 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grgrn\" (UniqueName: \"kubernetes.io/projected/a2e68ce4-81aa-4faa-bafa-9499102413b2-kube-api-access-grgrn\") pod \"a2e68ce4-81aa-4faa-bafa-9499102413b2\" (UID: \"a2e68ce4-81aa-4faa-bafa-9499102413b2\") " Sep 17 00:54:33.138463 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:33.138435 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e68ce4-81aa-4faa-bafa-9499102413b2-kube-api-access-grgrn" (OuterVolumeSpecName: "kube-api-access-grgrn") pod "a2e68ce4-81aa-4faa-bafa-9499102413b2" (UID: "a2e68ce4-81aa-4faa-bafa-9499102413b2"). InnerVolumeSpecName "kube-api-access-grgrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:54:33.237655 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:33.237620 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grgrn\" (UniqueName: \"kubernetes.io/projected/a2e68ce4-81aa-4faa-bafa-9499102413b2-kube-api-access-grgrn\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:54:33.292743 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:33.292709 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:54:33.653827 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:33.653786 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2e68ce4-81aa-4faa-bafa-9499102413b2" path="/var/lib/kubelet/pods/a2e68ce4-81aa-4faa-bafa-9499102413b2/volumes" Sep 17 00:54:33.882857 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:33.882824 2582 generic.go:356] "Generic (PLEG): container finished" podID="a2e68ce4-81aa-4faa-bafa-9499102413b2" containerID="665f45c4d5b6f07fe519af3066d95b717f7a4b30364ec964e30ccfa6d777cbf3" exitCode=137 Sep 17 00:54:33.883027 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:33.882889 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7693-04f7c3" Sep 17 00:54:33.883027 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:33.882896 2582 scope.go:117] "RemoveContainer" containerID="665f45c4d5b6f07fe519af3066d95b717f7a4b30364ec964e30ccfa6d777cbf3" Sep 17 00:54:33.892075 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:33.892052 2582 scope.go:117] "RemoveContainer" containerID="665f45c4d5b6f07fe519af3066d95b717f7a4b30364ec964e30ccfa6d777cbf3" Sep 17 00:54:33.892364 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:54:33.892346 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"665f45c4d5b6f07fe519af3066d95b717f7a4b30364ec964e30ccfa6d777cbf3\": container with ID starting with 665f45c4d5b6f07fe519af3066d95b717f7a4b30364ec964e30ccfa6d777cbf3 not found: ID does not exist" containerID="665f45c4d5b6f07fe519af3066d95b717f7a4b30364ec964e30ccfa6d777cbf3" Sep 17 00:54:33.892411 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:33.892376 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"665f45c4d5b6f07fe519af3066d95b717f7a4b30364ec964e30ccfa6d777cbf3"} err="failed to get container status \"665f45c4d5b6f07fe519af3066d95b717f7a4b30364ec964e30ccfa6d777cbf3\": rpc error: code = NotFound desc = could not find container \"665f45c4d5b6f07fe519af3066d95b717f7a4b30364ec964e30ccfa6d777cbf3\": container with ID starting with 665f45c4d5b6f07fe519af3066d95b717f7a4b30364ec964e30ccfa6d777cbf3 not found: ID does not exist" Sep 17 00:54:37.631912 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:37.631881 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68"] Sep 17 00:54:37.632458 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:37.632438 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2e68ce4-81aa-4faa-bafa-9499102413b2" containerName="test-curl-7693-04f7c3" Sep 17 00:54:37.632560 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:37.632460 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2e68ce4-81aa-4faa-bafa-9499102413b2" containerName="test-curl-7693-04f7c3" Sep 17 00:54:37.632627 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:37.632577 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2e68ce4-81aa-4faa-bafa-9499102413b2" containerName="test-curl-7693-04f7c3" Sep 17 00:54:37.637302 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:37.637280 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" Sep 17 00:54:37.639296 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:37.639276 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c7c845e1-serving-cert\"" Sep 17 00:54:37.644347 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:37.644143 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68"] Sep 17 00:54:37.782109 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:37.782073 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59297598-6e57-463c-85b4-d824c657de1d-openshift-service-ca\") pod \"maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68\" (UID: \"59297598-6e57-463c-85b4-d824c657de1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" Sep 17 00:54:37.782288 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:37.782125 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59297598-6e57-463c-85b4-d824c657de1d-metrics-certs\") pod \"maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68\" (UID: \"59297598-6e57-463c-85b4-d824c657de1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" Sep 17 00:54:37.782288 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:37.782154 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clvms\" (UniqueName: \"kubernetes.io/projected/59297598-6e57-463c-85b4-d824c657de1d-kube-api-access-clvms\") pod \"maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68\" (UID: \"59297598-6e57-463c-85b4-d824c657de1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" Sep 17 00:54:37.782288 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:37.782254 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59297598-6e57-463c-85b4-d824c657de1d-maas-api-tls\") pod \"maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68\" (UID: \"59297598-6e57-463c-85b4-d824c657de1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" Sep 17 00:54:37.883632 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:37.883533 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59297598-6e57-463c-85b4-d824c657de1d-openshift-service-ca\") pod \"maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68\" (UID: \"59297598-6e57-463c-85b4-d824c657de1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" Sep 17 00:54:37.883632 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:37.883621 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59297598-6e57-463c-85b4-d824c657de1d-metrics-certs\") pod \"maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68\" (UID: \"59297598-6e57-463c-85b4-d824c657de1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" Sep 17 00:54:37.883863 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:37.883684 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clvms\" (UniqueName: \"kubernetes.io/projected/59297598-6e57-463c-85b4-d824c657de1d-kube-api-access-clvms\") pod \"maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68\" (UID: \"59297598-6e57-463c-85b4-d824c657de1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" Sep 17 00:54:37.883863 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:37.883778 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59297598-6e57-463c-85b4-d824c657de1d-maas-api-tls\") pod \"maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68\" (UID: \"59297598-6e57-463c-85b4-d824c657de1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" Sep 17 00:54:37.883947 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:54:37.883881 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-serving-cert: secret "maas-api-e2e-mt-c7c845e1-serving-cert" not found Sep 17 00:54:37.883986 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:54:37.883952 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59297598-6e57-463c-85b4-d824c657de1d-maas-api-tls podName:59297598-6e57-463c-85b4-d824c657de1d nodeName:}" failed. No retries permitted until 2026-09-17 00:54:38.383931513 +0000 UTC m=+1491.314565002 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/59297598-6e57-463c-85b4-d824c657de1d-maas-api-tls") pod "maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" (UID: "59297598-6e57-463c-85b4-d824c657de1d") : secret "maas-api-e2e-mt-c7c845e1-serving-cert" not found Sep 17 00:54:37.884149 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:37.884131 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59297598-6e57-463c-85b4-d824c657de1d-openshift-service-ca\") pod \"maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68\" (UID: \"59297598-6e57-463c-85b4-d824c657de1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" Sep 17 00:54:37.886216 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:37.886196 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59297598-6e57-463c-85b4-d824c657de1d-metrics-certs\") pod \"maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68\" (UID: \"59297598-6e57-463c-85b4-d824c657de1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" Sep 17 00:54:37.891729 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:37.891705 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clvms\" (UniqueName: \"kubernetes.io/projected/59297598-6e57-463c-85b4-d824c657de1d-kube-api-access-clvms\") pod \"maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68\" (UID: \"59297598-6e57-463c-85b4-d824c657de1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" Sep 17 00:54:38.389706 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:38.389674 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59297598-6e57-463c-85b4-d824c657de1d-maas-api-tls\") pod \"maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68\" (UID: \"59297598-6e57-463c-85b4-d824c657de1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" Sep 17 00:54:38.392236 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:38.392213 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59297598-6e57-463c-85b4-d824c657de1d-maas-api-tls\") pod \"maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68\" (UID: \"59297598-6e57-463c-85b4-d824c657de1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" Sep 17 00:54:38.551060 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:38.551027 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" Sep 17 00:54:38.827102 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:38.827081 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68"] Sep 17 00:54:38.829355 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:54:38.829330 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59297598_6e57_463c_85b4_d824c657de1d.slice/crio-107097a0d1d29888ae049af95f98e3f163c4105a9fe91a22643ec16d151bec03 WatchSource:0}: Error finding container 107097a0d1d29888ae049af95f98e3f163c4105a9fe91a22643ec16d151bec03: Status 404 returned error can't find the container with id 107097a0d1d29888ae049af95f98e3f163c4105a9fe91a22643ec16d151bec03 Sep 17 00:54:38.912823 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:38.912790 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" event={"ID":"59297598-6e57-463c-85b4-d824c657de1d","Type":"ContainerStarted","Data":"107097a0d1d29888ae049af95f98e3f163c4105a9fe91a22643ec16d151bec03"} Sep 17 00:54:39.920060 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:39.920021 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" event={"ID":"59297598-6e57-463c-85b4-d824c657de1d","Type":"ContainerStarted","Data":"7fa47b4d4e888af984d02b9ab3ac8338d6da396b0706dd2f51bb3e4a16aea37c"} Sep 17 00:54:39.920463 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:39.920159 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" Sep 17 00:54:39.940211 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:39.940089 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" podStartSLOduration=2.510772554 podStartE2EDuration="2.940070005s" totalImagesPullingTime="429.297451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:54:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:54:39.939767765 +0000 UTC m=+1492.870401253" watchObservedRunningTime="2026-09-17 00:54:39.940070005 +0000 UTC m=+1492.870703493" Sep 17 00:54:40.401786 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.401741 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" podUID="f5d06746-0303-44bf-9bdd-26bf456b0779" containerName="main" containerID="cri-o://6a3caee63da0bb7236fe4478ab738a6b3fe9a90fa1af04ee68a6bf46847278df" gracePeriod=45 Sep 17 00:54:40.659857 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.659796 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:40.815272 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.815230 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-model-cache\") pod \"f5d06746-0303-44bf-9bdd-26bf456b0779\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " Sep 17 00:54:40.815272 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.815271 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5d06746-0303-44bf-9bdd-26bf456b0779-tls-certs\") pod \"f5d06746-0303-44bf-9bdd-26bf456b0779\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " Sep 17 00:54:40.815578 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.815322 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-home\") pod \"f5d06746-0303-44bf-9bdd-26bf456b0779\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " Sep 17 00:54:40.815578 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.815404 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz92k\" (UniqueName: \"kubernetes.io/projected/f5d06746-0303-44bf-9bdd-26bf456b0779-kube-api-access-gz92k\") pod \"f5d06746-0303-44bf-9bdd-26bf456b0779\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " Sep 17 00:54:40.815578 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.815421 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-tmp-dir\") pod \"f5d06746-0303-44bf-9bdd-26bf456b0779\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " Sep 17 00:54:40.815578 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.815447 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-dshm\") pod \"f5d06746-0303-44bf-9bdd-26bf456b0779\" (UID: \"f5d06746-0303-44bf-9bdd-26bf456b0779\") " Sep 17 00:54:40.815578 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.815551 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5d06746-0303-44bf-9bdd-26bf456b0779" (UID: "f5d06746-0303-44bf-9bdd-26bf456b0779"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:54:40.815814 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.815638 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5d06746-0303-44bf-9bdd-26bf456b0779" (UID: "f5d06746-0303-44bf-9bdd-26bf456b0779"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:54:40.815814 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.815656 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-home" (OuterVolumeSpecName: "home") pod "f5d06746-0303-44bf-9bdd-26bf456b0779" (UID: "f5d06746-0303-44bf-9bdd-26bf456b0779"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:54:40.815923 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.815848 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-model-cache\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:54:40.815923 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.815863 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-home\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:54:40.815923 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.815871 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-tmp-dir\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:54:40.817731 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.817709 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-dshm" (OuterVolumeSpecName: "dshm") pod "f5d06746-0303-44bf-9bdd-26bf456b0779" (UID: "f5d06746-0303-44bf-9bdd-26bf456b0779"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:54:40.817802 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.817771 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d06746-0303-44bf-9bdd-26bf456b0779-kube-api-access-gz92k" (OuterVolumeSpecName: "kube-api-access-gz92k") pod "f5d06746-0303-44bf-9bdd-26bf456b0779" (UID: "f5d06746-0303-44bf-9bdd-26bf456b0779"). InnerVolumeSpecName "kube-api-access-gz92k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:54:40.817802 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.817785 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5d06746-0303-44bf-9bdd-26bf456b0779-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5d06746-0303-44bf-9bdd-26bf456b0779" (UID: "f5d06746-0303-44bf-9bdd-26bf456b0779"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:54:40.917305 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.917225 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5d06746-0303-44bf-9bdd-26bf456b0779-tls-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:54:40.917305 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.917253 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz92k\" (UniqueName: \"kubernetes.io/projected/f5d06746-0303-44bf-9bdd-26bf456b0779-kube-api-access-gz92k\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:54:40.917305 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.917262 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5d06746-0303-44bf-9bdd-26bf456b0779-dshm\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:54:40.927375 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.927343 2582 generic.go:356] "Generic (PLEG): container finished" podID="f5d06746-0303-44bf-9bdd-26bf456b0779" containerID="6a3caee63da0bb7236fe4478ab738a6b3fe9a90fa1af04ee68a6bf46847278df" exitCode=0 Sep 17 00:54:40.927755 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.927389 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" Sep 17 00:54:40.927755 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.927437 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" event={"ID":"f5d06746-0303-44bf-9bdd-26bf456b0779","Type":"ContainerDied","Data":"6a3caee63da0bb7236fe4478ab738a6b3fe9a90fa1af04ee68a6bf46847278df"} Sep 17 00:54:40.927755 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.927482 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt" event={"ID":"f5d06746-0303-44bf-9bdd-26bf456b0779","Type":"ContainerDied","Data":"45e290ae42c9294f09a1ac725a9a311a08f811cb88294bc696795720ded693cd"} Sep 17 00:54:40.927755 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.927518 2582 scope.go:117] "RemoveContainer" containerID="6a3caee63da0bb7236fe4478ab738a6b3fe9a90fa1af04ee68a6bf46847278df" Sep 17 00:54:40.937064 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.937047 2582 scope.go:117] "RemoveContainer" containerID="6a3caee63da0bb7236fe4478ab738a6b3fe9a90fa1af04ee68a6bf46847278df" Sep 17 00:54:40.937338 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:54:40.937319 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a3caee63da0bb7236fe4478ab738a6b3fe9a90fa1af04ee68a6bf46847278df\": container with ID starting with 6a3caee63da0bb7236fe4478ab738a6b3fe9a90fa1af04ee68a6bf46847278df not found: ID does not exist" containerID="6a3caee63da0bb7236fe4478ab738a6b3fe9a90fa1af04ee68a6bf46847278df" Sep 17 00:54:40.937389 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.937348 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a3caee63da0bb7236fe4478ab738a6b3fe9a90fa1af04ee68a6bf46847278df"} err="failed to get container status \"6a3caee63da0bb7236fe4478ab738a6b3fe9a90fa1af04ee68a6bf46847278df\": rpc error: code = NotFound desc = could not find container \"6a3caee63da0bb7236fe4478ab738a6b3fe9a90fa1af04ee68a6bf46847278df\": container with ID starting with 6a3caee63da0bb7236fe4478ab738a6b3fe9a90fa1af04ee68a6bf46847278df not found: ID does not exist" Sep 17 00:54:40.947049 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.947024 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt"] Sep 17 00:54:40.950654 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:40.950635 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-5a9565/noten-model-5a9565-kserve-fb659677f-958jt"] Sep 17 00:54:41.609386 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:41.609324 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" podUID="d27af4e7-e46d-4489-bba4-60ef41cbcdd3" containerName="main" containerID="cri-o://7b454b145c5fe1254eca593e442f6fd5a7024dc3a088993fba232143c7d310d3" gracePeriod=45 Sep 17 00:54:41.651342 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:41.651317 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5d06746-0303-44bf-9bdd-26bf456b0779" path="/var/lib/kubelet/pods/f5d06746-0303-44bf-9bdd-26bf456b0779/volumes" Sep 17 00:54:41.867871 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:41.867813 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:54:41.934980 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:41.934949 2582 generic.go:356] "Generic (PLEG): container finished" podID="d27af4e7-e46d-4489-bba4-60ef41cbcdd3" containerID="7b454b145c5fe1254eca593e442f6fd5a7024dc3a088993fba232143c7d310d3" exitCode=0 Sep 17 00:54:41.935333 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:41.934992 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" Sep 17 00:54:41.935333 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:41.935034 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" event={"ID":"d27af4e7-e46d-4489-bba4-60ef41cbcdd3","Type":"ContainerDied","Data":"7b454b145c5fe1254eca593e442f6fd5a7024dc3a088993fba232143c7d310d3"} Sep 17 00:54:41.935333 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:41.935071 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5" event={"ID":"d27af4e7-e46d-4489-bba4-60ef41cbcdd3","Type":"ContainerDied","Data":"b861cd8681f5b6342d3796fb72927284bcf060c7381df3013aefeaa34faeff55"} Sep 17 00:54:41.935333 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:41.935087 2582 scope.go:117] "RemoveContainer" containerID="7b454b145c5fe1254eca593e442f6fd5a7024dc3a088993fba232143c7d310d3" Sep 17 00:54:41.944379 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:41.944361 2582 scope.go:117] "RemoveContainer" containerID="7b454b145c5fe1254eca593e442f6fd5a7024dc3a088993fba232143c7d310d3" Sep 17 00:54:41.944682 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:54:41.944664 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b454b145c5fe1254eca593e442f6fd5a7024dc3a088993fba232143c7d310d3\": container with ID starting with 7b454b145c5fe1254eca593e442f6fd5a7024dc3a088993fba232143c7d310d3 not found: ID does not exist" containerID="7b454b145c5fe1254eca593e442f6fd5a7024dc3a088993fba232143c7d310d3" Sep 17 00:54:41.944734 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:41.944688 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b454b145c5fe1254eca593e442f6fd5a7024dc3a088993fba232143c7d310d3"} err="failed to get container status \"7b454b145c5fe1254eca593e442f6fd5a7024dc3a088993fba232143c7d310d3\": rpc error: code = NotFound desc = could not find container \"7b454b145c5fe1254eca593e442f6fd5a7024dc3a088993fba232143c7d310d3\": container with ID starting with 7b454b145c5fe1254eca593e442f6fd5a7024dc3a088993fba232143c7d310d3 not found: ID does not exist" Sep 17 00:54:42.029155 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:42.029131 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-dshm\") pod \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " Sep 17 00:54:42.029282 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:42.029194 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-tls-certs\") pod \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " Sep 17 00:54:42.029282 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:42.029238 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-model-cache\") pod \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " Sep 17 00:54:42.029282 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:42.029275 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-home\") pod \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " Sep 17 00:54:42.029443 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:42.029410 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5w68\" (UniqueName: \"kubernetes.io/projected/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-kube-api-access-x5w68\") pod \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " Sep 17 00:54:42.029531 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:42.029485 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-model-cache" (OuterVolumeSpecName: "model-cache") pod "d27af4e7-e46d-4489-bba4-60ef41cbcdd3" (UID: "d27af4e7-e46d-4489-bba4-60ef41cbcdd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:54:42.029591 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:42.029528 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-home" (OuterVolumeSpecName: "home") pod "d27af4e7-e46d-4489-bba4-60ef41cbcdd3" (UID: "d27af4e7-e46d-4489-bba4-60ef41cbcdd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:54:42.029591 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:42.029490 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-tmp-dir\") pod \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\" (UID: \"d27af4e7-e46d-4489-bba4-60ef41cbcdd3\") " Sep 17 00:54:42.029747 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:42.029728 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d27af4e7-e46d-4489-bba4-60ef41cbcdd3" (UID: "d27af4e7-e46d-4489-bba4-60ef41cbcdd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:54:42.029933 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:42.029918 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-tmp-dir\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:54:42.029987 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:42.029938 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-model-cache\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:54:42.029987 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:42.029950 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-home\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:54:42.031762 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:42.031733 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-kube-api-access-x5w68" (OuterVolumeSpecName: "kube-api-access-x5w68") pod "d27af4e7-e46d-4489-bba4-60ef41cbcdd3" (UID: "d27af4e7-e46d-4489-bba4-60ef41cbcdd3"). InnerVolumeSpecName "kube-api-access-x5w68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:54:42.031937 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:42.031920 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-dshm" (OuterVolumeSpecName: "dshm") pod "d27af4e7-e46d-4489-bba4-60ef41cbcdd3" (UID: "d27af4e7-e46d-4489-bba4-60ef41cbcdd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:54:42.032014 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:42.031922 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d27af4e7-e46d-4489-bba4-60ef41cbcdd3" (UID: "d27af4e7-e46d-4489-bba4-60ef41cbcdd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:54:42.131485 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:42.131410 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-dshm\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:54:42.131485 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:42.131434 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-tls-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:54:42.131485 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:42.131446 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5w68\" (UniqueName: \"kubernetes.io/projected/d27af4e7-e46d-4489-bba4-60ef41cbcdd3-kube-api-access-x5w68\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:54:42.255553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:42.255523 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5"] Sep 17 00:54:42.260826 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:42.260799 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-e421ba/test-model-e421ba-kserve-854c5d7b4c-8xmx5"] Sep 17 00:54:43.651467 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:43.651435 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d27af4e7-e46d-4489-bba4-60ef41cbcdd3" path="/var/lib/kubelet/pods/d27af4e7-e46d-4489-bba4-60ef41cbcdd3/volumes" Sep 17 00:54:45.933236 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:45.933210 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" Sep 17 00:54:47.724145 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:47.724121 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/1.log" Sep 17 00:54:47.726941 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:47.726918 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/1.log" Sep 17 00:54:56.566704 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:56.566648 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz"] Sep 17 00:54:56.567169 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:56.566917 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" podUID="9b612ece-784b-4bff-942f-0c79d035e84c" containerName="maas-api" containerID="cri-o://3eb5f561939e2ce675e2d5ac558bd542d4ae5c173363630c1122e026ac9f319a" gracePeriod=30 Sep 17 00:54:57.593123 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:57.593087 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:54:57.818548 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:57.818488 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" Sep 17 00:54:57.886834 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:57.886753 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b612ece-784b-4bff-942f-0c79d035e84c-metrics-certs\") pod \"9b612ece-784b-4bff-942f-0c79d035e84c\" (UID: \"9b612ece-784b-4bff-942f-0c79d035e84c\") " Sep 17 00:54:57.886834 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:57.886805 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b612ece-784b-4bff-942f-0c79d035e84c-maas-api-tls\") pod \"9b612ece-784b-4bff-942f-0c79d035e84c\" (UID: \"9b612ece-784b-4bff-942f-0c79d035e84c\") " Sep 17 00:54:57.886834 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:57.886830 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9m6p\" (UniqueName: \"kubernetes.io/projected/9b612ece-784b-4bff-942f-0c79d035e84c-kube-api-access-v9m6p\") pod \"9b612ece-784b-4bff-942f-0c79d035e84c\" (UID: \"9b612ece-784b-4bff-942f-0c79d035e84c\") " Sep 17 00:54:57.887139 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:57.887017 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b612ece-784b-4bff-942f-0c79d035e84c-openshift-service-ca\") pod \"9b612ece-784b-4bff-942f-0c79d035e84c\" (UID: \"9b612ece-784b-4bff-942f-0c79d035e84c\") " Sep 17 00:54:57.887331 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:57.887306 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b612ece-784b-4bff-942f-0c79d035e84c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9b612ece-784b-4bff-942f-0c79d035e84c" (UID: "9b612ece-784b-4bff-942f-0c79d035e84c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 00:54:57.889032 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:57.889009 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b612ece-784b-4bff-942f-0c79d035e84c-kube-api-access-v9m6p" (OuterVolumeSpecName: "kube-api-access-v9m6p") pod "9b612ece-784b-4bff-942f-0c79d035e84c" (UID: "9b612ece-784b-4bff-942f-0c79d035e84c"). InnerVolumeSpecName "kube-api-access-v9m6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:54:57.889482 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:57.889461 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b612ece-784b-4bff-942f-0c79d035e84c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9b612ece-784b-4bff-942f-0c79d035e84c" (UID: "9b612ece-784b-4bff-942f-0c79d035e84c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:54:57.889482 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:57.889475 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b612ece-784b-4bff-942f-0c79d035e84c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9b612ece-784b-4bff-942f-0c79d035e84c" (UID: "9b612ece-784b-4bff-942f-0c79d035e84c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:54:57.988648 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:57.988617 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b612ece-784b-4bff-942f-0c79d035e84c-openshift-service-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:54:57.988648 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:57.988644 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b612ece-784b-4bff-942f-0c79d035e84c-metrics-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:54:57.988648 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:57.988654 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b612ece-784b-4bff-942f-0c79d035e84c-maas-api-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:54:57.988889 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:57.988663 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9m6p\" (UniqueName: \"kubernetes.io/projected/9b612ece-784b-4bff-942f-0c79d035e84c-kube-api-access-v9m6p\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:54:58.028585 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:58.028544 2582 generic.go:356] "Generic (PLEG): container finished" podID="9b612ece-784b-4bff-942f-0c79d035e84c" containerID="3eb5f561939e2ce675e2d5ac558bd542d4ae5c173363630c1122e026ac9f319a" exitCode=0 Sep 17 00:54:58.028768 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:58.028609 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" Sep 17 00:54:58.028768 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:58.028632 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" event={"ID":"9b612ece-784b-4bff-942f-0c79d035e84c","Type":"ContainerDied","Data":"3eb5f561939e2ce675e2d5ac558bd542d4ae5c173363630c1122e026ac9f319a"} Sep 17 00:54:58.028768 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:58.028720 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz" event={"ID":"9b612ece-784b-4bff-942f-0c79d035e84c","Type":"ContainerDied","Data":"6224151ca95e0aab7708c94dc26204512e88cfe6575a04657d290907d3ea6851"} Sep 17 00:54:58.028768 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:58.028749 2582 scope.go:117] "RemoveContainer" containerID="3eb5f561939e2ce675e2d5ac558bd542d4ae5c173363630c1122e026ac9f319a" Sep 17 00:54:58.039809 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:58.039793 2582 scope.go:117] "RemoveContainer" containerID="3eb5f561939e2ce675e2d5ac558bd542d4ae5c173363630c1122e026ac9f319a" Sep 17 00:54:58.041546 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:54:58.041522 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eb5f561939e2ce675e2d5ac558bd542d4ae5c173363630c1122e026ac9f319a\": container with ID starting with 3eb5f561939e2ce675e2d5ac558bd542d4ae5c173363630c1122e026ac9f319a not found: ID does not exist" containerID="3eb5f561939e2ce675e2d5ac558bd542d4ae5c173363630c1122e026ac9f319a" Sep 17 00:54:58.041628 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:58.041552 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eb5f561939e2ce675e2d5ac558bd542d4ae5c173363630c1122e026ac9f319a"} err="failed to get container status \"3eb5f561939e2ce675e2d5ac558bd542d4ae5c173363630c1122e026ac9f319a\": rpc error: code = NotFound desc = could not find container \"3eb5f561939e2ce675e2d5ac558bd542d4ae5c173363630c1122e026ac9f319a\": container with ID starting with 3eb5f561939e2ce675e2d5ac558bd542d4ae5c173363630c1122e026ac9f319a not found: ID does not exist" Sep 17 00:54:58.054114 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:58.054091 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz"] Sep 17 00:54:58.057988 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:58.057963 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-e421ba-cd9985567-9hbwz"] Sep 17 00:54:59.650992 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:54:59.650960 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b612ece-784b-4bff-942f-0c79d035e84c" path="/var/lib/kubelet/pods/9b612ece-784b-4bff-942f-0c79d035e84c/volumes" Sep 17 00:55:09.762955 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:09.762763 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt"] Sep 17 00:55:09.764809 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:09.764774 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b612ece-784b-4bff-942f-0c79d035e84c" containerName="maas-api" Sep 17 00:55:09.764809 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:09.764810 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b612ece-784b-4bff-942f-0c79d035e84c" containerName="maas-api" Sep 17 00:55:09.765016 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:09.764846 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27af4e7-e46d-4489-bba4-60ef41cbcdd3" containerName="main" Sep 17 00:55:09.765016 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:09.764855 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27af4e7-e46d-4489-bba4-60ef41cbcdd3" containerName="main" Sep 17 00:55:09.765016 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:09.764874 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5d06746-0303-44bf-9bdd-26bf456b0779" containerName="main" Sep 17 00:55:09.765016 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:09.764882 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5d06746-0303-44bf-9bdd-26bf456b0779" containerName="main" Sep 17 00:55:09.765232 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:09.765128 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5d06746-0303-44bf-9bdd-26bf456b0779" containerName="main" Sep 17 00:55:09.765232 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:09.765152 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b612ece-784b-4bff-942f-0c79d035e84c" containerName="maas-api" Sep 17 00:55:09.765232 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:09.765164 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d27af4e7-e46d-4489-bba4-60ef41cbcdd3" containerName="main" Sep 17 00:55:09.771023 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:09.770998 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" Sep 17 00:55:09.772707 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:09.772674 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt"] Sep 17 00:55:09.773544 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:09.773522 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-d2326d-serving-cert\"" Sep 17 00:55:09.906187 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:09.906152 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ba8810a-4bee-4105-899c-0356e2936a50-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt\" (UID: \"7ba8810a-4bee-4105-899c-0356e2936a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" Sep 17 00:55:09.906353 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:09.906260 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba8810a-4bee-4105-899c-0356e2936a50-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt\" (UID: \"7ba8810a-4bee-4105-899c-0356e2936a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" Sep 17 00:55:09.906353 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:09.906318 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba8810a-4bee-4105-899c-0356e2936a50-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt\" (UID: \"7ba8810a-4bee-4105-899c-0356e2936a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" Sep 17 00:55:09.906432 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:09.906355 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9744l\" (UniqueName: \"kubernetes.io/projected/7ba8810a-4bee-4105-899c-0356e2936a50-kube-api-access-9744l\") pod \"maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt\" (UID: \"7ba8810a-4bee-4105-899c-0356e2936a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" Sep 17 00:55:10.007734 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:10.007698 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ba8810a-4bee-4105-899c-0356e2936a50-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt\" (UID: \"7ba8810a-4bee-4105-899c-0356e2936a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" Sep 17 00:55:10.007900 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:10.007781 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba8810a-4bee-4105-899c-0356e2936a50-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt\" (UID: \"7ba8810a-4bee-4105-899c-0356e2936a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" Sep 17 00:55:10.007900 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:10.007819 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba8810a-4bee-4105-899c-0356e2936a50-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt\" (UID: \"7ba8810a-4bee-4105-899c-0356e2936a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" Sep 17 00:55:10.007900 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:10.007853 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9744l\" (UniqueName: \"kubernetes.io/projected/7ba8810a-4bee-4105-899c-0356e2936a50-kube-api-access-9744l\") pod \"maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt\" (UID: \"7ba8810a-4bee-4105-899c-0356e2936a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" Sep 17 00:55:10.008555 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:10.008531 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba8810a-4bee-4105-899c-0356e2936a50-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt\" (UID: \"7ba8810a-4bee-4105-899c-0356e2936a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" Sep 17 00:55:10.010427 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:10.010408 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba8810a-4bee-4105-899c-0356e2936a50-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt\" (UID: \"7ba8810a-4bee-4105-899c-0356e2936a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" Sep 17 00:55:10.010523 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:10.010439 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ba8810a-4bee-4105-899c-0356e2936a50-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt\" (UID: \"7ba8810a-4bee-4105-899c-0356e2936a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" Sep 17 00:55:10.015244 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:10.015189 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9744l\" (UniqueName: \"kubernetes.io/projected/7ba8810a-4bee-4105-899c-0356e2936a50-kube-api-access-9744l\") pod \"maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt\" (UID: \"7ba8810a-4bee-4105-899c-0356e2936a50\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" Sep 17 00:55:10.084133 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:10.084097 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" Sep 17 00:55:10.362938 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:10.362905 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt"] Sep 17 00:55:10.364544 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:55:10.364514 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba8810a_4bee_4105_899c_0356e2936a50.slice/crio-3d518fe1a16c604bc0eb10798fc2c3be256ee9b2d0193c6b28e1195b5bcae7a2 WatchSource:0}: Error finding container 3d518fe1a16c604bc0eb10798fc2c3be256ee9b2d0193c6b28e1195b5bcae7a2: Status 404 returned error can't find the container with id 3d518fe1a16c604bc0eb10798fc2c3be256ee9b2d0193c6b28e1195b5bcae7a2 Sep 17 00:55:11.101082 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:11.101030 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" event={"ID":"7ba8810a-4bee-4105-899c-0356e2936a50","Type":"ContainerStarted","Data":"85d61f023dfa6c5380f648e4ec5263caa3787fc31e5a30b35e75a3b9aef61410"} Sep 17 00:55:11.101563 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:11.101101 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" event={"ID":"7ba8810a-4bee-4105-899c-0356e2936a50","Type":"ContainerStarted","Data":"3d518fe1a16c604bc0eb10798fc2c3be256ee9b2d0193c6b28e1195b5bcae7a2"} Sep 17 00:55:11.101563 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:11.101216 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" Sep 17 00:55:11.116037 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:11.115985 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" podStartSLOduration=1.766537188 podStartE2EDuration="2.115969086s" totalImagesPullingTime="349.431898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:55:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:55:11.114110745 +0000 UTC m=+1524.044744232" watchObservedRunningTime="2026-09-17 00:55:11.115969086 +0000 UTC m=+1524.046602572" Sep 17 00:55:17.112595 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:17.112565 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" Sep 17 00:55:22.595513 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:22.595455 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:55:39.709648 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:39.709613 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj"] Sep 17 00:55:39.713683 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:39.713660 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" Sep 17 00:55:39.715548 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:39.715491 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-bcd112-serving-cert\"" Sep 17 00:55:39.720802 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:39.720772 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj"] Sep 17 00:55:39.795242 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:39.795203 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4nr\" (UniqueName: \"kubernetes.io/projected/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-kube-api-access-mp4nr\") pod \"maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj\" (UID: \"b2abdefa-266d-42a2-9a9a-ac9ddd845e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" Sep 17 00:55:39.795453 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:39.795265 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj\" (UID: \"b2abdefa-266d-42a2-9a9a-ac9ddd845e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" Sep 17 00:55:39.795453 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:39.795386 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj\" (UID: \"b2abdefa-266d-42a2-9a9a-ac9ddd845e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" Sep 17 00:55:39.795624 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:39.795526 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj\" (UID: \"b2abdefa-266d-42a2-9a9a-ac9ddd845e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" Sep 17 00:55:39.896644 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:39.896601 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj\" (UID: \"b2abdefa-266d-42a2-9a9a-ac9ddd845e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" Sep 17 00:55:39.896883 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:39.896693 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj\" (UID: \"b2abdefa-266d-42a2-9a9a-ac9ddd845e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" Sep 17 00:55:39.897204 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:39.897181 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4nr\" (UniqueName: \"kubernetes.io/projected/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-kube-api-access-mp4nr\") pod \"maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj\" (UID: \"b2abdefa-266d-42a2-9a9a-ac9ddd845e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" Sep 17 00:55:39.897948 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:39.897887 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj\" (UID: \"b2abdefa-266d-42a2-9a9a-ac9ddd845e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" Sep 17 00:55:39.898081 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:39.898009 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj\" (UID: \"b2abdefa-266d-42a2-9a9a-ac9ddd845e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" Sep 17 00:55:39.905727 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:39.905699 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4nr\" (UniqueName: \"kubernetes.io/projected/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-kube-api-access-mp4nr\") pod \"maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj\" (UID: \"b2abdefa-266d-42a2-9a9a-ac9ddd845e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" Sep 17 00:55:39.906128 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:39.906111 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj\" (UID: \"b2abdefa-266d-42a2-9a9a-ac9ddd845e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" Sep 17 00:55:39.906712 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:39.906691 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj\" (UID: \"b2abdefa-266d-42a2-9a9a-ac9ddd845e35\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" Sep 17 00:55:40.028326 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:40.028245 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" Sep 17 00:55:40.303420 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:40.303391 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj"] Sep 17 00:55:40.305631 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:55:40.305606 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2abdefa_266d_42a2_9a9a_ac9ddd845e35.slice/crio-822d4b241d204e29ffa14480c07b8c60dd3eeea4f329fd14d25c589adb5354be WatchSource:0}: Error finding container 822d4b241d204e29ffa14480c07b8c60dd3eeea4f329fd14d25c589adb5354be: Status 404 returned error can't find the container with id 822d4b241d204e29ffa14480c07b8c60dd3eeea4f329fd14d25c589adb5354be Sep 17 00:55:41.265307 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:41.265265 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" event={"ID":"b2abdefa-266d-42a2-9a9a-ac9ddd845e35","Type":"ContainerStarted","Data":"2f0ec55ea2edafabb991958140f80333cbf278960860b0490d4460067bd05117"} Sep 17 00:55:41.265758 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:41.265332 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" event={"ID":"b2abdefa-266d-42a2-9a9a-ac9ddd845e35","Type":"ContainerStarted","Data":"822d4b241d204e29ffa14480c07b8c60dd3eeea4f329fd14d25c589adb5354be"} Sep 17 00:55:41.265758 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:41.265462 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" Sep 17 00:55:41.280339 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:41.280278 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" podStartSLOduration=1.8276301240000001 podStartE2EDuration="2.280258209s" totalImagesPullingTime="452.628085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:55:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:55:41.279678309 +0000 UTC m=+1554.210311797" watchObservedRunningTime="2026-09-17 00:55:41.280258209 +0000 UTC m=+1554.210891696" Sep 17 00:55:47.276824 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:47.276792 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" Sep 17 00:55:56.098250 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:55:56.098207 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:56:15.387430 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.387391 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn"] Sep 17 00:56:15.393063 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.393041 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:15.395150 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.395130 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d2326d\"/\"openshift-service-ca.crt\"" Sep 17 00:56:15.395402 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.395383 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-d2326d\"/\"kube-root-ca.crt\"" Sep 17 00:56:15.395928 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.395907 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d2326d\"/\"default-dockercfg-qjg2n\"" Sep 17 00:56:15.395986 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.395916 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-d2326d\"/\"auth-test-model-d2326d-kserve-self-signed-certs\"" Sep 17 00:56:15.400868 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.400845 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn"] Sep 17 00:56:15.450449 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.450422 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-tmp-dir\") pod \"auth-test-model-d2326d-kserve-5f7455c759-nc4pn\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:15.450449 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.450450 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z5tm\" (UniqueName: \"kubernetes.io/projected/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-kube-api-access-9z5tm\") pod \"auth-test-model-d2326d-kserve-5f7455c759-nc4pn\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:15.450667 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.450492 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-home\") pod \"auth-test-model-d2326d-kserve-5f7455c759-nc4pn\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:15.450667 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.450586 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-dshm\") pod \"auth-test-model-d2326d-kserve-5f7455c759-nc4pn\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:15.450667 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.450610 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-model-cache\") pod \"auth-test-model-d2326d-kserve-5f7455c759-nc4pn\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:15.450667 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.450642 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-tls-certs\") pod \"auth-test-model-d2326d-kserve-5f7455c759-nc4pn\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:15.551644 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.551612 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-dshm\") pod \"auth-test-model-d2326d-kserve-5f7455c759-nc4pn\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:15.551810 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.551648 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-model-cache\") pod \"auth-test-model-d2326d-kserve-5f7455c759-nc4pn\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:15.551810 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.551678 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-tls-certs\") pod \"auth-test-model-d2326d-kserve-5f7455c759-nc4pn\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:15.551810 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.551759 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-tmp-dir\") pod \"auth-test-model-d2326d-kserve-5f7455c759-nc4pn\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:15.551810 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.551779 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z5tm\" (UniqueName: \"kubernetes.io/projected/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-kube-api-access-9z5tm\") pod \"auth-test-model-d2326d-kserve-5f7455c759-nc4pn\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:15.552015 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.551831 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-home\") pod \"auth-test-model-d2326d-kserve-5f7455c759-nc4pn\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:15.552157 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.552128 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-model-cache\") pod \"auth-test-model-d2326d-kserve-5f7455c759-nc4pn\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:15.552234 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.552192 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-home\") pod \"auth-test-model-d2326d-kserve-5f7455c759-nc4pn\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:15.552234 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.552208 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-tmp-dir\") pod \"auth-test-model-d2326d-kserve-5f7455c759-nc4pn\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:15.554012 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.553992 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-dshm\") pod \"auth-test-model-d2326d-kserve-5f7455c759-nc4pn\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:15.554408 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.554388 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-tls-certs\") pod \"auth-test-model-d2326d-kserve-5f7455c759-nc4pn\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:15.558963 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.558943 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z5tm\" (UniqueName: \"kubernetes.io/projected/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-kube-api-access-9z5tm\") pod \"auth-test-model-d2326d-kserve-5f7455c759-nc4pn\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:15.703553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.703456 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:15.973326 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:15.973303 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn"] Sep 17 00:56:15.975540 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:56:15.975508 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf837cc70_d4e0_4b89_bcda_cc1b40ff3179.slice/crio-6da8d3772a714202e55fdfd82993e7a66f09c96f5d26302f37b045e64ae60c11 WatchSource:0}: Error finding container 6da8d3772a714202e55fdfd82993e7a66f09c96f5d26302f37b045e64ae60c11: Status 404 returned error can't find the container with id 6da8d3772a714202e55fdfd82993e7a66f09c96f5d26302f37b045e64ae60c11 Sep 17 00:56:16.453527 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:16.453470 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" event={"ID":"f837cc70-d4e0-4b89-bcda-cc1b40ff3179","Type":"ContainerStarted","Data":"3c5b07a66f29ff989598ae2d1f8a657b566aeb90b56d85cb2dea50e6927cc436"} Sep 17 00:56:16.453947 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:16.453537 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" event={"ID":"f837cc70-d4e0-4b89-bcda-cc1b40ff3179","Type":"ContainerStarted","Data":"6da8d3772a714202e55fdfd82993e7a66f09c96f5d26302f37b045e64ae60c11"} Sep 17 00:56:16.469521 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:16.469446 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" podStartSLOduration=1.46942802 podStartE2EDuration="1.46942802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:56:16.468594371 +0000 UTC m=+1589.399227858" watchObservedRunningTime="2026-09-17 00:56:16.46942802 +0000 UTC m=+1589.400061508" Sep 17 00:56:21.389770 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:21.389737 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:56:25.704220 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:25.704179 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:25.704709 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:25.704236 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:25.716897 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:25.716873 2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:26.520916 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:26.520885 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 00:56:29.084027 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.083989 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68"] Sep 17 00:56:29.084436 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.084239 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" podUID="59297598-6e57-463c-85b4-d824c657de1d" containerName="maas-api" containerID="cri-o://7fa47b4d4e888af984d02b9ab3ac8338d6da396b0706dd2f51bb3e4a16aea37c" gracePeriod=30 Sep 17 00:56:29.339990 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.339920 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" Sep 17 00:56:29.496074 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.496042 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59297598-6e57-463c-85b4-d824c657de1d-openshift-service-ca\") pod \"59297598-6e57-463c-85b4-d824c657de1d\" (UID: \"59297598-6e57-463c-85b4-d824c657de1d\") " Sep 17 00:56:29.496264 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.496145 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clvms\" (UniqueName: \"kubernetes.io/projected/59297598-6e57-463c-85b4-d824c657de1d-kube-api-access-clvms\") pod \"59297598-6e57-463c-85b4-d824c657de1d\" (UID: \"59297598-6e57-463c-85b4-d824c657de1d\") " Sep 17 00:56:29.496341 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.496315 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59297598-6e57-463c-85b4-d824c657de1d-metrics-certs\") pod \"59297598-6e57-463c-85b4-d824c657de1d\" (UID: \"59297598-6e57-463c-85b4-d824c657de1d\") " Sep 17 00:56:29.496397 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.496366 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59297598-6e57-463c-85b4-d824c657de1d-maas-api-tls\") pod \"59297598-6e57-463c-85b4-d824c657de1d\" (UID: \"59297598-6e57-463c-85b4-d824c657de1d\") " Sep 17 00:56:29.496544 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.496521 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59297598-6e57-463c-85b4-d824c657de1d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "59297598-6e57-463c-85b4-d824c657de1d" (UID: "59297598-6e57-463c-85b4-d824c657de1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 00:56:29.496822 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.496792 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/59297598-6e57-463c-85b4-d824c657de1d-openshift-service-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:56:29.498628 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.498601 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59297598-6e57-463c-85b4-d824c657de1d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "59297598-6e57-463c-85b4-d824c657de1d" (UID: "59297598-6e57-463c-85b4-d824c657de1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:56:29.498628 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.498615 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59297598-6e57-463c-85b4-d824c657de1d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "59297598-6e57-463c-85b4-d824c657de1d" (UID: "59297598-6e57-463c-85b4-d824c657de1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:56:29.498878 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.498857 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59297598-6e57-463c-85b4-d824c657de1d-kube-api-access-clvms" (OuterVolumeSpecName: "kube-api-access-clvms") pod "59297598-6e57-463c-85b4-d824c657de1d" (UID: "59297598-6e57-463c-85b4-d824c657de1d"). InnerVolumeSpecName "kube-api-access-clvms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:56:29.526931 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.526893 2582 generic.go:356] "Generic (PLEG): container finished" podID="59297598-6e57-463c-85b4-d824c657de1d" containerID="7fa47b4d4e888af984d02b9ab3ac8338d6da396b0706dd2f51bb3e4a16aea37c" exitCode=0 Sep 17 00:56:29.526931 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.526944 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" Sep 17 00:56:29.527144 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.526985 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" event={"ID":"59297598-6e57-463c-85b4-d824c657de1d","Type":"ContainerDied","Data":"7fa47b4d4e888af984d02b9ab3ac8338d6da396b0706dd2f51bb3e4a16aea37c"} Sep 17 00:56:29.527144 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.527045 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68" event={"ID":"59297598-6e57-463c-85b4-d824c657de1d","Type":"ContainerDied","Data":"107097a0d1d29888ae049af95f98e3f163c4105a9fe91a22643ec16d151bec03"} Sep 17 00:56:29.527144 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.527071 2582 scope.go:117] "RemoveContainer" containerID="7fa47b4d4e888af984d02b9ab3ac8338d6da396b0706dd2f51bb3e4a16aea37c" Sep 17 00:56:29.537408 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.537382 2582 scope.go:117] "RemoveContainer" containerID="7fa47b4d4e888af984d02b9ab3ac8338d6da396b0706dd2f51bb3e4a16aea37c" Sep 17 00:56:29.537765 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:56:29.537745 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa47b4d4e888af984d02b9ab3ac8338d6da396b0706dd2f51bb3e4a16aea37c\": container with ID starting with 7fa47b4d4e888af984d02b9ab3ac8338d6da396b0706dd2f51bb3e4a16aea37c not found: ID does not exist" containerID="7fa47b4d4e888af984d02b9ab3ac8338d6da396b0706dd2f51bb3e4a16aea37c" Sep 17 00:56:29.537889 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.537774 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa47b4d4e888af984d02b9ab3ac8338d6da396b0706dd2f51bb3e4a16aea37c"} err="failed to get container status \"7fa47b4d4e888af984d02b9ab3ac8338d6da396b0706dd2f51bb3e4a16aea37c\": rpc error: code = NotFound desc = could not find container \"7fa47b4d4e888af984d02b9ab3ac8338d6da396b0706dd2f51bb3e4a16aea37c\": container with ID starting with 7fa47b4d4e888af984d02b9ab3ac8338d6da396b0706dd2f51bb3e4a16aea37c not found: ID does not exist" Sep 17 00:56:29.553281 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.553250 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68"] Sep 17 00:56:29.558817 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.558786 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c7c845e1-566f6dd88c-fhb68"] Sep 17 00:56:29.598378 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.598294 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clvms\" (UniqueName: \"kubernetes.io/projected/59297598-6e57-463c-85b4-d824c657de1d-kube-api-access-clvms\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:56:29.598378 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.598327 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/59297598-6e57-463c-85b4-d824c657de1d-metrics-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:56:29.598378 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.598337 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/59297598-6e57-463c-85b4-d824c657de1d-maas-api-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:56:29.651004 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:29.650961 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59297598-6e57-463c-85b4-d824c657de1d" path="/var/lib/kubelet/pods/59297598-6e57-463c-85b4-d824c657de1d/volumes" Sep 17 00:56:50.290562 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:56:50.290527 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:57:21.796220 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:57:21.796187 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:57:52.996534 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:57:52.996482 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:58:06.171888 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:06.171854 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n"] Sep 17 00:58:17.598550 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:17.598515 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:58:21.192320 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:21.192282 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" podUID="b2c18e33-b3f4-4197-9dab-9465b8327206" containerName="main" containerID="cri-o://69ef9b6d0ab87a822969a25b9cc05c23ea6e802db5bb7c8ae63cf1d30b41f97e" gracePeriod=45 Sep 17 00:58:21.464162 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:21.464135 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:58:21.554428 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:21.554396 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-dshm\") pod \"b2c18e33-b3f4-4197-9dab-9465b8327206\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " Sep 17 00:58:21.554632 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:21.554449 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c18e33-b3f4-4197-9dab-9465b8327206-tls-certs\") pod \"b2c18e33-b3f4-4197-9dab-9465b8327206\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " Sep 17 00:58:21.554632 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:21.554529 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8s9mg\" (UniqueName: \"kubernetes.io/projected/b2c18e33-b3f4-4197-9dab-9465b8327206-kube-api-access-8s9mg\") pod \"b2c18e33-b3f4-4197-9dab-9465b8327206\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " Sep 17 00:58:21.554632 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:21.554577 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-model-cache\") pod \"b2c18e33-b3f4-4197-9dab-9465b8327206\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " Sep 17 00:58:21.554632 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:21.554621 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-tmp-dir\") pod \"b2c18e33-b3f4-4197-9dab-9465b8327206\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " Sep 17 00:58:21.554862 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:21.554660 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-home\") pod \"b2c18e33-b3f4-4197-9dab-9465b8327206\" (UID: \"b2c18e33-b3f4-4197-9dab-9465b8327206\") " Sep 17 00:58:21.555054 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:21.554937 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2c18e33-b3f4-4197-9dab-9465b8327206" (UID: "b2c18e33-b3f4-4197-9dab-9465b8327206"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:58:21.555054 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:21.555000 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2c18e33-b3f4-4197-9dab-9465b8327206" (UID: "b2c18e33-b3f4-4197-9dab-9465b8327206"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:58:21.555054 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:21.555018 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-home" (OuterVolumeSpecName: "home") pod "b2c18e33-b3f4-4197-9dab-9465b8327206" (UID: "b2c18e33-b3f4-4197-9dab-9465b8327206"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:58:21.555431 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:21.555088 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-tmp-dir\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:58:21.556954 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:21.556848 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-dshm" (OuterVolumeSpecName: "dshm") pod "b2c18e33-b3f4-4197-9dab-9465b8327206" (UID: "b2c18e33-b3f4-4197-9dab-9465b8327206"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 00:58:21.556954 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:21.556908 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c18e33-b3f4-4197-9dab-9465b8327206-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2c18e33-b3f4-4197-9dab-9465b8327206" (UID: "b2c18e33-b3f4-4197-9dab-9465b8327206"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:58:21.557152 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:21.557002 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c18e33-b3f4-4197-9dab-9465b8327206-kube-api-access-8s9mg" (OuterVolumeSpecName: "kube-api-access-8s9mg") pod "b2c18e33-b3f4-4197-9dab-9465b8327206" (UID: "b2c18e33-b3f4-4197-9dab-9465b8327206"). InnerVolumeSpecName "kube-api-access-8s9mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:58:21.655815 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:21.655789 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c18e33-b3f4-4197-9dab-9465b8327206-tls-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:58:21.655815 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:21.655813 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s9mg\" (UniqueName: \"kubernetes.io/projected/b2c18e33-b3f4-4197-9dab-9465b8327206-kube-api-access-8s9mg\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:58:21.655993 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:21.655823 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-model-cache\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:58:21.655993 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:21.655833 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-home\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:58:21.655993 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:21.655840 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c18e33-b3f4-4197-9dab-9465b8327206-dshm\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:58:22.119469 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:22.119439 2582 generic.go:356] "Generic (PLEG): container finished" podID="b2c18e33-b3f4-4197-9dab-9465b8327206" containerID="69ef9b6d0ab87a822969a25b9cc05c23ea6e802db5bb7c8ae63cf1d30b41f97e" exitCode=0 Sep 17 00:58:22.119672 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:22.119487 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" Sep 17 00:58:22.119672 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:22.119529 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" event={"ID":"b2c18e33-b3f4-4197-9dab-9465b8327206","Type":"ContainerDied","Data":"69ef9b6d0ab87a822969a25b9cc05c23ea6e802db5bb7c8ae63cf1d30b41f97e"} Sep 17 00:58:22.119672 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:22.119571 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n" event={"ID":"b2c18e33-b3f4-4197-9dab-9465b8327206","Type":"ContainerDied","Data":"7fad9f99a3d441acd75750c7cfbe3413cb973431b1843a2219f57ebfd6e6e9f9"} Sep 17 00:58:22.119672 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:22.119590 2582 scope.go:117] "RemoveContainer" containerID="69ef9b6d0ab87a822969a25b9cc05c23ea6e802db5bb7c8ae63cf1d30b41f97e" Sep 17 00:58:22.128792 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:22.128774 2582 scope.go:117] "RemoveContainer" containerID="69ef9b6d0ab87a822969a25b9cc05c23ea6e802db5bb7c8ae63cf1d30b41f97e" Sep 17 00:58:22.129076 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:58:22.129058 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69ef9b6d0ab87a822969a25b9cc05c23ea6e802db5bb7c8ae63cf1d30b41f97e\": container with ID starting with 69ef9b6d0ab87a822969a25b9cc05c23ea6e802db5bb7c8ae63cf1d30b41f97e not found: ID does not exist" containerID="69ef9b6d0ab87a822969a25b9cc05c23ea6e802db5bb7c8ae63cf1d30b41f97e" Sep 17 00:58:22.129125 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:22.129084 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69ef9b6d0ab87a822969a25b9cc05c23ea6e802db5bb7c8ae63cf1d30b41f97e"} err="failed to get container status \"69ef9b6d0ab87a822969a25b9cc05c23ea6e802db5bb7c8ae63cf1d30b41f97e\": rpc error: code = NotFound desc = could not find container \"69ef9b6d0ab87a822969a25b9cc05c23ea6e802db5bb7c8ae63cf1d30b41f97e\": container with ID starting with 69ef9b6d0ab87a822969a25b9cc05c23ea6e802db5bb7c8ae63cf1d30b41f97e not found: ID does not exist" Sep 17 00:58:22.135689 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:22.135657 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n"] Sep 17 00:58:22.139346 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:22.139326 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a79f56/simulator-w0-a79f56-kserve-5cbc975465-svz5n"] Sep 17 00:58:23.650354 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:23.650318 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2c18e33-b3f4-4197-9dab-9465b8327206" path="/var/lib/kubelet/pods/b2c18e33-b3f4-4197-9dab-9465b8327206/volumes" Sep 17 00:58:25.927982 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:25.927944 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4"] Sep 17 00:58:25.928658 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:25.928634 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2c18e33-b3f4-4197-9dab-9465b8327206" containerName="main" Sep 17 00:58:25.928658 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:25.928660 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2c18e33-b3f4-4197-9dab-9465b8327206" containerName="main" Sep 17 00:58:25.928814 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:25.928674 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59297598-6e57-463c-85b4-d824c657de1d" containerName="maas-api" Sep 17 00:58:25.928814 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:25.928683 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="59297598-6e57-463c-85b4-d824c657de1d" containerName="maas-api" Sep 17 00:58:25.928884 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:25.928834 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="59297598-6e57-463c-85b4-d824c657de1d" containerName="maas-api" Sep 17 00:58:25.928884 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:25.928853 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2c18e33-b3f4-4197-9dab-9465b8327206" containerName="main" Sep 17 00:58:25.933950 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:25.933930 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" Sep 17 00:58:25.935932 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:25.935911 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-80e99a-serving-cert\"" Sep 17 00:58:25.943851 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:25.943441 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4"] Sep 17 00:58:26.096745 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:26.096713 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d29c4db-f870-462b-8ab7-d30dff78408d-maas-api-tls\") pod \"maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4\" (UID: \"1d29c4db-f870-462b-8ab7-d30dff78408d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" Sep 17 00:58:26.096745 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:26.096749 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d29c4db-f870-462b-8ab7-d30dff78408d-metrics-certs\") pod \"maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4\" (UID: \"1d29c4db-f870-462b-8ab7-d30dff78408d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" Sep 17 00:58:26.096958 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:26.096824 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d29c4db-f870-462b-8ab7-d30dff78408d-openshift-service-ca\") pod \"maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4\" (UID: \"1d29c4db-f870-462b-8ab7-d30dff78408d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" Sep 17 00:58:26.096958 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:26.096842 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tl9b\" (UniqueName: \"kubernetes.io/projected/1d29c4db-f870-462b-8ab7-d30dff78408d-kube-api-access-7tl9b\") pod \"maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4\" (UID: \"1d29c4db-f870-462b-8ab7-d30dff78408d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" Sep 17 00:58:26.198091 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:26.198015 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d29c4db-f870-462b-8ab7-d30dff78408d-maas-api-tls\") pod \"maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4\" (UID: \"1d29c4db-f870-462b-8ab7-d30dff78408d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" Sep 17 00:58:26.198091 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:26.198052 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d29c4db-f870-462b-8ab7-d30dff78408d-metrics-certs\") pod \"maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4\" (UID: \"1d29c4db-f870-462b-8ab7-d30dff78408d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" Sep 17 00:58:26.198259 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:26.198114 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d29c4db-f870-462b-8ab7-d30dff78408d-openshift-service-ca\") pod \"maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4\" (UID: \"1d29c4db-f870-462b-8ab7-d30dff78408d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" Sep 17 00:58:26.198259 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:26.198133 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tl9b\" (UniqueName: \"kubernetes.io/projected/1d29c4db-f870-462b-8ab7-d30dff78408d-kube-api-access-7tl9b\") pod \"maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4\" (UID: \"1d29c4db-f870-462b-8ab7-d30dff78408d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" Sep 17 00:58:26.198772 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:26.198750 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d29c4db-f870-462b-8ab7-d30dff78408d-openshift-service-ca\") pod \"maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4\" (UID: \"1d29c4db-f870-462b-8ab7-d30dff78408d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" Sep 17 00:58:26.200581 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:26.200560 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d29c4db-f870-462b-8ab7-d30dff78408d-maas-api-tls\") pod \"maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4\" (UID: \"1d29c4db-f870-462b-8ab7-d30dff78408d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" Sep 17 00:58:26.200660 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:26.200597 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d29c4db-f870-462b-8ab7-d30dff78408d-metrics-certs\") pod \"maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4\" (UID: \"1d29c4db-f870-462b-8ab7-d30dff78408d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" Sep 17 00:58:26.205789 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:26.205766 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tl9b\" (UniqueName: \"kubernetes.io/projected/1d29c4db-f870-462b-8ab7-d30dff78408d-kube-api-access-7tl9b\") pod \"maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4\" (UID: \"1d29c4db-f870-462b-8ab7-d30dff78408d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" Sep 17 00:58:26.248319 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:26.248294 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" Sep 17 00:58:26.527258 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:26.527064 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4"] Sep 17 00:58:26.530643 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:58:26.530615 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d29c4db_f870_462b_8ab7_d30dff78408d.slice/crio-9dd2a608b9f7fa0c9e6271456a0cd55e36f15db9e2a326660fa4577466954c6a WatchSource:0}: Error finding container 9dd2a608b9f7fa0c9e6271456a0cd55e36f15db9e2a326660fa4577466954c6a: Status 404 returned error can't find the container with id 9dd2a608b9f7fa0c9e6271456a0cd55e36f15db9e2a326660fa4577466954c6a Sep 17 00:58:26.532574 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:26.532551 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 00:58:27.151164 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:27.151129 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" event={"ID":"1d29c4db-f870-462b-8ab7-d30dff78408d","Type":"ContainerStarted","Data":"9dd2a608b9f7fa0c9e6271456a0cd55e36f15db9e2a326660fa4577466954c6a"} Sep 17 00:58:28.157969 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:28.157930 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" event={"ID":"1d29c4db-f870-462b-8ab7-d30dff78408d","Type":"ContainerStarted","Data":"c057ef4ed0a49f018e6f6b24631d89320ef6eb174994e3a1e203283112de7307"} Sep 17 00:58:28.158361 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:28.158004 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" Sep 17 00:58:28.174088 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:28.174038 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" podStartSLOduration=2.515124598 podStartE2EDuration="3.174022695s" totalImagesPullingTime="658.898097ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:58:28.171917315 +0000 UTC m=+1721.102550803" watchObservedRunningTime="2026-09-17 00:58:28.174022695 +0000 UTC m=+1721.104656182" Sep 17 00:58:34.168788 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:34.168754 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" Sep 17 00:58:50.788087 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:58:50.788052 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:59:00.780188 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:00.780145 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc"] Sep 17 00:59:00.784151 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:00.784134 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" Sep 17 00:59:00.785868 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:00.785848 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1c4c89-serving-cert\"" Sep 17 00:59:00.793200 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:00.793177 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc"] Sep 17 00:59:00.935828 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:00.935789 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/050fe979-8c9b-4211-8f81-dc9ae65a63ff-openshift-service-ca\") pod \"maas-api-e2e-api-b-1c4c89-6769899b69-hldlc\" (UID: \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" Sep 17 00:59:00.936014 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:00.935851 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/050fe979-8c9b-4211-8f81-dc9ae65a63ff-metrics-certs\") pod \"maas-api-e2e-api-b-1c4c89-6769899b69-hldlc\" (UID: \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" Sep 17 00:59:00.936014 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:00.935955 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js6kb\" (UniqueName: \"kubernetes.io/projected/050fe979-8c9b-4211-8f81-dc9ae65a63ff-kube-api-access-js6kb\") pod \"maas-api-e2e-api-b-1c4c89-6769899b69-hldlc\" (UID: \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" Sep 17 00:59:00.936148 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:00.936031 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/050fe979-8c9b-4211-8f81-dc9ae65a63ff-maas-api-tls\") pod \"maas-api-e2e-api-b-1c4c89-6769899b69-hldlc\" (UID: \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" Sep 17 00:59:01.037490 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:01.037390 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/050fe979-8c9b-4211-8f81-dc9ae65a63ff-maas-api-tls\") pod \"maas-api-e2e-api-b-1c4c89-6769899b69-hldlc\" (UID: \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" Sep 17 00:59:01.037700 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:01.037530 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/050fe979-8c9b-4211-8f81-dc9ae65a63ff-openshift-service-ca\") pod \"maas-api-e2e-api-b-1c4c89-6769899b69-hldlc\" (UID: \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" Sep 17 00:59:01.037700 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:01.037576 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/050fe979-8c9b-4211-8f81-dc9ae65a63ff-metrics-certs\") pod \"maas-api-e2e-api-b-1c4c89-6769899b69-hldlc\" (UID: \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" Sep 17 00:59:01.037700 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:01.037621 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js6kb\" (UniqueName: \"kubernetes.io/projected/050fe979-8c9b-4211-8f81-dc9ae65a63ff-kube-api-access-js6kb\") pod \"maas-api-e2e-api-b-1c4c89-6769899b69-hldlc\" (UID: \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" Sep 17 00:59:01.037700 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:59:01.037574 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-serving-cert: secret "maas-api-e2e-api-b-1c4c89-serving-cert" not found Sep 17 00:59:01.037909 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:59:01.037737 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/050fe979-8c9b-4211-8f81-dc9ae65a63ff-maas-api-tls podName:050fe979-8c9b-4211-8f81-dc9ae65a63ff nodeName:}" failed. No retries permitted until 2026-09-17 00:59:01.537716721 +0000 UTC m=+1754.468350198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/050fe979-8c9b-4211-8f81-dc9ae65a63ff-maas-api-tls") pod "maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" (UID: "050fe979-8c9b-4211-8f81-dc9ae65a63ff") : secret "maas-api-e2e-api-b-1c4c89-serving-cert" not found Sep 17 00:59:01.038233 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:01.038214 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/050fe979-8c9b-4211-8f81-dc9ae65a63ff-openshift-service-ca\") pod \"maas-api-e2e-api-b-1c4c89-6769899b69-hldlc\" (UID: \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" Sep 17 00:59:01.040163 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:01.040146 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/050fe979-8c9b-4211-8f81-dc9ae65a63ff-metrics-certs\") pod \"maas-api-e2e-api-b-1c4c89-6769899b69-hldlc\" (UID: \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" Sep 17 00:59:01.045399 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:01.045378 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-js6kb\" (UniqueName: \"kubernetes.io/projected/050fe979-8c9b-4211-8f81-dc9ae65a63ff-kube-api-access-js6kb\") pod \"maas-api-e2e-api-b-1c4c89-6769899b69-hldlc\" (UID: \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" Sep 17 00:59:01.543620 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:01.543569 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/050fe979-8c9b-4211-8f81-dc9ae65a63ff-maas-api-tls\") pod \"maas-api-e2e-api-b-1c4c89-6769899b69-hldlc\" (UID: \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" Sep 17 00:59:01.543826 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:59:01.543734 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-serving-cert: secret "maas-api-e2e-api-b-1c4c89-serving-cert" not found Sep 17 00:59:01.543826 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:59:01.543803 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/050fe979-8c9b-4211-8f81-dc9ae65a63ff-maas-api-tls podName:050fe979-8c9b-4211-8f81-dc9ae65a63ff nodeName:}" failed. No retries permitted until 2026-09-17 00:59:02.543787766 +0000 UTC m=+1755.474421231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/050fe979-8c9b-4211-8f81-dc9ae65a63ff-maas-api-tls") pod "maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" (UID: "050fe979-8c9b-4211-8f81-dc9ae65a63ff") : secret "maas-api-e2e-api-b-1c4c89-serving-cert" not found Sep 17 00:59:02.555800 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:02.555764 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/050fe979-8c9b-4211-8f81-dc9ae65a63ff-maas-api-tls\") pod \"maas-api-e2e-api-b-1c4c89-6769899b69-hldlc\" (UID: \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" Sep 17 00:59:02.558414 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:02.558388 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/050fe979-8c9b-4211-8f81-dc9ae65a63ff-maas-api-tls\") pod \"maas-api-e2e-api-b-1c4c89-6769899b69-hldlc\" (UID: \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" Sep 17 00:59:02.596299 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:02.596263 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" Sep 17 00:59:02.868163 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:02.868140 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc"] Sep 17 00:59:02.870364 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:59:02.870331 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050fe979_8c9b_4211_8f81_dc9ae65a63ff.slice/crio-a66e10552cd5c964ce4628555a82e5f001ee94fa93a72ffc7ca554aaf375f85d WatchSource:0}: Error finding container a66e10552cd5c964ce4628555a82e5f001ee94fa93a72ffc7ca554aaf375f85d: Status 404 returned error can't find the container with id a66e10552cd5c964ce4628555a82e5f001ee94fa93a72ffc7ca554aaf375f85d Sep 17 00:59:03.347553 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:03.347517 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" event={"ID":"050fe979-8c9b-4211-8f81-dc9ae65a63ff","Type":"ContainerStarted","Data":"76e11c3f7141b9a8f498f00c47b1692f9c83ede9f8ee4d102438a50ec7c2c5da"} Sep 17 00:59:03.347680 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:03.347630 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" event={"ID":"050fe979-8c9b-4211-8f81-dc9ae65a63ff","Type":"ContainerStarted","Data":"a66e10552cd5c964ce4628555a82e5f001ee94fa93a72ffc7ca554aaf375f85d"} Sep 17 00:59:03.347680 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:03.347661 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" Sep 17 00:59:03.362093 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:03.362047 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" podStartSLOduration=2.972044799 podStartE2EDuration="3.362032559s" totalImagesPullingTime="389.98776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:59:03.361087394 +0000 UTC m=+1756.291720884" watchObservedRunningTime="2026-09-17 00:59:03.362032559 +0000 UTC m=+1756.292666045" Sep 17 00:59:09.359604 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:09.359571 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" Sep 17 00:59:10.997322 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:10.997286 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:59:32.696283 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:32.696249 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 00:59:41.910915 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:41.910881 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7696-d9c7f3"] Sep 17 00:59:41.914932 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:41.914900 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-d9c7f3" Sep 17 00:59:41.917104 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:41.917076 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bd8qq\"" Sep 17 00:59:41.918069 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:41.918045 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-d9c7f3"] Sep 17 00:59:41.949112 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:41.949076 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj5np\" (UniqueName: \"kubernetes.io/projected/39b06042-b6ca-47ac-b49c-f26dfffa2a43-kube-api-access-dj5np\") pod \"test-curl-7696-d9c7f3\" (UID: \"39b06042-b6ca-47ac-b49c-f26dfffa2a43\") " pod="opendatahub/test-curl-7696-d9c7f3" Sep 17 00:59:42.050537 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:42.050447 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj5np\" (UniqueName: \"kubernetes.io/projected/39b06042-b6ca-47ac-b49c-f26dfffa2a43-kube-api-access-dj5np\") pod \"test-curl-7696-d9c7f3\" (UID: \"39b06042-b6ca-47ac-b49c-f26dfffa2a43\") " pod="opendatahub/test-curl-7696-d9c7f3" Sep 17 00:59:42.058374 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:42.058343 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj5np\" (UniqueName: \"kubernetes.io/projected/39b06042-b6ca-47ac-b49c-f26dfffa2a43-kube-api-access-dj5np\") pod \"test-curl-7696-d9c7f3\" (UID: \"39b06042-b6ca-47ac-b49c-f26dfffa2a43\") " pod="opendatahub/test-curl-7696-d9c7f3" Sep 17 00:59:42.226254 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:42.226159 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-d9c7f3" Sep 17 00:59:42.499035 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:42.499007 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-d9c7f3"] Sep 17 00:59:42.501188 ip-10-0-134-177 kubenswrapper[2582]: W0917 00:59:42.501160 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39b06042_b6ca_47ac_b49c_f26dfffa2a43.slice/crio-e7f982c7539c955411b5dd75840b3a06ad8dbd2edddb7c63df03194e4a8b40d9 WatchSource:0}: Error finding container e7f982c7539c955411b5dd75840b3a06ad8dbd2edddb7c63df03194e4a8b40d9: Status 404 returned error can't find the container with id e7f982c7539c955411b5dd75840b3a06ad8dbd2edddb7c63df03194e4a8b40d9 Sep 17 00:59:42.562745 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:42.562715 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-d9c7f3" event={"ID":"39b06042-b6ca-47ac-b49c-f26dfffa2a43","Type":"ContainerStarted","Data":"e7f982c7539c955411b5dd75840b3a06ad8dbd2edddb7c63df03194e4a8b40d9"} Sep 17 00:59:43.568989 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:43.568955 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-d9c7f3" event={"ID":"39b06042-b6ca-47ac-b49c-f26dfffa2a43","Type":"ContainerStarted","Data":"85f1260679d29c0e36db2c2934d91ffe52b338bc7bb09a98327be78d8d33c10c"} Sep 17 00:59:43.581330 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:43.581282 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7696-d9c7f3" podStartSLOduration=2.581261403 podStartE2EDuration="2.581261403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 00:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 00:59:43.58048567 +0000 UTC m=+1796.511119151" watchObservedRunningTime="2026-09-17 00:59:43.581261403 +0000 UTC m=+1796.511894891" Sep 17 00:59:44.159629 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:44.159594 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7696-d9c7f3"] Sep 17 00:59:44.162621 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:44.162592 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7696-d9c7f3"] Sep 17 00:59:44.575556 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:59:44.575527 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7696-d9c7f3" container="test-curl-7696-d9c7f3" Sep 17 00:59:45.447522 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:45.447471 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn"] Sep 17 00:59:45.581344 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:45.581307 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7696-d9c7f3" podUID="39b06042-b6ca-47ac-b49c-f26dfffa2a43" containerName="test-curl-7696-d9c7f3" containerID="cri-o://85f1260679d29c0e36db2c2934d91ffe52b338bc7bb09a98327be78d8d33c10c" gracePeriod=2 Sep 17 00:59:45.582924 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:59:45.582895 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7696-d9c7f3\" is forbidden: User \"system:node:ip-10-0-134-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-177.ec2.internal' and this object" podUID="39b06042-b6ca-47ac-b49c-f26dfffa2a43" pod="opendatahub/test-curl-7696-d9c7f3" Sep 17 00:59:45.727381 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:45.727284 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4"] Sep 17 00:59:45.727626 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:45.727597 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" podUID="1d29c4db-f870-462b-8ab7-d30dff78408d" containerName="maas-api" containerID="cri-o://c057ef4ed0a49f018e6f6b24631d89320ef6eb174994e3a1e203283112de7307" gracePeriod=30 Sep 17 00:59:45.979331 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:45.979260 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" Sep 17 00:59:46.090598 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.090564 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d29c4db-f870-462b-8ab7-d30dff78408d-maas-api-tls\") pod \"1d29c4db-f870-462b-8ab7-d30dff78408d\" (UID: \"1d29c4db-f870-462b-8ab7-d30dff78408d\") " Sep 17 00:59:46.090767 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.090645 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d29c4db-f870-462b-8ab7-d30dff78408d-openshift-service-ca\") pod \"1d29c4db-f870-462b-8ab7-d30dff78408d\" (UID: \"1d29c4db-f870-462b-8ab7-d30dff78408d\") " Sep 17 00:59:46.090767 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.090690 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7tl9b\" (UniqueName: \"kubernetes.io/projected/1d29c4db-f870-462b-8ab7-d30dff78408d-kube-api-access-7tl9b\") pod \"1d29c4db-f870-462b-8ab7-d30dff78408d\" (UID: \"1d29c4db-f870-462b-8ab7-d30dff78408d\") " Sep 17 00:59:46.090767 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.090711 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d29c4db-f870-462b-8ab7-d30dff78408d-metrics-certs\") pod \"1d29c4db-f870-462b-8ab7-d30dff78408d\" (UID: \"1d29c4db-f870-462b-8ab7-d30dff78408d\") " Sep 17 00:59:46.091045 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.091021 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d29c4db-f870-462b-8ab7-d30dff78408d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1d29c4db-f870-462b-8ab7-d30dff78408d" (UID: "1d29c4db-f870-462b-8ab7-d30dff78408d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 00:59:46.093390 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.093359 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d29c4db-f870-462b-8ab7-d30dff78408d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1d29c4db-f870-462b-8ab7-d30dff78408d" (UID: "1d29c4db-f870-462b-8ab7-d30dff78408d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:59:46.093554 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.093416 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d29c4db-f870-462b-8ab7-d30dff78408d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1d29c4db-f870-462b-8ab7-d30dff78408d" (UID: "1d29c4db-f870-462b-8ab7-d30dff78408d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 00:59:46.093554 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.093428 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d29c4db-f870-462b-8ab7-d30dff78408d-kube-api-access-7tl9b" (OuterVolumeSpecName: "kube-api-access-7tl9b") pod "1d29c4db-f870-462b-8ab7-d30dff78408d" (UID: "1d29c4db-f870-462b-8ab7-d30dff78408d"). InnerVolumeSpecName "kube-api-access-7tl9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:59:46.191581 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.191550 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d29c4db-f870-462b-8ab7-d30dff78408d-openshift-service-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:59:46.191581 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.191576 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tl9b\" (UniqueName: \"kubernetes.io/projected/1d29c4db-f870-462b-8ab7-d30dff78408d-kube-api-access-7tl9b\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:59:46.191581 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.191585 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d29c4db-f870-462b-8ab7-d30dff78408d-metrics-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:59:46.191799 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.191594 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d29c4db-f870-462b-8ab7-d30dff78408d-maas-api-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:59:46.588293 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.588258 2582 generic.go:356] "Generic (PLEG): container finished" podID="1d29c4db-f870-462b-8ab7-d30dff78408d" containerID="c057ef4ed0a49f018e6f6b24631d89320ef6eb174994e3a1e203283112de7307" exitCode=0 Sep 17 00:59:46.588712 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.588298 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" Sep 17 00:59:46.588712 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.588348 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" event={"ID":"1d29c4db-f870-462b-8ab7-d30dff78408d","Type":"ContainerDied","Data":"c057ef4ed0a49f018e6f6b24631d89320ef6eb174994e3a1e203283112de7307"} Sep 17 00:59:46.588712 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.588387 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4" event={"ID":"1d29c4db-f870-462b-8ab7-d30dff78408d","Type":"ContainerDied","Data":"9dd2a608b9f7fa0c9e6271456a0cd55e36f15db9e2a326660fa4577466954c6a"} Sep 17 00:59:46.588712 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.588408 2582 scope.go:117] "RemoveContainer" containerID="c057ef4ed0a49f018e6f6b24631d89320ef6eb174994e3a1e203283112de7307" Sep 17 00:59:46.600725 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.600697 2582 scope.go:117] "RemoveContainer" containerID="c057ef4ed0a49f018e6f6b24631d89320ef6eb174994e3a1e203283112de7307" Sep 17 00:59:46.601589 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:59:46.601566 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c057ef4ed0a49f018e6f6b24631d89320ef6eb174994e3a1e203283112de7307\": container with ID starting with c057ef4ed0a49f018e6f6b24631d89320ef6eb174994e3a1e203283112de7307 not found: ID does not exist" containerID="c057ef4ed0a49f018e6f6b24631d89320ef6eb174994e3a1e203283112de7307" Sep 17 00:59:46.601692 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.601597 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c057ef4ed0a49f018e6f6b24631d89320ef6eb174994e3a1e203283112de7307"} err="failed to get container status \"c057ef4ed0a49f018e6f6b24631d89320ef6eb174994e3a1e203283112de7307\": rpc error: code = NotFound desc = could not find container \"c057ef4ed0a49f018e6f6b24631d89320ef6eb174994e3a1e203283112de7307\": container with ID starting with c057ef4ed0a49f018e6f6b24631d89320ef6eb174994e3a1e203283112de7307 not found: ID does not exist" Sep 17 00:59:46.610284 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.610262 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4"] Sep 17 00:59:46.613742 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:46.613718 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-80e99a-65c857f6fd-stkr4"] Sep 17 00:59:47.651613 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:47.651581 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d29c4db-f870-462b-8ab7-d30dff78408d" path="/var/lib/kubelet/pods/1d29c4db-f870-462b-8ab7-d30dff78408d/volumes" Sep 17 00:59:47.766799 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:47.766776 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/1.log" Sep 17 00:59:47.767134 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:47.767117 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/1.log" Sep 17 00:59:47.785920 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:47.785890 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-d9c7f3" Sep 17 00:59:47.787661 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:59:47.787638 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7696-d9c7f3\" is forbidden: User \"system:node:ip-10-0-134-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-177.ec2.internal' and this object" podUID="39b06042-b6ca-47ac-b49c-f26dfffa2a43" pod="opendatahub/test-curl-7696-d9c7f3" Sep 17 00:59:47.908862 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:47.908773 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dj5np\" (UniqueName: \"kubernetes.io/projected/39b06042-b6ca-47ac-b49c-f26dfffa2a43-kube-api-access-dj5np\") pod \"39b06042-b6ca-47ac-b49c-f26dfffa2a43\" (UID: \"39b06042-b6ca-47ac-b49c-f26dfffa2a43\") " Sep 17 00:59:47.911103 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:47.911074 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b06042-b6ca-47ac-b49c-f26dfffa2a43-kube-api-access-dj5np" (OuterVolumeSpecName: "kube-api-access-dj5np") pod "39b06042-b6ca-47ac-b49c-f26dfffa2a43" (UID: "39b06042-b6ca-47ac-b49c-f26dfffa2a43"). InnerVolumeSpecName "kube-api-access-dj5np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 00:59:48.009842 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:48.009807 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj5np\" (UniqueName: \"kubernetes.io/projected/39b06042-b6ca-47ac-b49c-f26dfffa2a43-kube-api-access-dj5np\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 00:59:48.600894 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:48.600864 2582 generic.go:356] "Generic (PLEG): container finished" podID="39b06042-b6ca-47ac-b49c-f26dfffa2a43" containerID="85f1260679d29c0e36db2c2934d91ffe52b338bc7bb09a98327be78d8d33c10c" exitCode=137 Sep 17 00:59:48.601080 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:48.600902 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-d9c7f3" Sep 17 00:59:48.601080 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:48.600955 2582 scope.go:117] "RemoveContainer" containerID="85f1260679d29c0e36db2c2934d91ffe52b338bc7bb09a98327be78d8d33c10c" Sep 17 00:59:48.603226 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:59:48.603196 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7696-d9c7f3\" is forbidden: User \"system:node:ip-10-0-134-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-177.ec2.internal' and this object" podUID="39b06042-b6ca-47ac-b49c-f26dfffa2a43" pod="opendatahub/test-curl-7696-d9c7f3" Sep 17 00:59:48.610071 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:48.610056 2582 scope.go:117] "RemoveContainer" containerID="85f1260679d29c0e36db2c2934d91ffe52b338bc7bb09a98327be78d8d33c10c" Sep 17 00:59:48.610339 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:59:48.610323 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85f1260679d29c0e36db2c2934d91ffe52b338bc7bb09a98327be78d8d33c10c\": container with ID starting with 85f1260679d29c0e36db2c2934d91ffe52b338bc7bb09a98327be78d8d33c10c not found: ID does not exist" containerID="85f1260679d29c0e36db2c2934d91ffe52b338bc7bb09a98327be78d8d33c10c" Sep 17 00:59:48.610388 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:48.610346 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85f1260679d29c0e36db2c2934d91ffe52b338bc7bb09a98327be78d8d33c10c"} err="failed to get container status \"85f1260679d29c0e36db2c2934d91ffe52b338bc7bb09a98327be78d8d33c10c\": rpc error: code = NotFound desc = could not find container \"85f1260679d29c0e36db2c2934d91ffe52b338bc7bb09a98327be78d8d33c10c\": container with ID starting with 85f1260679d29c0e36db2c2934d91ffe52b338bc7bb09a98327be78d8d33c10c not found: ID does not exist" Sep 17 00:59:48.611167 ip-10-0-134-177 kubenswrapper[2582]: E0917 00:59:48.611145 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7696-d9c7f3\" is forbidden: User \"system:node:ip-10-0-134-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-177.ec2.internal' and this object" podUID="39b06042-b6ca-47ac-b49c-f26dfffa2a43" pod="opendatahub/test-curl-7696-d9c7f3" Sep 17 00:59:49.650475 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:49.650437 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39b06042-b6ca-47ac-b49c-f26dfffa2a43" path="/var/lib/kubelet/pods/39b06042-b6ca-47ac-b49c-f26dfffa2a43/volumes" Sep 17 00:59:51.493400 ip-10-0-134-177 kubenswrapper[2582]: I0917 00:59:51.493365 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:00:00.468252 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.468211 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" podUID="f837cc70-d4e0-4b89-bcda-cc1b40ff3179" containerName="main" containerID="cri-o://3c5b07a66f29ff989598ae2d1f8a657b566aeb90b56d85cb2dea50e6927cc436" gracePeriod=45 Sep 17 01:00:00.667926 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.667895 2582 generic.go:356] "Generic (PLEG): container finished" podID="f837cc70-d4e0-4b89-bcda-cc1b40ff3179" containerID="3c5b07a66f29ff989598ae2d1f8a657b566aeb90b56d85cb2dea50e6927cc436" exitCode=0 Sep 17 01:00:00.668081 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.667939 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" event={"ID":"f837cc70-d4e0-4b89-bcda-cc1b40ff3179","Type":"ContainerDied","Data":"3c5b07a66f29ff989598ae2d1f8a657b566aeb90b56d85cb2dea50e6927cc436"} Sep 17 01:00:00.721053 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.720994 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 01:00:00.829610 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.829570 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-model-cache\") pod \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " Sep 17 01:00:00.829610 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.829609 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-tmp-dir\") pod \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " Sep 17 01:00:00.829826 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.829651 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-home\") pod \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " Sep 17 01:00:00.829826 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.829681 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9z5tm\" (UniqueName: \"kubernetes.io/projected/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-kube-api-access-9z5tm\") pod \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " Sep 17 01:00:00.829826 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.829717 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-tls-certs\") pod \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " Sep 17 01:00:00.829826 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.829748 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-dshm\") pod \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\" (UID: \"f837cc70-d4e0-4b89-bcda-cc1b40ff3179\") " Sep 17 01:00:00.830024 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.829875 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-model-cache" (OuterVolumeSpecName: "model-cache") pod "f837cc70-d4e0-4b89-bcda-cc1b40ff3179" (UID: "f837cc70-d4e0-4b89-bcda-cc1b40ff3179"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:00:00.830024 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.829921 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-home" (OuterVolumeSpecName: "home") pod "f837cc70-d4e0-4b89-bcda-cc1b40ff3179" (UID: "f837cc70-d4e0-4b89-bcda-cc1b40ff3179"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:00:00.830024 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.829987 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f837cc70-d4e0-4b89-bcda-cc1b40ff3179" (UID: "f837cc70-d4e0-4b89-bcda-cc1b40ff3179"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:00:00.830238 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.830214 2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-model-cache\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:00:00.830238 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.830240 2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-tmp-dir\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:00:00.830362 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.830248 2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-home\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:00:00.832198 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.832168 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-dshm" (OuterVolumeSpecName: "dshm") pod "f837cc70-d4e0-4b89-bcda-cc1b40ff3179" (UID: "f837cc70-d4e0-4b89-bcda-cc1b40ff3179"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 17 01:00:00.832198 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.832180 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f837cc70-d4e0-4b89-bcda-cc1b40ff3179" (UID: "f837cc70-d4e0-4b89-bcda-cc1b40ff3179"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:00:00.832337 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.832276 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-kube-api-access-9z5tm" (OuterVolumeSpecName: "kube-api-access-9z5tm") pod "f837cc70-d4e0-4b89-bcda-cc1b40ff3179" (UID: "f837cc70-d4e0-4b89-bcda-cc1b40ff3179"). InnerVolumeSpecName "kube-api-access-9z5tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:00:00.931776 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.931737 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z5tm\" (UniqueName: \"kubernetes.io/projected/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-kube-api-access-9z5tm\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:00:00.931776 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.931770 2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-tls-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:00:00.931776 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:00.931779 2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f837cc70-d4e0-4b89-bcda-cc1b40ff3179-dshm\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:00:01.674471 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:01.674442 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" Sep 17 01:00:01.674877 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:01.674440 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn" event={"ID":"f837cc70-d4e0-4b89-bcda-cc1b40ff3179","Type":"ContainerDied","Data":"6da8d3772a714202e55fdfd82993e7a66f09c96f5d26302f37b045e64ae60c11"} Sep 17 01:00:01.674877 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:01.674574 2582 scope.go:117] "RemoveContainer" containerID="3c5b07a66f29ff989598ae2d1f8a657b566aeb90b56d85cb2dea50e6927cc436" Sep 17 01:00:01.694322 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:01.694289 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn"] Sep 17 01:00:01.696659 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:01.696638 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-d2326d/auth-test-model-d2326d-kserve-5f7455c759-nc4pn"] Sep 17 01:00:03.651756 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:03.651704 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f837cc70-d4e0-4b89-bcda-cc1b40ff3179" path="/var/lib/kubelet/pods/f837cc70-d4e0-4b89-bcda-cc1b40ff3179/volumes" Sep 17 01:00:06.496454 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:06.496420 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:00:08.374816 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.374780 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc"] Sep 17 01:00:08.375328 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.375000 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" podUID="050fe979-8c9b-4211-8f81-dc9ae65a63ff" containerName="maas-api" containerID="cri-o://76e11c3f7141b9a8f498f00c47b1692f9c83ede9f8ee4d102438a50ec7c2c5da" gracePeriod=30 Sep 17 01:00:08.628472 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.628405 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" Sep 17 01:00:08.708384 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.708349 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/050fe979-8c9b-4211-8f81-dc9ae65a63ff-maas-api-tls\") pod \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\" (UID: \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\") " Sep 17 01:00:08.708579 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.708441 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/050fe979-8c9b-4211-8f81-dc9ae65a63ff-openshift-service-ca\") pod \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\" (UID: \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\") " Sep 17 01:00:08.708579 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.708468 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-js6kb\" (UniqueName: \"kubernetes.io/projected/050fe979-8c9b-4211-8f81-dc9ae65a63ff-kube-api-access-js6kb\") pod \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\" (UID: \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\") " Sep 17 01:00:08.708579 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.708494 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/050fe979-8c9b-4211-8f81-dc9ae65a63ff-metrics-certs\") pod \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\" (UID: \"050fe979-8c9b-4211-8f81-dc9ae65a63ff\") " Sep 17 01:00:08.708855 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.708827 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/050fe979-8c9b-4211-8f81-dc9ae65a63ff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "050fe979-8c9b-4211-8f81-dc9ae65a63ff" (UID: "050fe979-8c9b-4211-8f81-dc9ae65a63ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:00:08.710918 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.710886 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/050fe979-8c9b-4211-8f81-dc9ae65a63ff-kube-api-access-js6kb" (OuterVolumeSpecName: "kube-api-access-js6kb") pod "050fe979-8c9b-4211-8f81-dc9ae65a63ff" (UID: "050fe979-8c9b-4211-8f81-dc9ae65a63ff"). InnerVolumeSpecName "kube-api-access-js6kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:00:08.710918 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.710884 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/050fe979-8c9b-4211-8f81-dc9ae65a63ff-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "050fe979-8c9b-4211-8f81-dc9ae65a63ff" (UID: "050fe979-8c9b-4211-8f81-dc9ae65a63ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:00:08.711062 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.710993 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/050fe979-8c9b-4211-8f81-dc9ae65a63ff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "050fe979-8c9b-4211-8f81-dc9ae65a63ff" (UID: "050fe979-8c9b-4211-8f81-dc9ae65a63ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:00:08.715447 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.715423 2582 generic.go:356] "Generic (PLEG): container finished" podID="050fe979-8c9b-4211-8f81-dc9ae65a63ff" containerID="76e11c3f7141b9a8f498f00c47b1692f9c83ede9f8ee4d102438a50ec7c2c5da" exitCode=0 Sep 17 01:00:08.715585 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.715465 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" Sep 17 01:00:08.715585 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.715514 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" event={"ID":"050fe979-8c9b-4211-8f81-dc9ae65a63ff","Type":"ContainerDied","Data":"76e11c3f7141b9a8f498f00c47b1692f9c83ede9f8ee4d102438a50ec7c2c5da"} Sep 17 01:00:08.715585 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.715569 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc" event={"ID":"050fe979-8c9b-4211-8f81-dc9ae65a63ff","Type":"ContainerDied","Data":"a66e10552cd5c964ce4628555a82e5f001ee94fa93a72ffc7ca554aaf375f85d"} Sep 17 01:00:08.715764 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.715591 2582 scope.go:117] "RemoveContainer" containerID="76e11c3f7141b9a8f498f00c47b1692f9c83ede9f8ee4d102438a50ec7c2c5da" Sep 17 01:00:08.732757 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.732738 2582 scope.go:117] "RemoveContainer" containerID="76e11c3f7141b9a8f498f00c47b1692f9c83ede9f8ee4d102438a50ec7c2c5da" Sep 17 01:00:08.733073 ip-10-0-134-177 kubenswrapper[2582]: E0917 01:00:08.733052 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e11c3f7141b9a8f498f00c47b1692f9c83ede9f8ee4d102438a50ec7c2c5da\": container with ID starting with 76e11c3f7141b9a8f498f00c47b1692f9c83ede9f8ee4d102438a50ec7c2c5da not found: ID does not exist" containerID="76e11c3f7141b9a8f498f00c47b1692f9c83ede9f8ee4d102438a50ec7c2c5da" Sep 17 01:00:08.733126 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.733084 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e11c3f7141b9a8f498f00c47b1692f9c83ede9f8ee4d102438a50ec7c2c5da"} err="failed to get container status \"76e11c3f7141b9a8f498f00c47b1692f9c83ede9f8ee4d102438a50ec7c2c5da\": rpc error: code = NotFound desc = could not find container \"76e11c3f7141b9a8f498f00c47b1692f9c83ede9f8ee4d102438a50ec7c2c5da\": container with ID starting with 76e11c3f7141b9a8f498f00c47b1692f9c83ede9f8ee4d102438a50ec7c2c5da not found: ID does not exist" Sep 17 01:00:08.741998 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.741978 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc"] Sep 17 01:00:08.745445 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.745425 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c4c89-6769899b69-hldlc"] Sep 17 01:00:08.810355 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.810318 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/050fe979-8c9b-4211-8f81-dc9ae65a63ff-maas-api-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:00:08.810355 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.810348 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/050fe979-8c9b-4211-8f81-dc9ae65a63ff-openshift-service-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:00:08.810355 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.810360 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js6kb\" (UniqueName: \"kubernetes.io/projected/050fe979-8c9b-4211-8f81-dc9ae65a63ff-kube-api-access-js6kb\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:00:08.810608 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:08.810373 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/050fe979-8c9b-4211-8f81-dc9ae65a63ff-metrics-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:00:09.650634 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:09.650606 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="050fe979-8c9b-4211-8f81-dc9ae65a63ff" path="/var/lib/kubelet/pods/050fe979-8c9b-4211-8f81-dc9ae65a63ff/volumes" Sep 17 01:00:20.796713 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:20.796672 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:00:25.857388 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:25.857354 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm"] Sep 17 01:00:25.857897 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:25.857622 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" podUID="a325eb70-884a-4177-82c6-7e20a20822d8" containerName="maas-api" containerID="cri-o://7caed02a01944480101077cbb163da010c0dbb6c0f93e20fd630ccf730212192" gracePeriod=30 Sep 17 01:00:26.103981 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.103955 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" Sep 17 01:00:26.178245 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.178164 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a325eb70-884a-4177-82c6-7e20a20822d8-metrics-certs\") pod \"a325eb70-884a-4177-82c6-7e20a20822d8\" (UID: \"a325eb70-884a-4177-82c6-7e20a20822d8\") " Sep 17 01:00:26.178384 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.178252 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a325eb70-884a-4177-82c6-7e20a20822d8-openshift-service-ca\") pod \"a325eb70-884a-4177-82c6-7e20a20822d8\" (UID: \"a325eb70-884a-4177-82c6-7e20a20822d8\") " Sep 17 01:00:26.178384 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.178272 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a325eb70-884a-4177-82c6-7e20a20822d8-maas-api-tls\") pod \"a325eb70-884a-4177-82c6-7e20a20822d8\" (UID: \"a325eb70-884a-4177-82c6-7e20a20822d8\") " Sep 17 01:00:26.178384 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.178331 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gj79\" (UniqueName: \"kubernetes.io/projected/a325eb70-884a-4177-82c6-7e20a20822d8-kube-api-access-4gj79\") pod \"a325eb70-884a-4177-82c6-7e20a20822d8\" (UID: \"a325eb70-884a-4177-82c6-7e20a20822d8\") " Sep 17 01:00:26.178672 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.178642 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a325eb70-884a-4177-82c6-7e20a20822d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a325eb70-884a-4177-82c6-7e20a20822d8" (UID: "a325eb70-884a-4177-82c6-7e20a20822d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:00:26.180662 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.180626 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a325eb70-884a-4177-82c6-7e20a20822d8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a325eb70-884a-4177-82c6-7e20a20822d8" (UID: "a325eb70-884a-4177-82c6-7e20a20822d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:00:26.180777 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.180638 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a325eb70-884a-4177-82c6-7e20a20822d8-kube-api-access-4gj79" (OuterVolumeSpecName: "kube-api-access-4gj79") pod "a325eb70-884a-4177-82c6-7e20a20822d8" (UID: "a325eb70-884a-4177-82c6-7e20a20822d8"). InnerVolumeSpecName "kube-api-access-4gj79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:00:26.180777 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.180716 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a325eb70-884a-4177-82c6-7e20a20822d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a325eb70-884a-4177-82c6-7e20a20822d8" (UID: "a325eb70-884a-4177-82c6-7e20a20822d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:00:26.279391 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.279360 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gj79\" (UniqueName: \"kubernetes.io/projected/a325eb70-884a-4177-82c6-7e20a20822d8-kube-api-access-4gj79\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:00:26.279391 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.279390 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a325eb70-884a-4177-82c6-7e20a20822d8-metrics-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:00:26.279649 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.279401 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a325eb70-884a-4177-82c6-7e20a20822d8-openshift-service-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:00:26.279649 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.279414 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a325eb70-884a-4177-82c6-7e20a20822d8-maas-api-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:00:26.812786 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.812749 2582 generic.go:356] "Generic (PLEG): container finished" podID="a325eb70-884a-4177-82c6-7e20a20822d8" containerID="7caed02a01944480101077cbb163da010c0dbb6c0f93e20fd630ccf730212192" exitCode=0 Sep 17 01:00:26.812990 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.812798 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" event={"ID":"a325eb70-884a-4177-82c6-7e20a20822d8","Type":"ContainerDied","Data":"7caed02a01944480101077cbb163da010c0dbb6c0f93e20fd630ccf730212192"} Sep 17 01:00:26.812990 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.812849 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" event={"ID":"a325eb70-884a-4177-82c6-7e20a20822d8","Type":"ContainerDied","Data":"dc538696c89d76dde5a8ed3270cd2688d5cd5416fe1aacc8407f2f735567adbc"} Sep 17 01:00:26.812990 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.812876 2582 scope.go:117] "RemoveContainer" containerID="7caed02a01944480101077cbb163da010c0dbb6c0f93e20fd630ccf730212192" Sep 17 01:00:26.813159 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.813028 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm" Sep 17 01:00:26.831561 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.831521 2582 scope.go:117] "RemoveContainer" containerID="7caed02a01944480101077cbb163da010c0dbb6c0f93e20fd630ccf730212192" Sep 17 01:00:26.831975 ip-10-0-134-177 kubenswrapper[2582]: E0917 01:00:26.831944 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7caed02a01944480101077cbb163da010c0dbb6c0f93e20fd630ccf730212192\": container with ID starting with 7caed02a01944480101077cbb163da010c0dbb6c0f93e20fd630ccf730212192 not found: ID does not exist" containerID="7caed02a01944480101077cbb163da010c0dbb6c0f93e20fd630ccf730212192" Sep 17 01:00:26.832058 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.831987 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7caed02a01944480101077cbb163da010c0dbb6c0f93e20fd630ccf730212192"} err="failed to get container status \"7caed02a01944480101077cbb163da010c0dbb6c0f93e20fd630ccf730212192\": rpc error: code = NotFound desc = could not find container \"7caed02a01944480101077cbb163da010c0dbb6c0f93e20fd630ccf730212192\": container with ID starting with 7caed02a01944480101077cbb163da010c0dbb6c0f93e20fd630ccf730212192 not found: ID does not exist" Sep 17 01:00:26.858114 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.858067 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm"] Sep 17 01:00:26.861802 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:26.861771 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-929f3a-5c77d4bdb7-mmxdm"] Sep 17 01:00:27.651140 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:27.651096 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a325eb70-884a-4177-82c6-7e20a20822d8" path="/var/lib/kubelet/pods/a325eb70-884a-4177-82c6-7e20a20822d8/volumes" Sep 17 01:00:32.692154 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:32.692122 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:00:42.987027 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:42.986989 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz"] Sep 17 01:00:42.987606 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:42.987259 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" podUID="78d09de2-5e5d-4aa8-a557-61df130c7e3d" containerName="maas-api" containerID="cri-o://4c88f4e64feef7fd5403565676ae35359845e78147668219364aafe6a4c0c9cf" gracePeriod=30 Sep 17 01:00:43.242812 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.242744 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" Sep 17 01:00:43.348850 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.348814 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78d09de2-5e5d-4aa8-a557-61df130c7e3d-maas-api-tls\") pod \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\" (UID: \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\") " Sep 17 01:00:43.349045 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.348879 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zx52q\" (UniqueName: \"kubernetes.io/projected/78d09de2-5e5d-4aa8-a557-61df130c7e3d-kube-api-access-zx52q\") pod \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\" (UID: \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\") " Sep 17 01:00:43.349045 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.348915 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78d09de2-5e5d-4aa8-a557-61df130c7e3d-openshift-service-ca\") pod \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\" (UID: \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\") " Sep 17 01:00:43.349045 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.348960 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78d09de2-5e5d-4aa8-a557-61df130c7e3d-metrics-certs\") pod \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\" (UID: \"78d09de2-5e5d-4aa8-a557-61df130c7e3d\") " Sep 17 01:00:43.349553 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.349310 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d09de2-5e5d-4aa8-a557-61df130c7e3d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "78d09de2-5e5d-4aa8-a557-61df130c7e3d" (UID: "78d09de2-5e5d-4aa8-a557-61df130c7e3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:00:43.351440 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.351408 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d09de2-5e5d-4aa8-a557-61df130c7e3d-kube-api-access-zx52q" (OuterVolumeSpecName: "kube-api-access-zx52q") pod "78d09de2-5e5d-4aa8-a557-61df130c7e3d" (UID: "78d09de2-5e5d-4aa8-a557-61df130c7e3d"). InnerVolumeSpecName "kube-api-access-zx52q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:00:43.351623 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.351595 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78d09de2-5e5d-4aa8-a557-61df130c7e3d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "78d09de2-5e5d-4aa8-a557-61df130c7e3d" (UID: "78d09de2-5e5d-4aa8-a557-61df130c7e3d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:00:43.351795 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.351773 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78d09de2-5e5d-4aa8-a557-61df130c7e3d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "78d09de2-5e5d-4aa8-a557-61df130c7e3d" (UID: "78d09de2-5e5d-4aa8-a557-61df130c7e3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:00:43.450070 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.450027 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78d09de2-5e5d-4aa8-a557-61df130c7e3d-maas-api-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:00:43.450070 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.450057 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zx52q\" (UniqueName: \"kubernetes.io/projected/78d09de2-5e5d-4aa8-a557-61df130c7e3d-kube-api-access-zx52q\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:00:43.450070 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.450066 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78d09de2-5e5d-4aa8-a557-61df130c7e3d-openshift-service-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:00:43.450070 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.450078 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78d09de2-5e5d-4aa8-a557-61df130c7e3d-metrics-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:00:43.904617 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.904576 2582 generic.go:356] "Generic (PLEG): container finished" podID="78d09de2-5e5d-4aa8-a557-61df130c7e3d" containerID="4c88f4e64feef7fd5403565676ae35359845e78147668219364aafe6a4c0c9cf" exitCode=0 Sep 17 01:00:43.904794 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.904619 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" event={"ID":"78d09de2-5e5d-4aa8-a557-61df130c7e3d","Type":"ContainerDied","Data":"4c88f4e64feef7fd5403565676ae35359845e78147668219364aafe6a4c0c9cf"} Sep 17 01:00:43.904794 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.904641 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" Sep 17 01:00:43.904794 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.904677 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz" event={"ID":"78d09de2-5e5d-4aa8-a557-61df130c7e3d","Type":"ContainerDied","Data":"93c844197f59b4606366bf6f0bdfc00454a3d0e0117c4ab6e591249fb1f8657b"} Sep 17 01:00:43.904794 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.904699 2582 scope.go:117] "RemoveContainer" containerID="4c88f4e64feef7fd5403565676ae35359845e78147668219364aafe6a4c0c9cf" Sep 17 01:00:43.915177 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.915151 2582 scope.go:117] "RemoveContainer" containerID="4c88f4e64feef7fd5403565676ae35359845e78147668219364aafe6a4c0c9cf" Sep 17 01:00:43.915480 ip-10-0-134-177 kubenswrapper[2582]: E0917 01:00:43.915458 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c88f4e64feef7fd5403565676ae35359845e78147668219364aafe6a4c0c9cf\": container with ID starting with 4c88f4e64feef7fd5403565676ae35359845e78147668219364aafe6a4c0c9cf not found: ID does not exist" containerID="4c88f4e64feef7fd5403565676ae35359845e78147668219364aafe6a4c0c9cf" Sep 17 01:00:43.915597 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.915488 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c88f4e64feef7fd5403565676ae35359845e78147668219364aafe6a4c0c9cf"} err="failed to get container status \"4c88f4e64feef7fd5403565676ae35359845e78147668219364aafe6a4c0c9cf\": rpc error: code = NotFound desc = could not find container \"4c88f4e64feef7fd5403565676ae35359845e78147668219364aafe6a4c0c9cf\": container with ID starting with 4c88f4e64feef7fd5403565676ae35359845e78147668219364aafe6a4c0c9cf not found: ID does not exist" Sep 17 01:00:43.920117 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.920089 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz"] Sep 17 01:00:43.925328 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:43.925301 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-93017d-6cc76484c6-kkkqz"] Sep 17 01:00:45.650773 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:45.650736 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78d09de2-5e5d-4aa8-a557-61df130c7e3d" path="/var/lib/kubelet/pods/78d09de2-5e5d-4aa8-a557-61df130c7e3d/volumes" Sep 17 01:00:48.189888 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:00:48.189852 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:01:01.087844 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:01.087805 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:01:05.108432 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.108395 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7696-2e2bc3"] Sep 17 01:01:05.108854 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.108811 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f837cc70-d4e0-4b89-bcda-cc1b40ff3179" containerName="main" Sep 17 01:01:05.108854 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.108822 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f837cc70-d4e0-4b89-bcda-cc1b40ff3179" containerName="main" Sep 17 01:01:05.108854 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.108846 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d29c4db-f870-462b-8ab7-d30dff78408d" containerName="maas-api" Sep 17 01:01:05.108854 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.108851 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d29c4db-f870-462b-8ab7-d30dff78408d" containerName="maas-api" Sep 17 01:01:05.109009 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.108857 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39b06042-b6ca-47ac-b49c-f26dfffa2a43" containerName="test-curl-7696-d9c7f3" Sep 17 01:01:05.109009 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.108863 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="39b06042-b6ca-47ac-b49c-f26dfffa2a43" containerName="test-curl-7696-d9c7f3" Sep 17 01:01:05.109009 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.108868 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a325eb70-884a-4177-82c6-7e20a20822d8" containerName="maas-api" Sep 17 01:01:05.109009 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.108873 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a325eb70-884a-4177-82c6-7e20a20822d8" containerName="maas-api" Sep 17 01:01:05.109009 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.108884 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78d09de2-5e5d-4aa8-a557-61df130c7e3d" containerName="maas-api" Sep 17 01:01:05.109009 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.108889 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="78d09de2-5e5d-4aa8-a557-61df130c7e3d" containerName="maas-api" Sep 17 01:01:05.109009 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.108896 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="050fe979-8c9b-4211-8f81-dc9ae65a63ff" containerName="maas-api" Sep 17 01:01:05.109009 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.108901 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="050fe979-8c9b-4211-8f81-dc9ae65a63ff" containerName="maas-api" Sep 17 01:01:05.109009 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.108960 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d29c4db-f870-462b-8ab7-d30dff78408d" containerName="maas-api" Sep 17 01:01:05.109009 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.108970 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="050fe979-8c9b-4211-8f81-dc9ae65a63ff" containerName="maas-api" Sep 17 01:01:05.109009 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.108977 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f837cc70-d4e0-4b89-bcda-cc1b40ff3179" containerName="main" Sep 17 01:01:05.109009 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.108984 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="78d09de2-5e5d-4aa8-a557-61df130c7e3d" containerName="maas-api" Sep 17 01:01:05.109009 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.108989 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="39b06042-b6ca-47ac-b49c-f26dfffa2a43" containerName="test-curl-7696-d9c7f3" Sep 17 01:01:05.109009 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.108995 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a325eb70-884a-4177-82c6-7e20a20822d8" containerName="maas-api" Sep 17 01:01:05.112184 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.112167 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-2e2bc3" Sep 17 01:01:05.114634 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.114609 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bd8qq\"" Sep 17 01:01:05.117130 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.117107 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-2e2bc3"] Sep 17 01:01:05.160820 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.160791 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wngn4\" (UniqueName: \"kubernetes.io/projected/8b1d0832-90a7-497d-8ccf-4806e940a35e-kube-api-access-wngn4\") pod \"test-curl-7696-2e2bc3\" (UID: \"8b1d0832-90a7-497d-8ccf-4806e940a35e\") " pod="opendatahub/test-curl-7696-2e2bc3" Sep 17 01:01:05.262118 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.262072 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wngn4\" (UniqueName: \"kubernetes.io/projected/8b1d0832-90a7-497d-8ccf-4806e940a35e-kube-api-access-wngn4\") pod \"test-curl-7696-2e2bc3\" (UID: \"8b1d0832-90a7-497d-8ccf-4806e940a35e\") " pod="opendatahub/test-curl-7696-2e2bc3" Sep 17 01:01:05.269103 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.269082 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wngn4\" (UniqueName: \"kubernetes.io/projected/8b1d0832-90a7-497d-8ccf-4806e940a35e-kube-api-access-wngn4\") pod \"test-curl-7696-2e2bc3\" (UID: \"8b1d0832-90a7-497d-8ccf-4806e940a35e\") " pod="opendatahub/test-curl-7696-2e2bc3" Sep 17 01:01:05.422208 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.422119 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-2e2bc3" Sep 17 01:01:05.691165 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:05.691089 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-2e2bc3"] Sep 17 01:01:05.694735 ip-10-0-134-177 kubenswrapper[2582]: W0917 01:01:05.694702 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b1d0832_90a7_497d_8ccf_4806e940a35e.slice/crio-40e3b6e9394fef512e47dbfed93d3131b8b40a08cc78bc2e4978d21eff27d253 WatchSource:0}: Error finding container 40e3b6e9394fef512e47dbfed93d3131b8b40a08cc78bc2e4978d21eff27d253: Status 404 returned error can't find the container with id 40e3b6e9394fef512e47dbfed93d3131b8b40a08cc78bc2e4978d21eff27d253 Sep 17 01:01:06.020318 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:06.020235 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-2e2bc3" event={"ID":"8b1d0832-90a7-497d-8ccf-4806e940a35e","Type":"ContainerStarted","Data":"a78e02d65d1713ee021d726db479dc0227c2b96358939f7920d547f442de8ced"} Sep 17 01:01:06.020318 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:06.020277 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-2e2bc3" event={"ID":"8b1d0832-90a7-497d-8ccf-4806e940a35e","Type":"ContainerStarted","Data":"40e3b6e9394fef512e47dbfed93d3131b8b40a08cc78bc2e4978d21eff27d253"} Sep 17 01:01:06.032284 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:06.032225 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7696-2e2bc3" podStartSLOduration=1.032206575 podStartE2EDuration="1.032206575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:01:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:01:06.032060912 +0000 UTC m=+1878.962694398" watchObservedRunningTime="2026-09-17 01:01:06.032206575 +0000 UTC m=+1878.962840062" Sep 17 01:01:06.657140 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:06.657100 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7696-2e2bc3"] Sep 17 01:01:06.661139 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:06.661105 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7696-2e2bc3"] Sep 17 01:01:06.781913 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:06.781818 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7696-a579a1"] Sep 17 01:01:06.782244 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:06.782232 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b1d0832-90a7-497d-8ccf-4806e940a35e" containerName="test-curl-7696-2e2bc3" Sep 17 01:01:06.782290 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:06.782245 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b1d0832-90a7-497d-8ccf-4806e940a35e" containerName="test-curl-7696-2e2bc3" Sep 17 01:01:06.782336 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:06.782327 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b1d0832-90a7-497d-8ccf-4806e940a35e" containerName="test-curl-7696-2e2bc3" Sep 17 01:01:06.785395 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:06.785377 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-a579a1" Sep 17 01:01:06.788263 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:06.788220 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-a579a1"] Sep 17 01:01:06.878869 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:06.878825 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptdxx\" (UniqueName: \"kubernetes.io/projected/4bb0b7f2-3080-473d-a059-358b37287899-kube-api-access-ptdxx\") pod \"test-curl-7696-a579a1\" (UID: \"4bb0b7f2-3080-473d-a059-358b37287899\") " pod="opendatahub/test-curl-7696-a579a1" Sep 17 01:01:06.980224 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:06.980186 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptdxx\" (UniqueName: \"kubernetes.io/projected/4bb0b7f2-3080-473d-a059-358b37287899-kube-api-access-ptdxx\") pod \"test-curl-7696-a579a1\" (UID: \"4bb0b7f2-3080-473d-a059-358b37287899\") " pod="opendatahub/test-curl-7696-a579a1" Sep 17 01:01:06.988017 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:06.987987 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptdxx\" (UniqueName: \"kubernetes.io/projected/4bb0b7f2-3080-473d-a059-358b37287899-kube-api-access-ptdxx\") pod \"test-curl-7696-a579a1\" (UID: \"4bb0b7f2-3080-473d-a059-358b37287899\") " pod="opendatahub/test-curl-7696-a579a1" Sep 17 01:01:07.025547 ip-10-0-134-177 kubenswrapper[2582]: E0917 01:01:07.025517 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7696-2e2bc3" container="test-curl-7696-2e2bc3" Sep 17 01:01:07.096881 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:07.096849 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-a579a1" Sep 17 01:01:07.365319 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:07.365290 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7696-a579a1"] Sep 17 01:01:07.367271 ip-10-0-134-177 kubenswrapper[2582]: W0917 01:01:07.367244 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bb0b7f2_3080_473d_a059_358b37287899.slice/crio-c58aaecdd442977bdec3bc705dcf56b74fff3f495ee97109ee9639e431848d8b WatchSource:0}: Error finding container c58aaecdd442977bdec3bc705dcf56b74fff3f495ee97109ee9639e431848d8b: Status 404 returned error can't find the container with id c58aaecdd442977bdec3bc705dcf56b74fff3f495ee97109ee9639e431848d8b Sep 17 01:01:08.032251 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:08.032211 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-a579a1" event={"ID":"4bb0b7f2-3080-473d-a059-358b37287899","Type":"ContainerStarted","Data":"b197a20ec9e5e7956f7b26de0d65b41674b67c0fb732f490ec26f4528bfe79db"} Sep 17 01:01:08.032724 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:08.032258 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7696-a579a1" event={"ID":"4bb0b7f2-3080-473d-a059-358b37287899","Type":"ContainerStarted","Data":"c58aaecdd442977bdec3bc705dcf56b74fff3f495ee97109ee9639e431848d8b"} Sep 17 01:01:08.032724 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:08.032332 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7696-2e2bc3" podUID="8b1d0832-90a7-497d-8ccf-4806e940a35e" containerName="test-curl-7696-2e2bc3" containerID="cri-o://a78e02d65d1713ee021d726db479dc0227c2b96358939f7920d547f442de8ced" gracePeriod=2 Sep 17 01:01:08.045737 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:08.045681 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7696-a579a1" podStartSLOduration=2.04566186 podStartE2EDuration="2.04566186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:01:08.044939954 +0000 UTC m=+1880.975573442" watchObservedRunningTime="2026-09-17 01:01:08.04566186 +0000 UTC m=+1880.976295348" Sep 17 01:01:08.666413 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:08.666332 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7696-a579a1"] Sep 17 01:01:08.670312 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:08.670282 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7696-a579a1"] Sep 17 01:01:09.037145 ip-10-0-134-177 kubenswrapper[2582]: E0917 01:01:09.037116 2582 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7696-a579a1" container="test-curl-7696-a579a1" Sep 17 01:01:10.042742 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:10.042705 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7696-a579a1" podUID="4bb0b7f2-3080-473d-a059-358b37287899" containerName="test-curl-7696-a579a1" containerID="cri-o://b197a20ec9e5e7956f7b26de0d65b41674b67c0fb732f490ec26f4528bfe79db" gracePeriod=2 Sep 17 01:01:10.222664 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:10.222638 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-2e2bc3" Sep 17 01:01:10.311079 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:10.311040 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wngn4\" (UniqueName: \"kubernetes.io/projected/8b1d0832-90a7-497d-8ccf-4806e940a35e-kube-api-access-wngn4\") pod \"8b1d0832-90a7-497d-8ccf-4806e940a35e\" (UID: \"8b1d0832-90a7-497d-8ccf-4806e940a35e\") " Sep 17 01:01:10.313279 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:10.313251 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1d0832-90a7-497d-8ccf-4806e940a35e-kube-api-access-wngn4" (OuterVolumeSpecName: "kube-api-access-wngn4") pod "8b1d0832-90a7-497d-8ccf-4806e940a35e" (UID: "8b1d0832-90a7-497d-8ccf-4806e940a35e"). InnerVolumeSpecName "kube-api-access-wngn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:01:10.412654 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:10.412616 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wngn4\" (UniqueName: \"kubernetes.io/projected/8b1d0832-90a7-497d-8ccf-4806e940a35e-kube-api-access-wngn4\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:01:11.049713 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:11.049679 2582 generic.go:356] "Generic (PLEG): container finished" podID="8b1d0832-90a7-497d-8ccf-4806e940a35e" containerID="a78e02d65d1713ee021d726db479dc0227c2b96358939f7920d547f442de8ced" exitCode=137 Sep 17 01:01:11.050109 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:11.049727 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-2e2bc3" Sep 17 01:01:11.050109 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:11.049782 2582 scope.go:117] "RemoveContainer" containerID="a78e02d65d1713ee021d726db479dc0227c2b96358939f7920d547f442de8ced" Sep 17 01:01:11.058969 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:11.058950 2582 scope.go:117] "RemoveContainer" containerID="a78e02d65d1713ee021d726db479dc0227c2b96358939f7920d547f442de8ced" Sep 17 01:01:11.059285 ip-10-0-134-177 kubenswrapper[2582]: E0917 01:01:11.059264 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a78e02d65d1713ee021d726db479dc0227c2b96358939f7920d547f442de8ced\": container with ID starting with a78e02d65d1713ee021d726db479dc0227c2b96358939f7920d547f442de8ced not found: ID does not exist" containerID="a78e02d65d1713ee021d726db479dc0227c2b96358939f7920d547f442de8ced" Sep 17 01:01:11.059352 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:11.059294 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a78e02d65d1713ee021d726db479dc0227c2b96358939f7920d547f442de8ced"} err="failed to get container status \"a78e02d65d1713ee021d726db479dc0227c2b96358939f7920d547f442de8ced\": rpc error: code = NotFound desc = could not find container \"a78e02d65d1713ee021d726db479dc0227c2b96358939f7920d547f442de8ced\": container with ID starting with a78e02d65d1713ee021d726db479dc0227c2b96358939f7920d547f442de8ced not found: ID does not exist" Sep 17 01:01:11.059556 ip-10-0-134-177 kubenswrapper[2582]: E0917 01:01:11.059538 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7696-2e2bc3\" is forbidden: User \"system:node:ip-10-0-134-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-177.ec2.internal' and this object" podUID="8b1d0832-90a7-497d-8ccf-4806e940a35e" pod="opendatahub/test-curl-7696-2e2bc3" Sep 17 01:01:11.650658 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:11.650627 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b1d0832-90a7-497d-8ccf-4806e940a35e" path="/var/lib/kubelet/pods/8b1d0832-90a7-497d-8ccf-4806e940a35e/volumes" Sep 17 01:01:12.238763 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:12.238734 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-a579a1" Sep 17 01:01:12.328761 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:12.328676 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptdxx\" (UniqueName: \"kubernetes.io/projected/4bb0b7f2-3080-473d-a059-358b37287899-kube-api-access-ptdxx\") pod \"4bb0b7f2-3080-473d-a059-358b37287899\" (UID: \"4bb0b7f2-3080-473d-a059-358b37287899\") " Sep 17 01:01:12.330931 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:12.330901 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb0b7f2-3080-473d-a059-358b37287899-kube-api-access-ptdxx" (OuterVolumeSpecName: "kube-api-access-ptdxx") pod "4bb0b7f2-3080-473d-a059-358b37287899" (UID: "4bb0b7f2-3080-473d-a059-358b37287899"). InnerVolumeSpecName "kube-api-access-ptdxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:01:12.430198 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:12.430161 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptdxx\" (UniqueName: \"kubernetes.io/projected/4bb0b7f2-3080-473d-a059-358b37287899-kube-api-access-ptdxx\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:01:13.061932 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:13.061894 2582 generic.go:356] "Generic (PLEG): container finished" podID="4bb0b7f2-3080-473d-a059-358b37287899" containerID="b197a20ec9e5e7956f7b26de0d65b41674b67c0fb732f490ec26f4528bfe79db" exitCode=137 Sep 17 01:01:13.062144 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:13.061937 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7696-a579a1" Sep 17 01:01:13.062144 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:13.061993 2582 scope.go:117] "RemoveContainer" containerID="b197a20ec9e5e7956f7b26de0d65b41674b67c0fb732f490ec26f4528bfe79db" Sep 17 01:01:13.071022 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:13.071004 2582 scope.go:117] "RemoveContainer" containerID="b197a20ec9e5e7956f7b26de0d65b41674b67c0fb732f490ec26f4528bfe79db" Sep 17 01:01:13.071299 ip-10-0-134-177 kubenswrapper[2582]: E0917 01:01:13.071278 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b197a20ec9e5e7956f7b26de0d65b41674b67c0fb732f490ec26f4528bfe79db\": container with ID starting with b197a20ec9e5e7956f7b26de0d65b41674b67c0fb732f490ec26f4528bfe79db not found: ID does not exist" containerID="b197a20ec9e5e7956f7b26de0d65b41674b67c0fb732f490ec26f4528bfe79db" Sep 17 01:01:13.071344 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:13.071308 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b197a20ec9e5e7956f7b26de0d65b41674b67c0fb732f490ec26f4528bfe79db"} err="failed to get container status \"b197a20ec9e5e7956f7b26de0d65b41674b67c0fb732f490ec26f4528bfe79db\": rpc error: code = NotFound desc = could not find container \"b197a20ec9e5e7956f7b26de0d65b41674b67c0fb732f490ec26f4528bfe79db\": container with ID starting with b197a20ec9e5e7956f7b26de0d65b41674b67c0fb732f490ec26f4528bfe79db not found: ID does not exist" Sep 17 01:01:13.071615 ip-10-0-134-177 kubenswrapper[2582]: E0917 01:01:13.071595 2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7696-a579a1\" is forbidden: User \"system:node:ip-10-0-134-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-177.ec2.internal' and this object" podUID="4bb0b7f2-3080-473d-a059-358b37287899" pod="opendatahub/test-curl-7696-a579a1" Sep 17 01:01:13.650753 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:13.650722 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bb0b7f2-3080-473d-a059-358b37287899" path="/var/lib/kubelet/pods/4bb0b7f2-3080-473d-a059-358b37287899/volumes" Sep 17 01:01:14.690616 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:14.690577 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:01:30.393933 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:30.393900 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:01:54.689984 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:01:54.689937 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:02:09.889316 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:09.889276 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:02:27.089030 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:27.088988 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:02:38.989400 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:38.989320 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:02:48.696326 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:48.696288 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:02:49.338387 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:49.338351 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr"] Sep 17 01:02:49.339134 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:49.339107 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bb0b7f2-3080-473d-a059-358b37287899" containerName="test-curl-7696-a579a1" Sep 17 01:02:49.339134 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:49.339135 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bb0b7f2-3080-473d-a059-358b37287899" containerName="test-curl-7696-a579a1" Sep 17 01:02:49.339319 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:49.339259 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bb0b7f2-3080-473d-a059-358b37287899" containerName="test-curl-7696-a579a1" Sep 17 01:02:49.342965 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:49.342946 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" Sep 17 01:02:49.344998 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:49.344979 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-7e9533-serving-cert\"" Sep 17 01:02:49.350894 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:49.350870 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr"] Sep 17 01:02:49.418767 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:49.418733 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ce6135-16ad-413b-a858-6a9c8b6d79ff-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr\" (UID: \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" Sep 17 01:02:49.418939 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:49.418784 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8lz8\" (UniqueName: \"kubernetes.io/projected/32ce6135-16ad-413b-a858-6a9c8b6d79ff-kube-api-access-w8lz8\") pod \"maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr\" (UID: \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" Sep 17 01:02:49.418939 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:49.418874 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ce6135-16ad-413b-a858-6a9c8b6d79ff-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr\" (UID: \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" Sep 17 01:02:49.418939 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:49.418930 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32ce6135-16ad-413b-a858-6a9c8b6d79ff-metrics-certs\") pod \"maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr\" (UID: \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" Sep 17 01:02:49.519750 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:49.519708 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ce6135-16ad-413b-a858-6a9c8b6d79ff-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr\" (UID: \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" Sep 17 01:02:49.519945 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:49.519809 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8lz8\" (UniqueName: \"kubernetes.io/projected/32ce6135-16ad-413b-a858-6a9c8b6d79ff-kube-api-access-w8lz8\") pod \"maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr\" (UID: \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" Sep 17 01:02:49.519945 ip-10-0-134-177 kubenswrapper[2582]: E0917 01:02:49.519860 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-serving-cert: secret "maas-api-e2e-ipp-a-7e9533-serving-cert" not found Sep 17 01:02:49.519945 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:49.519869 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ce6135-16ad-413b-a858-6a9c8b6d79ff-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr\" (UID: \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" Sep 17 01:02:49.519945 ip-10-0-134-177 kubenswrapper[2582]: E0917 01:02:49.519924 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32ce6135-16ad-413b-a858-6a9c8b6d79ff-maas-api-tls podName:32ce6135-16ad-413b-a858-6a9c8b6d79ff nodeName:}" failed. No retries permitted until 2026-09-17 01:02:50.019903555 +0000 UTC m=+1982.950537033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32ce6135-16ad-413b-a858-6a9c8b6d79ff-maas-api-tls") pod "maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" (UID: "32ce6135-16ad-413b-a858-6a9c8b6d79ff") : secret "maas-api-e2e-ipp-a-7e9533-serving-cert" not found Sep 17 01:02:49.520155 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:49.519956 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32ce6135-16ad-413b-a858-6a9c8b6d79ff-metrics-certs\") pod \"maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr\" (UID: \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" Sep 17 01:02:49.520485 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:49.520459 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ce6135-16ad-413b-a858-6a9c8b6d79ff-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr\" (UID: \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" Sep 17 01:02:49.522700 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:49.522676 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32ce6135-16ad-413b-a858-6a9c8b6d79ff-metrics-certs\") pod \"maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr\" (UID: \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" Sep 17 01:02:49.529964 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:49.529942 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8lz8\" (UniqueName: \"kubernetes.io/projected/32ce6135-16ad-413b-a858-6a9c8b6d79ff-kube-api-access-w8lz8\") pod \"maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr\" (UID: \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" Sep 17 01:02:50.023834 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:50.023799 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ce6135-16ad-413b-a858-6a9c8b6d79ff-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr\" (UID: \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" Sep 17 01:02:50.026401 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:50.026374 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ce6135-16ad-413b-a858-6a9c8b6d79ff-maas-api-tls\") pod \"maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr\" (UID: \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" Sep 17 01:02:50.255404 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:50.255367 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" Sep 17 01:02:50.527223 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:50.527198 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr"] Sep 17 01:02:50.529606 ip-10-0-134-177 kubenswrapper[2582]: W0917 01:02:50.529576 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ce6135_16ad_413b_a858_6a9c8b6d79ff.slice/crio-0592b7b508ce61ed03da4431907234f24eacc67da9db8262d579f4f81d7df30d WatchSource:0}: Error finding container 0592b7b508ce61ed03da4431907234f24eacc67da9db8262d579f4f81d7df30d: Status 404 returned error can't find the container with id 0592b7b508ce61ed03da4431907234f24eacc67da9db8262d579f4f81d7df30d Sep 17 01:02:50.569841 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:50.569757 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" event={"ID":"32ce6135-16ad-413b-a858-6a9c8b6d79ff","Type":"ContainerStarted","Data":"0592b7b508ce61ed03da4431907234f24eacc67da9db8262d579f4f81d7df30d"} Sep 17 01:02:51.576012 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:51.575975 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" event={"ID":"32ce6135-16ad-413b-a858-6a9c8b6d79ff","Type":"ContainerStarted","Data":"d03c28a7ede73ec5373a186e5f062ae4ae189d8642c91ddccb3c33e5bf54bb48"} Sep 17 01:02:51.576429 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:51.576040 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" Sep 17 01:02:51.594283 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:51.594235 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" podStartSLOduration=2.159522749 podStartE2EDuration="2.594218158s" totalImagesPullingTime="434.695409ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:02:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:02:51.592131817 +0000 UTC m=+1984.522765304" watchObservedRunningTime="2026-09-17 01:02:51.594218158 +0000 UTC m=+1984.524851688" Sep 17 01:02:57.586055 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:57.586025 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" Sep 17 01:02:59.593553 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:02:59.593515 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:03:18.824124 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:18.824089 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b"] Sep 17 01:03:18.828036 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:18.828015 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" Sep 17 01:03:18.830090 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:18.830071 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c13796-serving-cert\"" Sep 17 01:03:18.837772 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:18.837250 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b"] Sep 17 01:03:18.886304 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:18.886271 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a58770e-717f-4861-ab39-b09659eeb051-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b\" (UID: \"1a58770e-717f-4861-ab39-b09659eeb051\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" Sep 17 01:03:18.886482 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:18.886331 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9j6m\" (UniqueName: \"kubernetes.io/projected/1a58770e-717f-4861-ab39-b09659eeb051-kube-api-access-q9j6m\") pod \"maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b\" (UID: \"1a58770e-717f-4861-ab39-b09659eeb051\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" Sep 17 01:03:18.886482 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:18.886407 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a58770e-717f-4861-ab39-b09659eeb051-metrics-certs\") pod \"maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b\" (UID: \"1a58770e-717f-4861-ab39-b09659eeb051\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" Sep 17 01:03:18.886482 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:18.886458 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a58770e-717f-4861-ab39-b09659eeb051-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b\" (UID: \"1a58770e-717f-4861-ab39-b09659eeb051\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" Sep 17 01:03:18.987822 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:18.987783 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a58770e-717f-4861-ab39-b09659eeb051-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b\" (UID: \"1a58770e-717f-4861-ab39-b09659eeb051\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" Sep 17 01:03:18.988011 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:18.987865 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9j6m\" (UniqueName: \"kubernetes.io/projected/1a58770e-717f-4861-ab39-b09659eeb051-kube-api-access-q9j6m\") pod \"maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b\" (UID: \"1a58770e-717f-4861-ab39-b09659eeb051\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" Sep 17 01:03:18.988011 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:18.987915 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a58770e-717f-4861-ab39-b09659eeb051-metrics-certs\") pod \"maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b\" (UID: \"1a58770e-717f-4861-ab39-b09659eeb051\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" Sep 17 01:03:18.988011 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:18.987968 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a58770e-717f-4861-ab39-b09659eeb051-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b\" (UID: \"1a58770e-717f-4861-ab39-b09659eeb051\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" Sep 17 01:03:18.988143 ip-10-0-134-177 kubenswrapper[2582]: E0917 01:03:18.988087 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-serving-cert: secret "maas-api-e2e-ipp-b-c13796-serving-cert" not found Sep 17 01:03:18.988190 ip-10-0-134-177 kubenswrapper[2582]: E0917 01:03:18.988170 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a58770e-717f-4861-ab39-b09659eeb051-maas-api-tls podName:1a58770e-717f-4861-ab39-b09659eeb051 nodeName:}" failed. No retries permitted until 2026-09-17 01:03:19.488149192 +0000 UTC m=+2012.418782674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1a58770e-717f-4861-ab39-b09659eeb051-maas-api-tls") pod "maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" (UID: "1a58770e-717f-4861-ab39-b09659eeb051") : secret "maas-api-e2e-ipp-b-c13796-serving-cert" not found Sep 17 01:03:18.988611 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:18.988590 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a58770e-717f-4861-ab39-b09659eeb051-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b\" (UID: \"1a58770e-717f-4861-ab39-b09659eeb051\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" Sep 17 01:03:18.990468 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:18.990447 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a58770e-717f-4861-ab39-b09659eeb051-metrics-certs\") pod \"maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b\" (UID: \"1a58770e-717f-4861-ab39-b09659eeb051\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" Sep 17 01:03:18.995754 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:18.995733 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9j6m\" (UniqueName: \"kubernetes.io/projected/1a58770e-717f-4861-ab39-b09659eeb051-kube-api-access-q9j6m\") pod \"maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b\" (UID: \"1a58770e-717f-4861-ab39-b09659eeb051\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" Sep 17 01:03:19.491567 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:19.491493 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a58770e-717f-4861-ab39-b09659eeb051-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b\" (UID: \"1a58770e-717f-4861-ab39-b09659eeb051\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" Sep 17 01:03:19.494056 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:19.494037 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a58770e-717f-4861-ab39-b09659eeb051-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b\" (UID: \"1a58770e-717f-4861-ab39-b09659eeb051\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" Sep 17 01:03:19.741905 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:19.741812 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" Sep 17 01:03:20.015272 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:20.015243 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b"] Sep 17 01:03:20.017848 ip-10-0-134-177 kubenswrapper[2582]: W0917 01:03:20.017822 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a58770e_717f_4861_ab39_b09659eeb051.slice/crio-4513bf0a426709c53258bcd1382100dcc66b07022cf8547a8a2d58ffe92fd6b2 WatchSource:0}: Error finding container 4513bf0a426709c53258bcd1382100dcc66b07022cf8547a8a2d58ffe92fd6b2: Status 404 returned error can't find the container with id 4513bf0a426709c53258bcd1382100dcc66b07022cf8547a8a2d58ffe92fd6b2 Sep 17 01:03:20.725801 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:20.725707 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" event={"ID":"1a58770e-717f-4861-ab39-b09659eeb051","Type":"ContainerStarted","Data":"ce8fad54a174a5d54e6f11a4ef85099cf55e6465fd6b138488fd6f723c0b4ecc"} Sep 17 01:03:20.725801 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:20.725763 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" event={"ID":"1a58770e-717f-4861-ab39-b09659eeb051","Type":"ContainerStarted","Data":"4513bf0a426709c53258bcd1382100dcc66b07022cf8547a8a2d58ffe92fd6b2"} Sep 17 01:03:20.726019 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:20.725835 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" Sep 17 01:03:20.741800 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:20.741746 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" podStartSLOduration=2.425641748 podStartE2EDuration="2.741730167s" totalImagesPullingTime="316.088419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:03:20.739201692 +0000 UTC m=+2013.669835178" watchObservedRunningTime="2026-09-17 01:03:20.741730167 +0000 UTC m=+2013.672363654" Sep 17 01:03:24.596935 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:24.596895 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:03:26.736797 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:03:26.736762 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" Sep 17 01:04:00.695882 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:00.695847 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:04:13.896242 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:13.896156 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:04:26.492833 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:26.492798 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw"] Sep 17 01:04:26.504287 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:26.503886 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" Sep 17 01:04:26.506213 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:26.506187 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-9187f3-serving-cert\"" Sep 17 01:04:26.507983 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:26.507941 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw"] Sep 17 01:04:26.631637 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:26.631602 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4dxt\" (UniqueName: \"kubernetes.io/projected/32d66199-2f8a-493c-a000-80fc64d764a9-kube-api-access-q4dxt\") pod \"maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw\" (UID: \"32d66199-2f8a-493c-a000-80fc64d764a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" Sep 17 01:04:26.631801 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:26.631655 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32d66199-2f8a-493c-a000-80fc64d764a9-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw\" (UID: \"32d66199-2f8a-493c-a000-80fc64d764a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" Sep 17 01:04:26.631801 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:26.631734 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d66199-2f8a-493c-a000-80fc64d764a9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw\" (UID: \"32d66199-2f8a-493c-a000-80fc64d764a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" Sep 17 01:04:26.631888 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:26.631850 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d66199-2f8a-493c-a000-80fc64d764a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw\" (UID: \"32d66199-2f8a-493c-a000-80fc64d764a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" Sep 17 01:04:26.732912 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:26.732885 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4dxt\" (UniqueName: \"kubernetes.io/projected/32d66199-2f8a-493c-a000-80fc64d764a9-kube-api-access-q4dxt\") pod \"maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw\" (UID: \"32d66199-2f8a-493c-a000-80fc64d764a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" Sep 17 01:04:26.733041 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:26.732928 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32d66199-2f8a-493c-a000-80fc64d764a9-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw\" (UID: \"32d66199-2f8a-493c-a000-80fc64d764a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" Sep 17 01:04:26.733041 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:26.732970 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d66199-2f8a-493c-a000-80fc64d764a9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw\" (UID: \"32d66199-2f8a-493c-a000-80fc64d764a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" Sep 17 01:04:26.733151 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:26.733052 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d66199-2f8a-493c-a000-80fc64d764a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw\" (UID: \"32d66199-2f8a-493c-a000-80fc64d764a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" Sep 17 01:04:26.733206 ip-10-0-134-177 kubenswrapper[2582]: E0917 01:04:26.733158 2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-serving-cert: secret "maas-api-e2e-ipp-cleanup-9187f3-serving-cert" not found Sep 17 01:04:26.733258 ip-10-0-134-177 kubenswrapper[2582]: E0917 01:04:26.733243 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d66199-2f8a-493c-a000-80fc64d764a9-maas-api-tls podName:32d66199-2f8a-493c-a000-80fc64d764a9 nodeName:}" failed. No retries permitted until 2026-09-17 01:04:27.233220725 +0000 UTC m=+2080.163854210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32d66199-2f8a-493c-a000-80fc64d764a9-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" (UID: "32d66199-2f8a-493c-a000-80fc64d764a9") : secret "maas-api-e2e-ipp-cleanup-9187f3-serving-cert" not found Sep 17 01:04:26.733723 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:26.733697 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d66199-2f8a-493c-a000-80fc64d764a9-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw\" (UID: \"32d66199-2f8a-493c-a000-80fc64d764a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" Sep 17 01:04:26.735700 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:26.735679 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32d66199-2f8a-493c-a000-80fc64d764a9-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw\" (UID: \"32d66199-2f8a-493c-a000-80fc64d764a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" Sep 17 01:04:26.740418 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:26.740390 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4dxt\" (UniqueName: \"kubernetes.io/projected/32d66199-2f8a-493c-a000-80fc64d764a9-kube-api-access-q4dxt\") pod \"maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw\" (UID: \"32d66199-2f8a-493c-a000-80fc64d764a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" Sep 17 01:04:27.237939 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:27.237902 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d66199-2f8a-493c-a000-80fc64d764a9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw\" (UID: \"32d66199-2f8a-493c-a000-80fc64d764a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" Sep 17 01:04:27.240431 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:27.240410 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d66199-2f8a-493c-a000-80fc64d764a9-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw\" (UID: \"32d66199-2f8a-493c-a000-80fc64d764a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" Sep 17 01:04:27.418603 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:27.418567 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" Sep 17 01:04:27.691565 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:27.691539 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw"] Sep 17 01:04:27.693622 ip-10-0-134-177 kubenswrapper[2582]: W0917 01:04:27.693593 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d66199_2f8a_493c_a000_80fc64d764a9.slice/crio-9f5cb000d720e493278195af2aec2e4e965e6f75eb571a860e14414dc5174244 WatchSource:0}: Error finding container 9f5cb000d720e493278195af2aec2e4e965e6f75eb571a860e14414dc5174244: Status 404 returned error can't find the container with id 9f5cb000d720e493278195af2aec2e4e965e6f75eb571a860e14414dc5174244 Sep 17 01:04:27.695002 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:27.694976 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 01:04:28.073003 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:28.072976 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" event={"ID":"32d66199-2f8a-493c-a000-80fc64d764a9","Type":"ContainerStarted","Data":"9f5cb000d720e493278195af2aec2e4e965e6f75eb571a860e14414dc5174244"} Sep 17 01:04:29.079311 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:29.079276 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" event={"ID":"32d66199-2f8a-493c-a000-80fc64d764a9","Type":"ContainerStarted","Data":"0a4e9fe68d79bb75ffb9b431370af72301542a7c83d1819a636c7ebff30985b4"} Sep 17 01:04:29.079740 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:29.079339 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" Sep 17 01:04:29.093726 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:29.093636 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" podStartSLOduration=2.7439260819999998 podStartE2EDuration="3.093618546s" totalImagesPullingTime="349.692464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:04:29.093341238 +0000 UTC m=+2082.023974726" watchObservedRunningTime="2026-09-17 01:04:29.093618546 +0000 UTC m=+2082.024252034" Sep 17 01:04:32.296907 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:32.296872 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:04:35.089767 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:35.089733 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" Sep 17 01:04:47.802673 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:47.802646 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/1.log" Sep 17 01:04:47.804971 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:47.804360 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/1.log" Sep 17 01:04:55.377158 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:55.377123 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw"] Sep 17 01:04:55.377584 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:55.377386 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" podUID="32d66199-2f8a-493c-a000-80fc64d764a9" containerName="maas-api" containerID="cri-o://0a4e9fe68d79bb75ffb9b431370af72301542a7c83d1819a636c7ebff30985b4" gracePeriod=30 Sep 17 01:04:55.632004 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:55.631947 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" Sep 17 01:04:55.697967 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:55.697939 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4dxt\" (UniqueName: \"kubernetes.io/projected/32d66199-2f8a-493c-a000-80fc64d764a9-kube-api-access-q4dxt\") pod \"32d66199-2f8a-493c-a000-80fc64d764a9\" (UID: \"32d66199-2f8a-493c-a000-80fc64d764a9\") " Sep 17 01:04:55.698121 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:55.697992 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d66199-2f8a-493c-a000-80fc64d764a9-openshift-service-ca\") pod \"32d66199-2f8a-493c-a000-80fc64d764a9\" (UID: \"32d66199-2f8a-493c-a000-80fc64d764a9\") " Sep 17 01:04:55.698121 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:55.698027 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d66199-2f8a-493c-a000-80fc64d764a9-maas-api-tls\") pod \"32d66199-2f8a-493c-a000-80fc64d764a9\" (UID: \"32d66199-2f8a-493c-a000-80fc64d764a9\") " Sep 17 01:04:55.698121 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:55.698049 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32d66199-2f8a-493c-a000-80fc64d764a9-metrics-certs\") pod \"32d66199-2f8a-493c-a000-80fc64d764a9\" (UID: \"32d66199-2f8a-493c-a000-80fc64d764a9\") " Sep 17 01:04:55.698406 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:55.698386 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d66199-2f8a-493c-a000-80fc64d764a9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "32d66199-2f8a-493c-a000-80fc64d764a9" (UID: "32d66199-2f8a-493c-a000-80fc64d764a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:04:55.700335 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:55.700305 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d66199-2f8a-493c-a000-80fc64d764a9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "32d66199-2f8a-493c-a000-80fc64d764a9" (UID: "32d66199-2f8a-493c-a000-80fc64d764a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:04:55.700443 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:55.700369 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d66199-2f8a-493c-a000-80fc64d764a9-kube-api-access-q4dxt" (OuterVolumeSpecName: "kube-api-access-q4dxt") pod "32d66199-2f8a-493c-a000-80fc64d764a9" (UID: "32d66199-2f8a-493c-a000-80fc64d764a9"). InnerVolumeSpecName "kube-api-access-q4dxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:04:55.700443 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:55.700390 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d66199-2f8a-493c-a000-80fc64d764a9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "32d66199-2f8a-493c-a000-80fc64d764a9" (UID: "32d66199-2f8a-493c-a000-80fc64d764a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:04:55.799867 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:55.799833 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32d66199-2f8a-493c-a000-80fc64d764a9-metrics-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:04:55.799867 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:55.799861 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4dxt\" (UniqueName: \"kubernetes.io/projected/32d66199-2f8a-493c-a000-80fc64d764a9-kube-api-access-q4dxt\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:04:55.799867 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:55.799870 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32d66199-2f8a-493c-a000-80fc64d764a9-openshift-service-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:04:55.800101 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:55.799879 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32d66199-2f8a-493c-a000-80fc64d764a9-maas-api-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:04:56.221009 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:56.220973 2582 generic.go:356] "Generic (PLEG): container finished" podID="32d66199-2f8a-493c-a000-80fc64d764a9" containerID="0a4e9fe68d79bb75ffb9b431370af72301542a7c83d1819a636c7ebff30985b4" exitCode=0 Sep 17 01:04:56.221212 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:56.221024 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" Sep 17 01:04:56.221212 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:56.221029 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" event={"ID":"32d66199-2f8a-493c-a000-80fc64d764a9","Type":"ContainerDied","Data":"0a4e9fe68d79bb75ffb9b431370af72301542a7c83d1819a636c7ebff30985b4"} Sep 17 01:04:56.221212 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:56.221143 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw" event={"ID":"32d66199-2f8a-493c-a000-80fc64d764a9","Type":"ContainerDied","Data":"9f5cb000d720e493278195af2aec2e4e965e6f75eb571a860e14414dc5174244"} Sep 17 01:04:56.221212 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:56.221166 2582 scope.go:117] "RemoveContainer" containerID="0a4e9fe68d79bb75ffb9b431370af72301542a7c83d1819a636c7ebff30985b4" Sep 17 01:04:56.238834 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:56.238814 2582 scope.go:117] "RemoveContainer" containerID="0a4e9fe68d79bb75ffb9b431370af72301542a7c83d1819a636c7ebff30985b4" Sep 17 01:04:56.239090 ip-10-0-134-177 kubenswrapper[2582]: E0917 01:04:56.239069 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a4e9fe68d79bb75ffb9b431370af72301542a7c83d1819a636c7ebff30985b4\": container with ID starting with 0a4e9fe68d79bb75ffb9b431370af72301542a7c83d1819a636c7ebff30985b4 not found: ID does not exist" containerID="0a4e9fe68d79bb75ffb9b431370af72301542a7c83d1819a636c7ebff30985b4" Sep 17 01:04:56.239139 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:56.239097 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4e9fe68d79bb75ffb9b431370af72301542a7c83d1819a636c7ebff30985b4"} err="failed to get container status \"0a4e9fe68d79bb75ffb9b431370af72301542a7c83d1819a636c7ebff30985b4\": rpc error: code = NotFound desc = could not find container \"0a4e9fe68d79bb75ffb9b431370af72301542a7c83d1819a636c7ebff30985b4\": container with ID starting with 0a4e9fe68d79bb75ffb9b431370af72301542a7c83d1819a636c7ebff30985b4 not found: ID does not exist" Sep 17 01:04:56.250332 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:56.250299 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw"] Sep 17 01:04:56.253975 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:56.253953 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-9187f3-548c55bf66-6gfzw"] Sep 17 01:04:57.651509 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:57.651458 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32d66199-2f8a-493c-a000-80fc64d764a9" path="/var/lib/kubelet/pods/32d66199-2f8a-493c-a000-80fc64d764a9/volumes" Sep 17 01:04:58.696754 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:04:58.696722 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:05:18.698017 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:18.697979 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:05:21.102519 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.102466 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr"] Sep 17 01:05:21.102988 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.102778 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" podUID="32ce6135-16ad-413b-a858-6a9c8b6d79ff" containerName="maas-api" containerID="cri-o://d03c28a7ede73ec5373a186e5f062ae4ae189d8642c91ddccb3c33e5bf54bb48" gracePeriod=30 Sep 17 01:05:21.352171 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.352146 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" Sep 17 01:05:21.369702 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.369611 2582 generic.go:356] "Generic (PLEG): container finished" podID="32ce6135-16ad-413b-a858-6a9c8b6d79ff" containerID="d03c28a7ede73ec5373a186e5f062ae4ae189d8642c91ddccb3c33e5bf54bb48" exitCode=0 Sep 17 01:05:21.369702 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.369666 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" event={"ID":"32ce6135-16ad-413b-a858-6a9c8b6d79ff","Type":"ContainerDied","Data":"d03c28a7ede73ec5373a186e5f062ae4ae189d8642c91ddccb3c33e5bf54bb48"} Sep 17 01:05:21.369914 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.369674 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" Sep 17 01:05:21.369914 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.369715 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr" event={"ID":"32ce6135-16ad-413b-a858-6a9c8b6d79ff","Type":"ContainerDied","Data":"0592b7b508ce61ed03da4431907234f24eacc67da9db8262d579f4f81d7df30d"} Sep 17 01:05:21.369914 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.369727 2582 scope.go:117] "RemoveContainer" containerID="d03c28a7ede73ec5373a186e5f062ae4ae189d8642c91ddccb3c33e5bf54bb48" Sep 17 01:05:21.379717 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.379691 2582 scope.go:117] "RemoveContainer" containerID="d03c28a7ede73ec5373a186e5f062ae4ae189d8642c91ddccb3c33e5bf54bb48" Sep 17 01:05:21.380042 ip-10-0-134-177 kubenswrapper[2582]: E0917 01:05:21.380006 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03c28a7ede73ec5373a186e5f062ae4ae189d8642c91ddccb3c33e5bf54bb48\": container with ID starting with d03c28a7ede73ec5373a186e5f062ae4ae189d8642c91ddccb3c33e5bf54bb48 not found: ID does not exist" containerID="d03c28a7ede73ec5373a186e5f062ae4ae189d8642c91ddccb3c33e5bf54bb48" Sep 17 01:05:21.380147 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.380054 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03c28a7ede73ec5373a186e5f062ae4ae189d8642c91ddccb3c33e5bf54bb48"} err="failed to get container status \"d03c28a7ede73ec5373a186e5f062ae4ae189d8642c91ddccb3c33e5bf54bb48\": rpc error: code = NotFound desc = could not find container \"d03c28a7ede73ec5373a186e5f062ae4ae189d8642c91ddccb3c33e5bf54bb48\": container with ID starting with d03c28a7ede73ec5373a186e5f062ae4ae189d8642c91ddccb3c33e5bf54bb48 not found: ID does not exist" Sep 17 01:05:21.442573 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.442536 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ce6135-16ad-413b-a858-6a9c8b6d79ff-maas-api-tls\") pod \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\" (UID: \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\") " Sep 17 01:05:21.442743 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.442618 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32ce6135-16ad-413b-a858-6a9c8b6d79ff-metrics-certs\") pod \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\" (UID: \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\") " Sep 17 01:05:21.442743 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.442676 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8lz8\" (UniqueName: \"kubernetes.io/projected/32ce6135-16ad-413b-a858-6a9c8b6d79ff-kube-api-access-w8lz8\") pod \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\" (UID: \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\") " Sep 17 01:05:21.442743 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.442716 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ce6135-16ad-413b-a858-6a9c8b6d79ff-openshift-service-ca\") pod \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\" (UID: \"32ce6135-16ad-413b-a858-6a9c8b6d79ff\") " Sep 17 01:05:21.443199 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.443166 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ce6135-16ad-413b-a858-6a9c8b6d79ff-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "32ce6135-16ad-413b-a858-6a9c8b6d79ff" (UID: "32ce6135-16ad-413b-a858-6a9c8b6d79ff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:05:21.444969 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.444943 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ce6135-16ad-413b-a858-6a9c8b6d79ff-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "32ce6135-16ad-413b-a858-6a9c8b6d79ff" (UID: "32ce6135-16ad-413b-a858-6a9c8b6d79ff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:05:21.444969 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.444955 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ce6135-16ad-413b-a858-6a9c8b6d79ff-kube-api-access-w8lz8" (OuterVolumeSpecName: "kube-api-access-w8lz8") pod "32ce6135-16ad-413b-a858-6a9c8b6d79ff" (UID: "32ce6135-16ad-413b-a858-6a9c8b6d79ff"). InnerVolumeSpecName "kube-api-access-w8lz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:05:21.445194 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.445170 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ce6135-16ad-413b-a858-6a9c8b6d79ff-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "32ce6135-16ad-413b-a858-6a9c8b6d79ff" (UID: "32ce6135-16ad-413b-a858-6a9c8b6d79ff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:05:21.543615 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.543580 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ce6135-16ad-413b-a858-6a9c8b6d79ff-openshift-service-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:05:21.543615 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.543609 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32ce6135-16ad-413b-a858-6a9c8b6d79ff-maas-api-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:05:21.543615 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.543618 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32ce6135-16ad-413b-a858-6a9c8b6d79ff-metrics-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:05:21.543863 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.543626 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8lz8\" (UniqueName: \"kubernetes.io/projected/32ce6135-16ad-413b-a858-6a9c8b6d79ff-kube-api-access-w8lz8\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:05:21.683993 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.683962 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr"] Sep 17 01:05:21.687050 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:21.687025 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-7e9533-56cb89bbb9-ghzdr"] Sep 17 01:05:23.650640 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:23.650609 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32ce6135-16ad-413b-a858-6a9c8b6d79ff" path="/var/lib/kubelet/pods/32ce6135-16ad-413b-a858-6a9c8b6d79ff/volumes" Sep 17 01:05:31.694206 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:05:31.694172 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:06:20.345215 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.345181 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b"] Sep 17 01:06:20.345848 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.345453 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" podUID="1a58770e-717f-4861-ab39-b09659eeb051" containerName="maas-api" containerID="cri-o://ce8fad54a174a5d54e6f11a4ef85099cf55e6465fd6b138488fd6f723c0b4ecc" gracePeriod=30 Sep 17 01:06:20.604407 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.604332 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" Sep 17 01:06:20.687325 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.687294 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9j6m\" (UniqueName: \"kubernetes.io/projected/1a58770e-717f-4861-ab39-b09659eeb051-kube-api-access-q9j6m\") pod \"1a58770e-717f-4861-ab39-b09659eeb051\" (UID: \"1a58770e-717f-4861-ab39-b09659eeb051\") " Sep 17 01:06:20.687530 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.687343 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a58770e-717f-4861-ab39-b09659eeb051-maas-api-tls\") pod \"1a58770e-717f-4861-ab39-b09659eeb051\" (UID: \"1a58770e-717f-4861-ab39-b09659eeb051\") " Sep 17 01:06:20.687530 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.687406 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a58770e-717f-4861-ab39-b09659eeb051-metrics-certs\") pod \"1a58770e-717f-4861-ab39-b09659eeb051\" (UID: \"1a58770e-717f-4861-ab39-b09659eeb051\") " Sep 17 01:06:20.687530 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.687451 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a58770e-717f-4861-ab39-b09659eeb051-openshift-service-ca\") pod \"1a58770e-717f-4861-ab39-b09659eeb051\" (UID: \"1a58770e-717f-4861-ab39-b09659eeb051\") " Sep 17 01:06:20.688561 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.687960 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a58770e-717f-4861-ab39-b09659eeb051-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1a58770e-717f-4861-ab39-b09659eeb051" (UID: "1a58770e-717f-4861-ab39-b09659eeb051"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:06:20.688561 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.688059 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a58770e-717f-4861-ab39-b09659eeb051-openshift-service-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:06:20.690022 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.689988 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a58770e-717f-4861-ab39-b09659eeb051-kube-api-access-q9j6m" (OuterVolumeSpecName: "kube-api-access-q9j6m") pod "1a58770e-717f-4861-ab39-b09659eeb051" (UID: "1a58770e-717f-4861-ab39-b09659eeb051"). InnerVolumeSpecName "kube-api-access-q9j6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:06:20.690259 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.690233 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a58770e-717f-4861-ab39-b09659eeb051-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1a58770e-717f-4861-ab39-b09659eeb051" (UID: "1a58770e-717f-4861-ab39-b09659eeb051"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:06:20.690259 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.690244 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a58770e-717f-4861-ab39-b09659eeb051-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1a58770e-717f-4861-ab39-b09659eeb051" (UID: "1a58770e-717f-4861-ab39-b09659eeb051"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:06:20.690412 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.690283 2582 generic.go:356] "Generic (PLEG): container finished" podID="1a58770e-717f-4861-ab39-b09659eeb051" containerID="ce8fad54a174a5d54e6f11a4ef85099cf55e6465fd6b138488fd6f723c0b4ecc" exitCode=0 Sep 17 01:06:20.690412 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.690330 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" Sep 17 01:06:20.690412 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.690376 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" event={"ID":"1a58770e-717f-4861-ab39-b09659eeb051","Type":"ContainerDied","Data":"ce8fad54a174a5d54e6f11a4ef85099cf55e6465fd6b138488fd6f723c0b4ecc"} Sep 17 01:06:20.690412 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.690414 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b" event={"ID":"1a58770e-717f-4861-ab39-b09659eeb051","Type":"ContainerDied","Data":"4513bf0a426709c53258bcd1382100dcc66b07022cf8547a8a2d58ffe92fd6b2"} Sep 17 01:06:20.690657 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.690434 2582 scope.go:117] "RemoveContainer" containerID="ce8fad54a174a5d54e6f11a4ef85099cf55e6465fd6b138488fd6f723c0b4ecc" Sep 17 01:06:20.710491 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.710476 2582 scope.go:117] "RemoveContainer" containerID="ce8fad54a174a5d54e6f11a4ef85099cf55e6465fd6b138488fd6f723c0b4ecc" Sep 17 01:06:20.710813 ip-10-0-134-177 kubenswrapper[2582]: E0917 01:06:20.710796 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8fad54a174a5d54e6f11a4ef85099cf55e6465fd6b138488fd6f723c0b4ecc\": container with ID starting with ce8fad54a174a5d54e6f11a4ef85099cf55e6465fd6b138488fd6f723c0b4ecc not found: ID does not exist" containerID="ce8fad54a174a5d54e6f11a4ef85099cf55e6465fd6b138488fd6f723c0b4ecc" Sep 17 01:06:20.710854 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.710822 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8fad54a174a5d54e6f11a4ef85099cf55e6465fd6b138488fd6f723c0b4ecc"} err="failed to get container status \"ce8fad54a174a5d54e6f11a4ef85099cf55e6465fd6b138488fd6f723c0b4ecc\": rpc error: code = NotFound desc = could not find container \"ce8fad54a174a5d54e6f11a4ef85099cf55e6465fd6b138488fd6f723c0b4ecc\": container with ID starting with ce8fad54a174a5d54e6f11a4ef85099cf55e6465fd6b138488fd6f723c0b4ecc not found: ID does not exist" Sep 17 01:06:20.719123 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.719101 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b"] Sep 17 01:06:20.725825 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.725800 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c13796-59b87d64cf-t7h7b"] Sep 17 01:06:20.789582 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.789552 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a58770e-717f-4861-ab39-b09659eeb051-metrics-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:06:20.789582 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.789576 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9j6m\" (UniqueName: \"kubernetes.io/projected/1a58770e-717f-4861-ab39-b09659eeb051-kube-api-access-q9j6m\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:06:20.789582 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:20.789586 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a58770e-717f-4861-ab39-b09659eeb051-maas-api-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:06:21.651019 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:21.650974 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a58770e-717f-4861-ab39-b09659eeb051" path="/var/lib/kubelet/pods/1a58770e-717f-4861-ab39-b09659eeb051/volumes" Sep 17 01:06:24.896941 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:24.896910 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:06:37.544388 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:37.544351 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt"] Sep 17 01:06:37.547070 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:37.547011 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" podUID="7ba8810a-4bee-4105-899c-0356e2936a50" containerName="maas-api" containerID="cri-o://85d61f023dfa6c5380f648e4ec5263caa3787fc31e5a30b35e75a3b9aef61410" gracePeriod=30 Sep 17 01:06:37.776809 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:37.776777 2582 generic.go:356] "Generic (PLEG): container finished" podID="7ba8810a-4bee-4105-899c-0356e2936a50" containerID="85d61f023dfa6c5380f648e4ec5263caa3787fc31e5a30b35e75a3b9aef61410" exitCode=0 Sep 17 01:06:37.776955 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:37.776860 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" event={"ID":"7ba8810a-4bee-4105-899c-0356e2936a50","Type":"ContainerDied","Data":"85d61f023dfa6c5380f648e4ec5263caa3787fc31e5a30b35e75a3b9aef61410"} Sep 17 01:06:37.805484 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:37.805429 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" Sep 17 01:06:37.946406 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:37.946377 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9744l\" (UniqueName: \"kubernetes.io/projected/7ba8810a-4bee-4105-899c-0356e2936a50-kube-api-access-9744l\") pod \"7ba8810a-4bee-4105-899c-0356e2936a50\" (UID: \"7ba8810a-4bee-4105-899c-0356e2936a50\") " Sep 17 01:06:37.946595 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:37.946444 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba8810a-4bee-4105-899c-0356e2936a50-openshift-service-ca\") pod \"7ba8810a-4bee-4105-899c-0356e2936a50\" (UID: \"7ba8810a-4bee-4105-899c-0356e2936a50\") " Sep 17 01:06:37.946595 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:37.946478 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba8810a-4bee-4105-899c-0356e2936a50-metrics-certs\") pod \"7ba8810a-4bee-4105-899c-0356e2936a50\" (UID: \"7ba8810a-4bee-4105-899c-0356e2936a50\") " Sep 17 01:06:37.946595 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:37.946519 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ba8810a-4bee-4105-899c-0356e2936a50-maas-api-tls\") pod \"7ba8810a-4bee-4105-899c-0356e2936a50\" (UID: \"7ba8810a-4bee-4105-899c-0356e2936a50\") " Sep 17 01:06:37.946905 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:37.946866 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba8810a-4bee-4105-899c-0356e2936a50-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ba8810a-4bee-4105-899c-0356e2936a50" (UID: "7ba8810a-4bee-4105-899c-0356e2936a50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:06:37.948675 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:37.948651 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba8810a-4bee-4105-899c-0356e2936a50-kube-api-access-9744l" (OuterVolumeSpecName: "kube-api-access-9744l") pod "7ba8810a-4bee-4105-899c-0356e2936a50" (UID: "7ba8810a-4bee-4105-899c-0356e2936a50"). InnerVolumeSpecName "kube-api-access-9744l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:06:37.948817 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:37.948798 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ba8810a-4bee-4105-899c-0356e2936a50-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7ba8810a-4bee-4105-899c-0356e2936a50" (UID: "7ba8810a-4bee-4105-899c-0356e2936a50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:06:37.948895 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:37.948872 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ba8810a-4bee-4105-899c-0356e2936a50-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ba8810a-4bee-4105-899c-0356e2936a50" (UID: "7ba8810a-4bee-4105-899c-0356e2936a50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:06:38.047757 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:38.047727 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9744l\" (UniqueName: \"kubernetes.io/projected/7ba8810a-4bee-4105-899c-0356e2936a50-kube-api-access-9744l\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:06:38.047757 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:38.047751 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ba8810a-4bee-4105-899c-0356e2936a50-openshift-service-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:06:38.047757 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:38.047760 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ba8810a-4bee-4105-899c-0356e2936a50-metrics-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:06:38.047947 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:38.047767 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ba8810a-4bee-4105-899c-0356e2936a50-maas-api-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:06:38.782950 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:38.782914 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" event={"ID":"7ba8810a-4bee-4105-899c-0356e2936a50","Type":"ContainerDied","Data":"3d518fe1a16c604bc0eb10798fc2c3be256ee9b2d0193c6b28e1195b5bcae7a2"} Sep 17 01:06:38.782950 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:38.782935 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt" Sep 17 01:06:38.783444 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:38.782976 2582 scope.go:117] "RemoveContainer" containerID="85d61f023dfa6c5380f648e4ec5263caa3787fc31e5a30b35e75a3b9aef61410" Sep 17 01:06:38.802457 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:38.802432 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt"] Sep 17 01:06:38.805484 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:38.805458 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d2326d-57cbf4cb87-rjvkt"] Sep 17 01:06:39.651195 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:39.651161 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ba8810a-4bee-4105-899c-0356e2936a50" path="/var/lib/kubelet/pods/7ba8810a-4bee-4105-899c-0356e2936a50/volumes" Sep 17 01:06:40.398422 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:40.398388 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:06:55.005087 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.005050 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj"] Sep 17 01:06:55.005722 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.005283 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" podUID="b2abdefa-266d-42a2-9a9a-ac9ddd845e35" containerName="maas-api" containerID="cri-o://2f0ec55ea2edafabb991958140f80333cbf278960860b0490d4460067bd05117" gracePeriod=30 Sep 17 01:06:55.255291 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.255237 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" Sep 17 01:06:55.300111 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.300079 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-openshift-service-ca\") pod \"b2abdefa-266d-42a2-9a9a-ac9ddd845e35\" (UID: \"b2abdefa-266d-42a2-9a9a-ac9ddd845e35\") " Sep 17 01:06:55.300253 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.300130 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-maas-api-tls\") pod \"b2abdefa-266d-42a2-9a9a-ac9ddd845e35\" (UID: \"b2abdefa-266d-42a2-9a9a-ac9ddd845e35\") " Sep 17 01:06:55.300253 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.300208 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-metrics-certs\") pod \"b2abdefa-266d-42a2-9a9a-ac9ddd845e35\" (UID: \"b2abdefa-266d-42a2-9a9a-ac9ddd845e35\") " Sep 17 01:06:55.300377 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.300254 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp4nr\" (UniqueName: \"kubernetes.io/projected/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-kube-api-access-mp4nr\") pod \"b2abdefa-266d-42a2-9a9a-ac9ddd845e35\" (UID: \"b2abdefa-266d-42a2-9a9a-ac9ddd845e35\") " Sep 17 01:06:55.300473 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.300450 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b2abdefa-266d-42a2-9a9a-ac9ddd845e35" (UID: "b2abdefa-266d-42a2-9a9a-ac9ddd845e35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:06:55.300597 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.300575 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-openshift-service-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:06:55.302392 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.302365 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b2abdefa-266d-42a2-9a9a-ac9ddd845e35" (UID: "b2abdefa-266d-42a2-9a9a-ac9ddd845e35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:06:55.302474 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.302448 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-kube-api-access-mp4nr" (OuterVolumeSpecName: "kube-api-access-mp4nr") pod "b2abdefa-266d-42a2-9a9a-ac9ddd845e35" (UID: "b2abdefa-266d-42a2-9a9a-ac9ddd845e35"). InnerVolumeSpecName "kube-api-access-mp4nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:06:55.302570 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.302549 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b2abdefa-266d-42a2-9a9a-ac9ddd845e35" (UID: "b2abdefa-266d-42a2-9a9a-ac9ddd845e35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:06:55.401580 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.401554 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-metrics-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:06:55.401580 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.401579 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp4nr\" (UniqueName: \"kubernetes.io/projected/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-kube-api-access-mp4nr\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:06:55.401740 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.401591 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b2abdefa-266d-42a2-9a9a-ac9ddd845e35-maas-api-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:06:55.872078 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.872037 2582 generic.go:356] "Generic (PLEG): container finished" podID="b2abdefa-266d-42a2-9a9a-ac9ddd845e35" containerID="2f0ec55ea2edafabb991958140f80333cbf278960860b0490d4460067bd05117" exitCode=0 Sep 17 01:06:55.872078 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.872079 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" Sep 17 01:06:55.872277 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.872126 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" event={"ID":"b2abdefa-266d-42a2-9a9a-ac9ddd845e35","Type":"ContainerDied","Data":"2f0ec55ea2edafabb991958140f80333cbf278960860b0490d4460067bd05117"} Sep 17 01:06:55.872277 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.872170 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj" event={"ID":"b2abdefa-266d-42a2-9a9a-ac9ddd845e35","Type":"ContainerDied","Data":"822d4b241d204e29ffa14480c07b8c60dd3eeea4f329fd14d25c589adb5354be"} Sep 17 01:06:55.872277 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.872204 2582 scope.go:117] "RemoveContainer" containerID="2f0ec55ea2edafabb991958140f80333cbf278960860b0490d4460067bd05117" Sep 17 01:06:55.881257 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.881240 2582 scope.go:117] "RemoveContainer" containerID="2f0ec55ea2edafabb991958140f80333cbf278960860b0490d4460067bd05117" Sep 17 01:06:55.881563 ip-10-0-134-177 kubenswrapper[2582]: E0917 01:06:55.881543 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f0ec55ea2edafabb991958140f80333cbf278960860b0490d4460067bd05117\": container with ID starting with 2f0ec55ea2edafabb991958140f80333cbf278960860b0490d4460067bd05117 not found: ID does not exist" containerID="2f0ec55ea2edafabb991958140f80333cbf278960860b0490d4460067bd05117" Sep 17 01:06:55.881623 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.881573 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f0ec55ea2edafabb991958140f80333cbf278960860b0490d4460067bd05117"} err="failed to get container status \"2f0ec55ea2edafabb991958140f80333cbf278960860b0490d4460067bd05117\": rpc error: code = NotFound desc = could not find container \"2f0ec55ea2edafabb991958140f80333cbf278960860b0490d4460067bd05117\": container with ID starting with 2f0ec55ea2edafabb991958140f80333cbf278960860b0490d4460067bd05117 not found: ID does not exist" Sep 17 01:06:55.886532 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.886488 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj"] Sep 17 01:06:55.895817 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:55.891365 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bcd112-55f7b758c8-969xj"] Sep 17 01:06:57.651003 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:57.650963 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2abdefa-266d-42a2-9a9a-ac9ddd845e35" path="/var/lib/kubelet/pods/b2abdefa-266d-42a2-9a9a-ac9ddd845e35/volumes" Sep 17 01:06:57.684008 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:06:57.683967 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:07:06.488447 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:07:06.488407 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:08:27.897904 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:08:27.897872 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:08:53.691345 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:08:53.691268 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:08:59.994305 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:08:59.994268 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:09:07.291817 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:09:07.291781 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:09:14.691596 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:09:14.691562 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:09:27.495174 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:09:27.495139 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:09:37.485346 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:09:37.485315 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:09:47.837019 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:09:47.836916 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/1.log" Sep 17 01:09:47.842752 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:09:47.842195 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/1.log" Sep 17 01:11:07.386381 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:11:07.386347 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:11:15.403963 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:11:15.403924 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:11:22.594555 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:11:22.594518 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:11:36.196158 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:11:36.196121 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:12:13.496324 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:13.496233 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:12:20.994092 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:20.994054 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:12:28.194469 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.194431 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:12:28.717616 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.717579 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb"] Sep 17 01:12:28.717982 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.717969 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2abdefa-266d-42a2-9a9a-ac9ddd845e35" containerName="maas-api" Sep 17 01:12:28.717982 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.717983 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2abdefa-266d-42a2-9a9a-ac9ddd845e35" containerName="maas-api" Sep 17 01:12:28.718071 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.717992 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a58770e-717f-4861-ab39-b09659eeb051" containerName="maas-api" Sep 17 01:12:28.718071 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.717998 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a58770e-717f-4861-ab39-b09659eeb051" containerName="maas-api" Sep 17 01:12:28.718071 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.718005 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32d66199-2f8a-493c-a000-80fc64d764a9" containerName="maas-api" Sep 17 01:12:28.718071 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.718011 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="32d66199-2f8a-493c-a000-80fc64d764a9" containerName="maas-api" Sep 17 01:12:28.718071 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.718019 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ba8810a-4bee-4105-899c-0356e2936a50" containerName="maas-api" Sep 17 01:12:28.718071 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.718025 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ba8810a-4bee-4105-899c-0356e2936a50" containerName="maas-api" Sep 17 01:12:28.718071 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.718043 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ce6135-16ad-413b-a858-6a9c8b6d79ff" containerName="maas-api" Sep 17 01:12:28.718071 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.718048 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ce6135-16ad-413b-a858-6a9c8b6d79ff" containerName="maas-api" Sep 17 01:12:28.718353 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.718106 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a58770e-717f-4861-ab39-b09659eeb051" containerName="maas-api" Sep 17 01:12:28.718353 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.718115 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ce6135-16ad-413b-a858-6a9c8b6d79ff" containerName="maas-api" Sep 17 01:12:28.718353 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.718121 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2abdefa-266d-42a2-9a9a-ac9ddd845e35" containerName="maas-api" Sep 17 01:12:28.718353 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.718127 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="32d66199-2f8a-493c-a000-80fc64d764a9" containerName="maas-api" Sep 17 01:12:28.718353 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.718133 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ba8810a-4bee-4105-899c-0356e2936a50" containerName="maas-api" Sep 17 01:12:28.721301 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.721282 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" Sep 17 01:12:28.723902 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.723879 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 17 01:12:28.724012 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.723900 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 17 01:12:28.724012 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.723909 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 17 01:12:28.724012 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.723887 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gxvb4\"" Sep 17 01:12:28.724012 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.723879 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 17 01:12:28.730534 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.730492 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb"] Sep 17 01:12:28.801654 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.801613 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7e32060-b3be-4c72-8cd8-98d099a7d02c-metrics-certs\") pod \"maas-api-cf64cbd4c-rwmgb\" (UID: \"e7e32060-b3be-4c72-8cd8-98d099a7d02c\") " pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" Sep 17 01:12:28.801828 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.801726 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e32060-b3be-4c72-8cd8-98d099a7d02c-openshift-service-ca\") pod \"maas-api-cf64cbd4c-rwmgb\" (UID: \"e7e32060-b3be-4c72-8cd8-98d099a7d02c\") " pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" Sep 17 01:12:28.801828 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.801774 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7e32060-b3be-4c72-8cd8-98d099a7d02c-maas-api-tls\") pod \"maas-api-cf64cbd4c-rwmgb\" (UID: \"e7e32060-b3be-4c72-8cd8-98d099a7d02c\") " pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" Sep 17 01:12:28.801828 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.801796 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d85j\" (UniqueName: \"kubernetes.io/projected/e7e32060-b3be-4c72-8cd8-98d099a7d02c-kube-api-access-2d85j\") pod \"maas-api-cf64cbd4c-rwmgb\" (UID: \"e7e32060-b3be-4c72-8cd8-98d099a7d02c\") " pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" Sep 17 01:12:28.902694 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.902652 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e32060-b3be-4c72-8cd8-98d099a7d02c-openshift-service-ca\") pod \"maas-api-cf64cbd4c-rwmgb\" (UID: \"e7e32060-b3be-4c72-8cd8-98d099a7d02c\") " pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" Sep 17 01:12:28.902886 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.902719 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7e32060-b3be-4c72-8cd8-98d099a7d02c-maas-api-tls\") pod \"maas-api-cf64cbd4c-rwmgb\" (UID: \"e7e32060-b3be-4c72-8cd8-98d099a7d02c\") " pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" Sep 17 01:12:28.902886 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.902752 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d85j\" (UniqueName: \"kubernetes.io/projected/e7e32060-b3be-4c72-8cd8-98d099a7d02c-kube-api-access-2d85j\") pod \"maas-api-cf64cbd4c-rwmgb\" (UID: \"e7e32060-b3be-4c72-8cd8-98d099a7d02c\") " pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" Sep 17 01:12:28.902886 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.902791 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7e32060-b3be-4c72-8cd8-98d099a7d02c-metrics-certs\") pod \"maas-api-cf64cbd4c-rwmgb\" (UID: \"e7e32060-b3be-4c72-8cd8-98d099a7d02c\") " pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" Sep 17 01:12:28.903375 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.903351 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e32060-b3be-4c72-8cd8-98d099a7d02c-openshift-service-ca\") pod \"maas-api-cf64cbd4c-rwmgb\" (UID: \"e7e32060-b3be-4c72-8cd8-98d099a7d02c\") " pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" Sep 17 01:12:28.905416 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.905391 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7e32060-b3be-4c72-8cd8-98d099a7d02c-metrics-certs\") pod \"maas-api-cf64cbd4c-rwmgb\" (UID: \"e7e32060-b3be-4c72-8cd8-98d099a7d02c\") " pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" Sep 17 01:12:28.905582 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.905561 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7e32060-b3be-4c72-8cd8-98d099a7d02c-maas-api-tls\") pod \"maas-api-cf64cbd4c-rwmgb\" (UID: \"e7e32060-b3be-4c72-8cd8-98d099a7d02c\") " pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" Sep 17 01:12:28.910319 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:28.910293 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d85j\" (UniqueName: \"kubernetes.io/projected/e7e32060-b3be-4c72-8cd8-98d099a7d02c-kube-api-access-2d85j\") pod \"maas-api-cf64cbd4c-rwmgb\" (UID: \"e7e32060-b3be-4c72-8cd8-98d099a7d02c\") " pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" Sep 17 01:12:29.033763 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:29.033671 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" Sep 17 01:12:29.305719 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:29.305668 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb"] Sep 17 01:12:29.308200 ip-10-0-134-177 kubenswrapper[2582]: W0917 01:12:29.308163 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e32060_b3be_4c72_8cd8_98d099a7d02c.slice/crio-4e914b3885b3b9fe480e0c7cf368387f25bdc441881931ff03f17fba553b25e3 WatchSource:0}: Error finding container 4e914b3885b3b9fe480e0c7cf368387f25bdc441881931ff03f17fba553b25e3: Status 404 returned error can't find the container with id 4e914b3885b3b9fe480e0c7cf368387f25bdc441881931ff03f17fba553b25e3 Sep 17 01:12:29.309702 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:29.309687 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 17 01:12:29.560775 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:29.560682 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" event={"ID":"e7e32060-b3be-4c72-8cd8-98d099a7d02c","Type":"ContainerStarted","Data":"4e914b3885b3b9fe480e0c7cf368387f25bdc441881931ff03f17fba553b25e3"} Sep 17 01:12:30.567467 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:30.567431 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" event={"ID":"e7e32060-b3be-4c72-8cd8-98d099a7d02c","Type":"ContainerStarted","Data":"28007e4b282d2d6d2193c64ad890266dedc731ebc52591dca2b8cbd47562c323"} Sep 17 01:12:30.567881 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:30.567555 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" Sep 17 01:12:30.583153 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:30.583100 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" podStartSLOduration=2.121877157 podStartE2EDuration="2.583084672s" totalImagesPullingTime="461.207515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:12:30.581163624 +0000 UTC m=+2563.511797127" watchObservedRunningTime="2026-09-17 01:12:30.583084672 +0000 UTC m=+2563.513718158" Sep 17 01:12:36.076021 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.075982 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb"] Sep 17 01:12:36.076518 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.076207 2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" podUID="e7e32060-b3be-4c72-8cd8-98d099a7d02c" containerName="maas-api" containerID="cri-o://28007e4b282d2d6d2193c64ad890266dedc731ebc52591dca2b8cbd47562c323" gracePeriod=30 Sep 17 01:12:36.082400 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.082377 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" Sep 17 01:12:36.105875 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.105846 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w"] Sep 17 01:12:36.109545 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.109523 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w" Sep 17 01:12:36.123814 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.123785 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w"] Sep 17 01:12:36.171004 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.170976 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc538c7b-5b1e-4ee3-8751-03ccd6e0231c-metrics-certs\") pod \"maas-api-79c8ddf6d7-phd9w\" (UID: \"fc538c7b-5b1e-4ee3-8751-03ccd6e0231c\") " pod="odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w" Sep 17 01:12:36.171136 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.171020 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc538c7b-5b1e-4ee3-8751-03ccd6e0231c-openshift-service-ca\") pod \"maas-api-79c8ddf6d7-phd9w\" (UID: \"fc538c7b-5b1e-4ee3-8751-03ccd6e0231c\") " pod="odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w" Sep 17 01:12:36.171136 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.171095 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx6zs\" (UniqueName: \"kubernetes.io/projected/fc538c7b-5b1e-4ee3-8751-03ccd6e0231c-kube-api-access-nx6zs\") pod \"maas-api-79c8ddf6d7-phd9w\" (UID: \"fc538c7b-5b1e-4ee3-8751-03ccd6e0231c\") " pod="odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w" Sep 17 01:12:36.171222 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.171172 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc538c7b-5b1e-4ee3-8751-03ccd6e0231c-maas-api-tls\") pod \"maas-api-79c8ddf6d7-phd9w\" (UID: \"fc538c7b-5b1e-4ee3-8751-03ccd6e0231c\") " pod="odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w" Sep 17 01:12:36.272157 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.272126 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx6zs\" (UniqueName: \"kubernetes.io/projected/fc538c7b-5b1e-4ee3-8751-03ccd6e0231c-kube-api-access-nx6zs\") pod \"maas-api-79c8ddf6d7-phd9w\" (UID: \"fc538c7b-5b1e-4ee3-8751-03ccd6e0231c\") " pod="odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w" Sep 17 01:12:36.272306 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.272177 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc538c7b-5b1e-4ee3-8751-03ccd6e0231c-maas-api-tls\") pod \"maas-api-79c8ddf6d7-phd9w\" (UID: \"fc538c7b-5b1e-4ee3-8751-03ccd6e0231c\") " pod="odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w" Sep 17 01:12:36.272306 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.272227 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc538c7b-5b1e-4ee3-8751-03ccd6e0231c-metrics-certs\") pod \"maas-api-79c8ddf6d7-phd9w\" (UID: \"fc538c7b-5b1e-4ee3-8751-03ccd6e0231c\") " pod="odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w" Sep 17 01:12:36.272306 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.272248 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc538c7b-5b1e-4ee3-8751-03ccd6e0231c-openshift-service-ca\") pod \"maas-api-79c8ddf6d7-phd9w\" (UID: \"fc538c7b-5b1e-4ee3-8751-03ccd6e0231c\") " pod="odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w" Sep 17 01:12:36.273014 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.272985 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc538c7b-5b1e-4ee3-8751-03ccd6e0231c-openshift-service-ca\") pod \"maas-api-79c8ddf6d7-phd9w\" (UID: \"fc538c7b-5b1e-4ee3-8751-03ccd6e0231c\") " pod="odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w" Sep 17 01:12:36.275343 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.275320 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc538c7b-5b1e-4ee3-8751-03ccd6e0231c-metrics-certs\") pod \"maas-api-79c8ddf6d7-phd9w\" (UID: \"fc538c7b-5b1e-4ee3-8751-03ccd6e0231c\") " pod="odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w" Sep 17 01:12:36.275463 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.275419 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc538c7b-5b1e-4ee3-8751-03ccd6e0231c-maas-api-tls\") pod \"maas-api-79c8ddf6d7-phd9w\" (UID: \"fc538c7b-5b1e-4ee3-8751-03ccd6e0231c\") " pod="odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w" Sep 17 01:12:36.279420 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.279395 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx6zs\" (UniqueName: \"kubernetes.io/projected/fc538c7b-5b1e-4ee3-8751-03ccd6e0231c-kube-api-access-nx6zs\") pod \"maas-api-79c8ddf6d7-phd9w\" (UID: \"fc538c7b-5b1e-4ee3-8751-03ccd6e0231c\") " pod="odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w" Sep 17 01:12:36.319574 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.319553 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" Sep 17 01:12:36.373362 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.373281 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7e32060-b3be-4c72-8cd8-98d099a7d02c-maas-api-tls\") pod \"e7e32060-b3be-4c72-8cd8-98d099a7d02c\" (UID: \"e7e32060-b3be-4c72-8cd8-98d099a7d02c\") " Sep 17 01:12:36.373362 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.373331 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7e32060-b3be-4c72-8cd8-98d099a7d02c-metrics-certs\") pod \"e7e32060-b3be-4c72-8cd8-98d099a7d02c\" (UID: \"e7e32060-b3be-4c72-8cd8-98d099a7d02c\") " Sep 17 01:12:36.373362 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.373353 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e32060-b3be-4c72-8cd8-98d099a7d02c-openshift-service-ca\") pod \"e7e32060-b3be-4c72-8cd8-98d099a7d02c\" (UID: \"e7e32060-b3be-4c72-8cd8-98d099a7d02c\") " Sep 17 01:12:36.373768 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.373368 2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2d85j\" (UniqueName: \"kubernetes.io/projected/e7e32060-b3be-4c72-8cd8-98d099a7d02c-kube-api-access-2d85j\") pod \"e7e32060-b3be-4c72-8cd8-98d099a7d02c\" (UID: \"e7e32060-b3be-4c72-8cd8-98d099a7d02c\") " Sep 17 01:12:36.373768 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.373710 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e32060-b3be-4c72-8cd8-98d099a7d02c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e7e32060-b3be-4c72-8cd8-98d099a7d02c" (UID: "e7e32060-b3be-4c72-8cd8-98d099a7d02c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 17 01:12:36.375695 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.375672 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e32060-b3be-4c72-8cd8-98d099a7d02c-kube-api-access-2d85j" (OuterVolumeSpecName: "kube-api-access-2d85j") pod "e7e32060-b3be-4c72-8cd8-98d099a7d02c" (UID: "e7e32060-b3be-4c72-8cd8-98d099a7d02c"). InnerVolumeSpecName "kube-api-access-2d85j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 17 01:12:36.375782 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.375688 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7e32060-b3be-4c72-8cd8-98d099a7d02c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e7e32060-b3be-4c72-8cd8-98d099a7d02c" (UID: "e7e32060-b3be-4c72-8cd8-98d099a7d02c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:12:36.375782 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.375706 2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7e32060-b3be-4c72-8cd8-98d099a7d02c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e7e32060-b3be-4c72-8cd8-98d099a7d02c" (UID: "e7e32060-b3be-4c72-8cd8-98d099a7d02c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 17 01:12:36.468569 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.468536 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w" Sep 17 01:12:36.474848 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.474822 2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7e32060-b3be-4c72-8cd8-98d099a7d02c-maas-api-tls\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:12:36.474848 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.474843 2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7e32060-b3be-4c72-8cd8-98d099a7d02c-metrics-certs\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:12:36.474848 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.474853 2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7e32060-b3be-4c72-8cd8-98d099a7d02c-openshift-service-ca\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:12:36.475066 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.474862 2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d85j\" (UniqueName: \"kubernetes.io/projected/e7e32060-b3be-4c72-8cd8-98d099a7d02c-kube-api-access-2d85j\") on node \"ip-10-0-134-177.ec2.internal\" DevicePath \"\"" Sep 17 01:12:36.600428 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.600396 2582 generic.go:356] "Generic (PLEG): container finished" podID="e7e32060-b3be-4c72-8cd8-98d099a7d02c" containerID="28007e4b282d2d6d2193c64ad890266dedc731ebc52591dca2b8cbd47562c323" exitCode=0 Sep 17 01:12:36.600610 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.600440 2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" Sep 17 01:12:36.600610 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.600477 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" event={"ID":"e7e32060-b3be-4c72-8cd8-98d099a7d02c","Type":"ContainerDied","Data":"28007e4b282d2d6d2193c64ad890266dedc731ebc52591dca2b8cbd47562c323"} Sep 17 01:12:36.600610 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.600554 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb" event={"ID":"e7e32060-b3be-4c72-8cd8-98d099a7d02c","Type":"ContainerDied","Data":"4e914b3885b3b9fe480e0c7cf368387f25bdc441881931ff03f17fba553b25e3"} Sep 17 01:12:36.600610 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.600577 2582 scope.go:117] "RemoveContainer" containerID="28007e4b282d2d6d2193c64ad890266dedc731ebc52591dca2b8cbd47562c323" Sep 17 01:12:36.612192 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.612173 2582 scope.go:117] "RemoveContainer" containerID="28007e4b282d2d6d2193c64ad890266dedc731ebc52591dca2b8cbd47562c323" Sep 17 01:12:36.612957 ip-10-0-134-177 kubenswrapper[2582]: E0917 01:12:36.612934 2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28007e4b282d2d6d2193c64ad890266dedc731ebc52591dca2b8cbd47562c323\": container with ID starting with 28007e4b282d2d6d2193c64ad890266dedc731ebc52591dca2b8cbd47562c323 not found: ID does not exist" containerID="28007e4b282d2d6d2193c64ad890266dedc731ebc52591dca2b8cbd47562c323" Sep 17 01:12:36.613056 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.612963 2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28007e4b282d2d6d2193c64ad890266dedc731ebc52591dca2b8cbd47562c323"} err="failed to get container status \"28007e4b282d2d6d2193c64ad890266dedc731ebc52591dca2b8cbd47562c323\": rpc error: code = NotFound desc = could not find container \"28007e4b282d2d6d2193c64ad890266dedc731ebc52591dca2b8cbd47562c323\": container with ID starting with 28007e4b282d2d6d2193c64ad890266dedc731ebc52591dca2b8cbd47562c323 not found: ID does not exist" Sep 17 01:12:36.620967 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.620946 2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb"] Sep 17 01:12:36.624263 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.624215 2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-cf64cbd4c-rwmgb"] Sep 17 01:12:36.738127 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:36.738095 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w"] Sep 17 01:12:36.742017 ip-10-0-134-177 kubenswrapper[2582]: W0917 01:12:36.741978 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc538c7b_5b1e_4ee3_8751_03ccd6e0231c.slice/crio-8df112f50c8e3c9d87a74907d196f45a9c57b02323b1732424b1317d8b86c634 WatchSource:0}: Error finding container 8df112f50c8e3c9d87a74907d196f45a9c57b02323b1732424b1317d8b86c634: Status 404 returned error can't find the container with id 8df112f50c8e3c9d87a74907d196f45a9c57b02323b1732424b1317d8b86c634 Sep 17 01:12:37.607811 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:37.607769 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w" event={"ID":"fc538c7b-5b1e-4ee3-8751-03ccd6e0231c","Type":"ContainerStarted","Data":"9cbe290b0f54f866c22e5176b3270a4e1ceb33b885b87c5596ca28115d6750c4"} Sep 17 01:12:37.608266 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:37.607840 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w" event={"ID":"fc538c7b-5b1e-4ee3-8751-03ccd6e0231c","Type":"ContainerStarted","Data":"8df112f50c8e3c9d87a74907d196f45a9c57b02323b1732424b1317d8b86c634"} Sep 17 01:12:37.608266 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:37.607872 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w" Sep 17 01:12:37.623919 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:37.623875 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w" podStartSLOduration=1.3214926249999999 podStartE2EDuration="1.623857858s" totalImagesPullingTime="302.365233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 01:12:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:12:37.621572704 +0000 UTC m=+2570.552206192" watchObservedRunningTime="2026-09-17 01:12:37.623857858 +0000 UTC m=+2570.554491345" Sep 17 01:12:37.650967 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:37.650933 2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7e32060-b3be-4c72-8cd8-98d099a7d02c" path="/var/lib/kubelet/pods/e7e32060-b3be-4c72-8cd8-98d099a7d02c/volumes" Sep 17 01:12:38.989799 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:38.989763 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:12:43.618515 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:43.618468 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79c8ddf6d7-phd9w" Sep 17 01:12:45.992045 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:45.992006 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:12:51.690837 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:51.690799 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:12:58.291868 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:12:58.291834 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-g59k9"] Sep 17 01:13:01.467527 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:01.467480 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cc5f98657-8w2m9_291bccc7-15e3-4fc5-8b83-b2d0920151a5/authorino/0.log" Sep 17 01:13:07.855945 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:07.855917 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-596c59f645-s9t67_5942742c-211c-4a48-ad10-effed1266242/copy-manifests/0.log" Sep 17 01:13:07.862494 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:07.862469 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-596c59f645-s9t67_5942742c-211c-4a48-ad10-effed1266242/manager/0.log" Sep 17 01:13:08.322156 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:08.322131 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79c8ddf6d7-phd9w_fc538c7b-5b1e-4ee3-8751-03ccd6e0231c/maas-api/0.log" Sep 17 01:13:09.423140 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:09.423113 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cc5f98657-8w2m9_291bccc7-15e3-4fc5-8b83-b2d0920151a5/authorino/0.log" Sep 17 01:13:09.748577 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:09.748464 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-zp6sg_af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5/kuadrant-console-plugin/0.log" Sep 17 01:13:09.999038 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:09.998958 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-g59k9_303a349d-cec8-4a13-b881-5c72bd957a37/limitador/0.log" Sep 17 01:13:10.808934 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:10.808906 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67475995c-q8sn2_84e2eca0-6325-479c-83ef-93d23a0d5c9f/kube-auth-proxy/0.log" Sep 17 01:13:11.740677 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:11.740647 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-9vr95_81343895-85b3-4e34-afb4-5cf2fc6466ef/main/0.log" Sep 17 01:13:12.077599 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:12.077571 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fpmflz_184376f5-676f-4d5b-b32e-0182b96cef13/main/0.log" Sep 17 01:13:20.538429 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:20.538393 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bs8f6_dda66642-c725-4b58-82ac-283b33d09dab/global-pull-secret-syncer/0.log" Sep 17 01:13:20.692193 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:20.692162 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-55r97_ec239ad8-714d-4805-b7c4-a0159e28b289/konnectivity-agent/0.log" Sep 17 01:13:20.759336 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:20.759313 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-177.ec2.internal_81937eb52bebe19931264bfcfb8d032c/haproxy/0.log" Sep 17 01:13:25.934484 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:25.934449 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7cc5f98657-8w2m9_291bccc7-15e3-4fc5-8b83-b2d0920151a5/authorino/0.log" Sep 17 01:13:26.015412 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:26.015382 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-zp6sg_af9ef9fe-5e78-4fcf-9d14-e58e7e2be3f5/kuadrant-console-plugin/0.log" Sep 17 01:13:26.079230 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:26.079200 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-g59k9_303a349d-cec8-4a13-b881-5c72bd957a37/limitador/0.log" Sep 17 01:13:27.388902 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:27.388870 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3ad5398-9310-4305-89e7-fd2edb38f0a0/alertmanager/0.log" Sep 17 01:13:27.408956 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:27.408929 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3ad5398-9310-4305-89e7-fd2edb38f0a0/config-reloader/0.log" Sep 17 01:13:27.427043 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:27.427018 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3ad5398-9310-4305-89e7-fd2edb38f0a0/kube-rbac-proxy-web/0.log" Sep 17 01:13:27.450525 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:27.450473 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3ad5398-9310-4305-89e7-fd2edb38f0a0/kube-rbac-proxy/0.log" Sep 17 01:13:27.469128 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:27.469102 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3ad5398-9310-4305-89e7-fd2edb38f0a0/kube-rbac-proxy-metric/0.log" Sep 17 01:13:27.493493 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:27.493468 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3ad5398-9310-4305-89e7-fd2edb38f0a0/prom-label-proxy/0.log" Sep 17 01:13:27.511941 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:27.511917 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a3ad5398-9310-4305-89e7-fd2edb38f0a0/init-config-reloader/0.log" Sep 17 01:13:27.576284 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:27.576253 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rp4km_81ae202e-68da-46af-84c4-572272397c69/kube-state-metrics/0.log" Sep 17 01:13:27.595092 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:27.595065 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rp4km_81ae202e-68da-46af-84c4-572272397c69/kube-rbac-proxy-main/0.log" Sep 17 01:13:27.617321 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:27.617294 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-rp4km_81ae202e-68da-46af-84c4-572272397c69/kube-rbac-proxy-self/0.log" Sep 17 01:13:27.647717 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:27.647640 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-76f456b74c-f8bcc_22b3bd80-9c5d-4926-9331-9bf124eb903c/metrics-server/0.log" Sep 17 01:13:27.890057 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:27.890027 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tsscd_f0af8806-ec80-41b8-bb64-73837f89b294/node-exporter/0.log" Sep 17 01:13:27.908595 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:27.908523 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tsscd_f0af8806-ec80-41b8-bb64-73837f89b294/kube-rbac-proxy/0.log" Sep 17 01:13:27.934900 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:27.934876 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tsscd_f0af8806-ec80-41b8-bb64-73837f89b294/init-textfile/0.log" Sep 17 01:13:27.968148 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:27.968117 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-sn9w5_8593ed54-bd76-4ca8-9f9e-78be23ae0f77/kube-rbac-proxy-main/0.log" Sep 17 01:13:27.987835 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:27.987809 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-sn9w5_8593ed54-bd76-4ca8-9f9e-78be23ae0f77/kube-rbac-proxy-self/0.log" Sep 17 01:13:28.009523 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:28.009478 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-sn9w5_8593ed54-bd76-4ca8-9f9e-78be23ae0f77/openshift-state-metrics/0.log" Sep 17 01:13:28.206697 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:28.206609 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-kqtm9_913b3f67-2d48-42ca-879a-7406e4bc0abe/prometheus-operator/0.log" Sep 17 01:13:28.225489 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:28.225462 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-kqtm9_913b3f67-2d48-42ca-879a-7406e4bc0abe/kube-rbac-proxy/0.log" Sep 17 01:13:28.247778 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:28.247731 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-n2s4x_dc8cc146-d16c-455c-96f3-28bf57efc91c/prometheus-operator-admission-webhook/0.log" Sep 17 01:13:28.277375 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:28.277343 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5554f8dfff-27h4m_9bfac6f7-c098-40bd-9d2d-8654c39e178c/telemeter-client/0.log" Sep 17 01:13:28.305083 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:28.305055 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5554f8dfff-27h4m_9bfac6f7-c098-40bd-9d2d-8654c39e178c/reload/0.log" Sep 17 01:13:28.324678 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:28.324646 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5554f8dfff-27h4m_9bfac6f7-c098-40bd-9d2d-8654c39e178c/kube-rbac-proxy/0.log" Sep 17 01:13:28.368734 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:28.368700 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-645649dfc7-f88xn_6dc1abe4-a68e-4cf7-890f-473426223b55/thanos-query/0.log" Sep 17 01:13:28.386934 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:28.386907 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-645649dfc7-f88xn_6dc1abe4-a68e-4cf7-890f-473426223b55/kube-rbac-proxy-web/0.log" Sep 17 01:13:28.404674 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:28.404649 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-645649dfc7-f88xn_6dc1abe4-a68e-4cf7-890f-473426223b55/kube-rbac-proxy/0.log" Sep 17 01:13:28.423554 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:28.423530 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-645649dfc7-f88xn_6dc1abe4-a68e-4cf7-890f-473426223b55/prom-label-proxy/0.log" Sep 17 01:13:28.441945 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:28.441921 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-645649dfc7-f88xn_6dc1abe4-a68e-4cf7-890f-473426223b55/kube-rbac-proxy-rules/0.log" Sep 17 01:13:28.462863 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:28.462803 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-645649dfc7-f88xn_6dc1abe4-a68e-4cf7-890f-473426223b55/kube-rbac-proxy-metrics/0.log" Sep 17 01:13:29.764311 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:29.764285 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-b6l4k_5cc2bc3b-4179-4230-9a3b-de67ca98ad50/networking-console-plugin/0.log" Sep 17 01:13:30.347897 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.347864 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/1.log" Sep 17 01:13:30.362166 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.362131 2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf"] Sep 17 01:13:30.362613 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.362576 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-nbp24_8da41563-ba0d-45a3-a634-6d95efdf8857/console-operator/2.log" Sep 17 01:13:30.362863 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.362840 2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7e32060-b3be-4c72-8cd8-98d099a7d02c" containerName="maas-api" Sep 17 01:13:30.362967 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.362867 2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7e32060-b3be-4c72-8cd8-98d099a7d02c" containerName="maas-api" Sep 17 01:13:30.363029 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.362991 2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7e32060-b3be-4c72-8cd8-98d099a7d02c" containerName="maas-api" Sep 17 01:13:30.366511 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.366476 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf"] Sep 17 01:13:30.366625 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.366589 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" Sep 17 01:13:30.368733 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.368712 2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jsqsk\"/\"default-dockercfg-bt7hf\"" Sep 17 01:13:30.368846 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.368744 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jsqsk\"/\"kube-root-ca.crt\"" Sep 17 01:13:30.368846 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.368797 2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jsqsk\"/\"openshift-service-ca.crt\"" Sep 17 01:13:30.462647 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.462616 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/767db43f-a963-46f3-ba38-4344bbb9f358-sys\") pod \"perf-node-gather-daemonset-lfqnf\" (UID: \"767db43f-a963-46f3-ba38-4344bbb9f358\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" Sep 17 01:13:30.462647 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.462653 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/767db43f-a963-46f3-ba38-4344bbb9f358-proc\") pod \"perf-node-gather-daemonset-lfqnf\" (UID: \"767db43f-a963-46f3-ba38-4344bbb9f358\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" Sep 17 01:13:30.462868 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.462677 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/767db43f-a963-46f3-ba38-4344bbb9f358-lib-modules\") pod \"perf-node-gather-daemonset-lfqnf\" (UID: \"767db43f-a963-46f3-ba38-4344bbb9f358\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" Sep 17 01:13:30.462868 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.462700 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvxmm\" (UniqueName: \"kubernetes.io/projected/767db43f-a963-46f3-ba38-4344bbb9f358-kube-api-access-zvxmm\") pod \"perf-node-gather-daemonset-lfqnf\" (UID: \"767db43f-a963-46f3-ba38-4344bbb9f358\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" Sep 17 01:13:30.462868 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.462787 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/767db43f-a963-46f3-ba38-4344bbb9f358-podres\") pod \"perf-node-gather-daemonset-lfqnf\" (UID: \"767db43f-a963-46f3-ba38-4344bbb9f358\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" Sep 17 01:13:30.563456 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.563422 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/767db43f-a963-46f3-ba38-4344bbb9f358-lib-modules\") pod \"perf-node-gather-daemonset-lfqnf\" (UID: \"767db43f-a963-46f3-ba38-4344bbb9f358\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" Sep 17 01:13:30.563650 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.563465 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvxmm\" (UniqueName: \"kubernetes.io/projected/767db43f-a963-46f3-ba38-4344bbb9f358-kube-api-access-zvxmm\") pod \"perf-node-gather-daemonset-lfqnf\" (UID: \"767db43f-a963-46f3-ba38-4344bbb9f358\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" Sep 17 01:13:30.563650 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.563574 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/767db43f-a963-46f3-ba38-4344bbb9f358-podres\") pod \"perf-node-gather-daemonset-lfqnf\" (UID: \"767db43f-a963-46f3-ba38-4344bbb9f358\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" Sep 17 01:13:30.563650 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.563640 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/767db43f-a963-46f3-ba38-4344bbb9f358-lib-modules\") pod \"perf-node-gather-daemonset-lfqnf\" (UID: \"767db43f-a963-46f3-ba38-4344bbb9f358\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" Sep 17 01:13:30.563765 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.563646 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/767db43f-a963-46f3-ba38-4344bbb9f358-sys\") pod \"perf-node-gather-daemonset-lfqnf\" (UID: \"767db43f-a963-46f3-ba38-4344bbb9f358\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" Sep 17 01:13:30.563765 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.563692 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/767db43f-a963-46f3-ba38-4344bbb9f358-sys\") pod \"perf-node-gather-daemonset-lfqnf\" (UID: \"767db43f-a963-46f3-ba38-4344bbb9f358\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" Sep 17 01:13:30.563765 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.563719 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/767db43f-a963-46f3-ba38-4344bbb9f358-proc\") pod \"perf-node-gather-daemonset-lfqnf\" (UID: \"767db43f-a963-46f3-ba38-4344bbb9f358\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" Sep 17 01:13:30.563765 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.563741 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/767db43f-a963-46f3-ba38-4344bbb9f358-podres\") pod \"perf-node-gather-daemonset-lfqnf\" (UID: \"767db43f-a963-46f3-ba38-4344bbb9f358\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" Sep 17 01:13:30.563896 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.563793 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/767db43f-a963-46f3-ba38-4344bbb9f358-proc\") pod \"perf-node-gather-daemonset-lfqnf\" (UID: \"767db43f-a963-46f3-ba38-4344bbb9f358\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" Sep 17 01:13:30.570897 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.570878 2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvxmm\" (UniqueName: \"kubernetes.io/projected/767db43f-a963-46f3-ba38-4344bbb9f358-kube-api-access-zvxmm\") pod \"perf-node-gather-daemonset-lfqnf\" (UID: \"767db43f-a963-46f3-ba38-4344bbb9f358\") " pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" Sep 17 01:13:30.676572 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.676443 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" Sep 17 01:13:30.923480 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.923447 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-cq9pb_a7c70a6f-dd20-4a97-bc6c-79eaf9cd53e9/download-server/0.log" Sep 17 01:13:30.946356 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:30.946329 2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf"] Sep 17 01:13:30.949655 ip-10-0-134-177 kubenswrapper[2582]: W0917 01:13:30.949626 2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod767db43f_a963_46f3_ba38_4344bbb9f358.slice/crio-cbaf04a950399b81cc3347eebf0f9de724c8a3c8af0cc7b030cf2682fad725eb WatchSource:0}: Error finding container cbaf04a950399b81cc3347eebf0f9de724c8a3c8af0cc7b030cf2682fad725eb: Status 404 returned error can't find the container with id cbaf04a950399b81cc3347eebf0f9de724c8a3c8af0cc7b030cf2682fad725eb Sep 17 01:13:31.878272 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:31.878235 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" event={"ID":"767db43f-a963-46f3-ba38-4344bbb9f358","Type":"ContainerStarted","Data":"2461abdb05c650bdf8d83d87643d5ac611a7564bd6e5da373527109796bdd7f6"} Sep 17 01:13:31.878488 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:31.878285 2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" event={"ID":"767db43f-a963-46f3-ba38-4344bbb9f358","Type":"ContainerStarted","Data":"cbaf04a950399b81cc3347eebf0f9de724c8a3c8af0cc7b030cf2682fad725eb"} Sep 17 01:13:31.878488 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:31.878333 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" Sep 17 01:13:31.892075 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:31.892016 2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" podStartSLOduration=1.8919973159999999 podStartE2EDuration="1.891997316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 01:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 01:13:31.891551466 +0000 UTC m=+2624.822184948" watchObservedRunningTime="2026-09-17 01:13:31.891997316 +0000 UTC m=+2624.822630807" Sep 17 01:13:32.721315 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:32.721284 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8vx7h_3f05fc1f-6243-45c9-8f0b-d8d6a40b6225/dns/0.log" Sep 17 01:13:32.741657 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:32.741632 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8vx7h_3f05fc1f-6243-45c9-8f0b-d8d6a40b6225/kube-rbac-proxy/0.log" Sep 17 01:13:32.868907 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:32.868881 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kzvtd_6d6d5c16-9a78-4f6e-b33b-bd62d16302b0/dns-node-resolver/0.log" Sep 17 01:13:33.417699 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:33.417670 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d7jth_ddb33aca-e881-421e-b64f-ca6aaa46298d/node-ca/0.log" Sep 17 01:13:34.712802 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:34.712772 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67475995c-q8sn2_84e2eca0-6325-479c-83ef-93d23a0d5c9f/kube-auth-proxy/0.log" Sep 17 01:13:35.683071 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:35.683039 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6qb8z_446d7623-90a7-453f-b8cc-dc54feffd5b2/serve-healthcheck-canary/0.log" Sep 17 01:13:36.162836 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:36.162808 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-gcss2_c7a57e9f-7ddf-46db-86f6-6772869c49d1/insights-operator/0.log" Sep 17 01:13:36.163807 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:36.163787 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-gcss2_c7a57e9f-7ddf-46db-86f6-6772869c49d1/insights-operator/1.log" Sep 17 01:13:36.184367 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:36.184336 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5gblm_176754c0-81e9-405a-b0e4-6b9feb1f1320/kube-rbac-proxy/0.log" Sep 17 01:13:36.202985 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:36.202961 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5gblm_176754c0-81e9-405a-b0e4-6b9feb1f1320/exporter/0.log" Sep 17 01:13:36.221265 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:36.221236 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5gblm_176754c0-81e9-405a-b0e4-6b9feb1f1320/extractor/0.log" Sep 17 01:13:37.894963 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:37.894936 2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jsqsk/perf-node-gather-daemonset-lfqnf" Sep 17 01:13:39.141928 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:39.141901 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-596c59f645-s9t67_5942742c-211c-4a48-ad10-effed1266242/manager/0.log" Sep 17 01:13:39.160546 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:39.160524 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-596c59f645-s9t67_5942742c-211c-4a48-ad10-effed1266242/copy-manifests/0.log" Sep 17 01:13:40.586882 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:40.586840 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-68945d9444-x5wrh_e11fff9b-5466-4dd4-a845-51b2a07aefe5/manager/0.log" Sep 17 01:13:41.482896 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:41.482871 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67475995c-q8sn2_84e2eca0-6325-479c-83ef-93d23a0d5c9f/kube-auth-proxy/0.log" Sep 17 01:13:43.432869 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:43.432832 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67475995c-q8sn2_84e2eca0-6325-479c-83ef-93d23a0d5c9f/kube-auth-proxy/0.log" Sep 17 01:13:45.892156 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:45.892118 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-69rz8_2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c/kube-storage-version-migrator-operator/1.log" Sep 17 01:13:45.893045 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:45.893013 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-69rz8_2a87c150-6ca6-43b5-a7fc-e8a6b43feb3c/kube-storage-version-migrator-operator/0.log" Sep 17 01:13:46.923287 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:46.923259 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6pld9_8bdcdeea-d660-4ba8-8186-a2c495239b90/kube-multus-additional-cni-plugins/0.log" Sep 17 01:13:46.941672 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:46.941643 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6pld9_8bdcdeea-d660-4ba8-8186-a2c495239b90/egress-router-binary-copy/0.log" Sep 17 01:13:46.959444 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:46.959423 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6pld9_8bdcdeea-d660-4ba8-8186-a2c495239b90/cni-plugins/0.log" Sep 17 01:13:46.977630 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:46.977607 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6pld9_8bdcdeea-d660-4ba8-8186-a2c495239b90/bond-cni-plugin/0.log" Sep 17 01:13:46.995397 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:46.995379 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6pld9_8bdcdeea-d660-4ba8-8186-a2c495239b90/routeoverride-cni/0.log" Sep 17 01:13:47.014159 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:47.014135 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6pld9_8bdcdeea-d660-4ba8-8186-a2c495239b90/whereabouts-cni-bincopy/0.log" Sep 17 01:13:47.032170 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:47.032150 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6pld9_8bdcdeea-d660-4ba8-8186-a2c495239b90/whereabouts-cni/0.log" Sep 17 01:13:47.215649 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:47.215555 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fmx98_29c30e84-c3d7-4e23-a4dd-efc80b57ccb5/kube-multus/0.log" Sep 17 01:13:47.383978 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:47.383945 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rh4cz_6155b312-55da-43b1-9b1f-4d51dfbb0d42/network-metrics-daemon/0.log" Sep 17 01:13:47.400948 ip-10-0-134-177 kubenswrapper[2582]: I0917 01:13:47.400927 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rh4cz_6155b312-55da-43b1-9b1f-4d51dfbb0d42/kube-rbac-proxy/0.log"